发信人: sandbird (sandbird), 信区: Statistics 标 题: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Fri Jan 30 18:10:32 2009) Can I ask R to "read.csv" of a 150G file and extract records from this table . Anyone has similar experience doing this in Windows 64bit machine? or any suggestions of doing it under Unix R or Windows R? Thanks -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 128.231.]
发信人: hezhi (荷芝), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Fri Jan 30 19:48:28 2009), 转信 太强了吧 【 在 sandbird (sandbird) 的大作中提到: 】 : Can I ask R to "read.csv" of a 150G file and extract records from this table : . : Anyone has similar experience doing this in Windows 64bit machine? : or any suggestions of doing it under Unix R or Windows R? : Thanks -- ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 98.212.]
发信人: leohawk (leohawk), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Fri Jan 30 20:14:13 2009), 转信 kao, niu a ... if you know what you want, better use grep to pre-process ba... I do not think you can do that on even a super computer 【 在 sandbird (sandbird) 的大作中提到: 】 : Can I ask R to "read.csv" of a 150G file and extract records from this table : . : Anyone has similar experience doing this in Windows 64bit machine? : or any suggestions of doing it under Unix R or Windows R? : Thanks -- ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]
发信人: RProgrammer (RP), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Fri Jan 30 22:42:25 2009) Almost impossible. So far as I know, there is no stable version of 64-bit version of R under windows, and if I remembered correctly,u nder windows, the maximum memory of R that can use is 2 G. For Unix or Linux, there are 64-bit version of R, but the maximum memory space R can utilize is still limited, say, some package can not handle more than a certain number of entries (2^(32)-1) in a matrix, or an object can not be more than a certain size that C or Fortran can handle, no matter how large your machine's physical memory is. Maybe you can do a feature selection/dimension reduction, etc. to bring down the volume of the data, or do all the calculation using low level programming language other than R, or probably do all the calculation in the database using the built in database syntax, or select a portion of the data samples to load into R and do it iteratively if it is appropriate for your calculation. 【 在 sandbird (sandbird) 的大作中提到: 】 : Can I ask R to "read.csv" of a 150G file and extract records from this table : . : Anyone has similar experience doing this in Windows 64bit machine? : or any suggestions of doing it under Unix R or Windows R? : Thanks -- 各有所好吗~~~ ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.]
发信人: leftHand (Hold on...), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 13:46:40 2009), 转信 is there a 64bit R for windows? is there a compile version that can be downloaded? 【 在 RProgrammer (RP) 的大作中提到: 】 : 标 题: Re: 能不能让R 读一个 150G 的 table? : 发信站: BBS 未名空间站 (Fri Jan 30 22:42:25 2009) : : Almost impossible. : : So far as I know, there is no stable version of 64-bit version of R under : windows, and if I remembered correctly,u nder windows, the maximum memory of : R : that can use is 2 G. : : For Unix or Linux, there are 64-bit version of R, but the maximum memory : space : R can utilize is still limited, say, some package can not handle more than a : certain number of entries (2^(32)-1) in a matrix, or an object can not be : more : than a certain size that C or Fortran can handle, no matter how large your : machine's physical memory is. : : Maybe you can do a feature selection/dimension reduction, etc. to bring down : the volume of the data, or do all the calculation using low level : programming : language other than R, or probably do all the calculation in the database : using : the built in database syntax, or select a portion of the data samples to : load : into R and do it iteratively if it is appropriate for your calculation. : : : : : 【 在 sandbird (sandbird) 的大作中提到: 】 : : Can I ask R to "read.csv" of a 150G file and extract records from this : table : : . : : Anyone has similar experience doing this in Windows 64bit machine? : : or any suggestions of doing it under Unix R or Windows R? : : Thanks : : : : : -- : 各有所好吗~~~ : : ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.] -- ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 68.50.]
发信人: daydayup1 (咳咳), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 15:33:37 2009) 那什么语言能干这种事情呢? SAS 行吗? MATLAB, S-PLUS? 【 在 RProgrammer (RP) 的大作中提到: 】 : Almost impossible. : So far as I know, there is no stable version of 64-bit version of R under : windows, and if I remembered correctly,u nder windows, the maximum memory of : R : that can use is 2 G. : For Unix or Linux, there are 64-bit version of R, but the maximum memory : space : R can utilize is still limited, say, some package can not handle more than a : certain number of entries (2^(32)-1) in a matrix, or an object can not be : more : ................... -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 67.80.]
发信人: sir ( 郎 ), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 16:11:00 2009) read line by line read one, throw one -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 64.69.]
发信人: leohawk (leohawk), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 16:32:29 2009), 转信 it is gonna awefully slow 【 在 sir ( 郎 ) 的大作中提到: 】 : read line by line : read one, throw one -- ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]
发信人: sir ( 郎 ), 信区: Statistics 标 题: Re: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 18:30:48 2009) 用时间换空间啦 -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.233.]
发信人: RProgrammer (RP), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sat Jan 31 23:32:58 2009) I remembered Brian Ripley said he had compiled one but crashes all the time. I am not sure whether there is a stable compiled version ready to be downloaded now. 【 在 leftHand (Hold on...) 的大作中提到: 】 : is there a 64bit R for windows? is there a compile version : that can be downloaded? : of : a : down -- 公司来了个老印...对我有些儿崇拜... ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.]
发信人: sandbird (sandbird), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sun Feb 1 01:00:07 2009) thanks for all of your replies, though i still haven't found a solution yet. -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.114.]
发信人: NKVD (Chuikov), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sun Feb 1 02:07:05 2009), 转信 hadoop 【 在 sandbird (sandbird) 的大作中提到: 】 : Can I ask R to "read.csv" of a 150G file and extract records from this table : . : Anyone has similar experience doing this in Windows 64bit machine? : or any suggestions of doing it under Unix R or Windows R? : Thanks -- In dream: D700 + 300 F/4 VR. ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.87.]
发信人: leohawk (leohawk), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sun Feb 1 05:28:42 2009), 转信 I compiled R 2.7 on a 64 bit SLED, it took me a while to get it done, mainly getting all required packages. I do not think there are compiled version for download. 【 在 RProgrammer (RP) 的大作中提到: 】 : I remembered Brian Ripley said he had compiled one but crashes all the time. : I am not sure whether there is a stable compiled version ready to be : downloaded now. -- ※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]
发信人: barrybarry (barrybarry), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sun Feb 1 12:34:16 2009) Depending on the nature of your problem, you may try using the SCAN function to read line-by-line and do the processing. Or, you can first load to data into SQL database, and process only a portion of the whole data. The bottom line is that R could not handle such a database all at once. SAS is better suited for this kind of job. -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.108.]
发信人: statcompute (statcompute), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Sun Feb 1 16:23:54 2009) if i were you, i will: 1) read the data in to R piece by piece 2) insert cases into a light-weight database, such as slqite. 3) R has a very good interface with sqlite and you can easily extract data from sqlite by R. -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 68.83.]
发信人: barrybarry (barrybarry), 信区: Statistics 标 题: Re: 能不能让R 读一个 150G 的 table? 发信站: BBS 未名空间站 (Fri Feb 6 10:25:51 2009) try the "filehash" package. -- ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 208.62.]
版权所有,未名空间(mitbbs.com),since 1996 Site Map - Contact Us - Terms and Conditions - Privacy Policy