Memory problems in R

Question

Here is a phenomenon that happens all to often. I try to manipulate some sort of big data, for example

a <- matrix( rnorm( 1e4 * 200 ), ncol= 1e4 ) gr <- factor( rep( 1:2, each= 100 ) ) l <- lm( a ~ gr ) covs <- estVar( l ) cors <- cov2cor( covs )

Quite often, the following error is reported: Error: cannot allocate vector of size 509.5 Mb

Fine. I remove some variables I don't need any more and call the garbage collector:

rm( a, l ) gc( TRUE )

However, the error persists. Now I save R and start it again. And -- a miracle happens: the memory is now available. Why? If there was not enough memory for R to allocate before, but there is enough now, what changed? Can I force R somehow to clean up without saving the data to disk and waiting until it loads them again? I don't get it.

my sessionInfo():

> sessionInfo() R version 3.0.1 (2013-05-16) Platform: i486-pc-linux-gnu (32-bit) locale: [1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 LC_MONETARY=en_US.utf8 [6] LC_MESSAGES=en_US.utf8 LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C attached base packages: [1] graphics utils datasets grDevices stats methods base

P.S.: The system appears to have plenty of unused memory left, as reported by free. top reports that my R process (before the error) is using up ~ 2GB out of my 8, and there is still plenty more left.

Did you look at any of the existing posts here on R and memory management? — Dirk is no longer here
– Dirk is no longer here, Commented Sep 4, 2013 at 13:45
@DirkEddelbuettel: Yep. Several mention using gc(), some are using (now obsolete) command line switches, also data.table() is recommended (but merging of large data is not the problem here). Most are proposing ad hoc modifications for the problem at hand such that less memory is used. I have not found an answer to my specific questions (why gc() doesn't work, why do I have to restart R). — January
– January, Commented Sep 4, 2013 at 13:48
gc() clears stuff you don't need, but if you have lots of objects in your environment using memory then it is possible you can't get a block of contiguous memory big enough. Run a session to recreate your problem again, then use rm( list = ls() ) ; gc() and run the code again and see if it works. — Simon O'Hanlon
– Simon O'Hanlon, Commented Sep 4, 2013 at 13:50
I have found that calling gc() a few times in a row helps as well--just wrap a (for i in 1:n) around it. But in essence: "get more RAM" or "use smaller objects" is where it is at. — Dirk is no longer here
– Dirk is no longer here, Commented Sep 4, 2013 at 13:53
"Big data" and "32-bit" are generally incompatible. RAM is super cheap these days.... — Ari B. Friedman
– Ari B. Friedman, Commented Sep 4, 2013 at 14:06

Simon O'Hanlon · Accepted Answer · 2013-09-04 14:19:53Z

2

Install and use 64-bit R to take advantage of more RAM. From ?Memory-limits:

Unix
The address-space limit is system-specific: 32-bit OSes imposes a limit of no more than 4Gb: it is often 3Gb. Running 32-bit executables on a 64-bit OS will have similar limits: 64-bit executables will have an essentially infinite system-specific limit (e.g. 128Tb for Linux on x86_64 cpus).

answered Sep 4, 2013 at 14:19

Simon O'Hanlon

60.2k15 gold badges145 silver badges189 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

January Over a year ago

This is weird. Address-space depends on the kernel, kernel uses the PAE extension and is able to address the full memory.

gwatson Over a year ago

It appears he is using a 32 bit machine, so this wouldn't be possible. Note Platform: i486-pc-linux-gnu (32-bit) from sessionInfo

Simon O'Hanlon Over a year ago

@gwatson so why have 8GB ram on a 32bit OS? I assume there is a dual-boot or he is using some kind of virtualbox setup?

Dirk is no longer here Over a year ago

@January: AFAIK you get access to all your memory with PAE but the single largest chunk is still 3 gb. You can't make a 32-bit int any larger via a kernel extension.

January Over a year ago

@gwatson: nah, I'm just old and still treat 64b systems as a novelty. This is the first time I see a reason to move from 32b.

|

gwatson · Accepted Answer · 2013-09-04 14:30:34Z

Not sure what else can be done on a 32 bit machine. Maybe trying using one of the popular out of memory processing packages like bigmemory? Here's a link, http://www.bigmemory.org/

Collectives™ on Stack Overflow

Memory problems in R

2 Answers 2

10 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Related