2

Is it possible to manually allocate memory in an R session? I work on a shared windows server that sometimes runs into memory issues. When using Stata is it possible to allocate memory for that session with: 

set min_memory 50g

There is something similar in R?

Edit to provide better context:

Why would I need such a command?

Let's say we have a script that peaks the RAM usage at 20Gb but only uses 5Gb most of the time. The natural way that R handle this is to allocate the memory only as needed( which in most cases seems reasonable). But imagine you need to run this program on a shared server( with a lot more RAM than that) that can, due to other users needs, not have 20Gb free memory at the point the program peaks.

In that situation I think would be nice to avoid such a risk by previous allocating the 20Gb you will need for that R session( as it is possible in Stata) when starting the program, and then avoiding to waste resources in a run that will not even finish.

Improve this question
7
  • 5
    memory.limit() Commented Jul 27, 2017 at 19:36
  • @HFBrowning memory.limit() returns the maximum allocation( in most cases the total RAM of your computer). I would like to change the minimun memory ( the amount of memory allocated to the process). It would be something equivalent to creating a big fake object that uses the memory and then deleting it when it is needed( so no other process or user could have used it first). Commented Jul 27, 2017 at 20:05
  • @HFBrowning That would change the maximum amount of memory that the r session can allocate, wich is not what I am trying to do. Commented Jul 27, 2017 at 20:14
  • I don't think I'm really following how what you want to do is different than what memory.limit() can accomplish - have you looked at: stackoverflow.com/questions/1395229/… ? Commented Jul 27, 2017 at 20:21
  • 1
    @HFBrowning I haved edited the question to provide better context on why would I need such a command, and maybe that clarifies why I think memory.limit() do not help Commented Jul 31, 2017 at 19:15

1 Answer 1

Reset to default
2

TL;DR: I did a lot of digging and could not find anything that even hinted at a "minimum memory size" setting.

If I understand your question correctly, you would like to have R request a very large (20 Gb) amount of memory up front for a session and then manage this memory on its own in order to ensure that resources are available when needed. From what I have been able to gather, this is not possible and it is approximately the opposite of the strategy employed by R’s designers.

As you have stated, R allocates and manages memory by:

  1. Setting a max limit for the session
  2. Allocating as objects are created
  3. Within certain limits, reclaiming memory as soon as the object is done

For small allocations, I think step 3 is done using the C function alloca() which uses the stack and for larger ones it uses malloc() which allocates on the heap. What this effectively means is that R can request a small pool of memory (128 bytes, according to Advanced R) to manage itself for small objects that come and go pretty quickly, but for larger objects it lets the operating system handle it.

All of the focus I have seen from comments in the source and R internals suggests that the dev team was focused on writing software that would only hold the amount of memory needed for current computation and then promptly release it:

We want the heap sizes to get set at levels adequate for the current computations. The present mechanism uses only the size of the current live heap to provide information about the current needs; since the current live heap size can be very volatile, the adjustment mechanism only makes gradual adjustments. A more sophisticated strategy would use more of the live heap history (lines 314 – 320 in source)

That being said, there could be a package or extension that has re-written some of this core functionality that I am not aware of – although this seems like such a dramatic departure from the way R was written it would probably be a fork of the project rather than a package. If you are not interested in doing that yourself, I would recommend a more mundane solution, like running your program overnight to avoid competition for resources. Or incrementally processing your data, or seeing if something like a database or the bigmemory package could help you. Good luck!

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.