2

My C++ application occasionally runs out of memory due to large amounts of data being retrieved from a database. It has to run on 32bit WinXP machines.

Is it possible to transparently (for most of the existing code) swap out the data objects to disk and read them into memory only on demand, so I'm not limited to the 2GB that 32bit Windows gives to the process?

I've looked at VirtualAlloc and Address Window Extensions but I'm not sure it's what I want.

I also found this SO question where the questioner creates a file mapping and wants to create objects in there. One answer suggests using placement new which sounds like it would be pretty transparent to the rest of the code.

Will this prevent my application to run out of physical memory? I'm not entirely sure of it because after all there is still the 32bit address space limit. Or is this a different kind of problem that will occur when trying to create a lot of objects?

Improve this question
6
  • Doesn't windows use Swap file exactly for this use case? It's slow as hell, but it will also be with your solution. Commented Aug 23, 2012 at 11:17
  • 3
    @BartekBanachewicz: A swap file also can not do anything against the 2/4GB virtual address space limits on common 32bit OSs Commented Aug 23, 2012 at 11:20
  • The usual solution is not to retrieve all the data at once. If you already have it in the database, read it out in sections. Commented Aug 23, 2012 at 11:21
  • Ah, indeed. Well, the custom memory management seems good. Look for ready implementations, though. Commented Aug 23, 2012 at 11:21
  • 1
    I've not used it myself, but STXXL sounds like a good way to handle data structures that won't fit in memory. Commented Aug 23, 2012 at 12:43

1 Answer 1

Reset to default
1

So long as you are using a 32-bit operating system there is nothing you can do about this. There is no way to have more than 3GB (2GB in the case of Windows) of data in virtual memory, whether or not it's actually swapped out to disk.

Historically databases have always handled this problem by using read, write and seek. So rather than accessing data directly from memory, they use a fake (64-bit) pointer. Data is split into blocks (normally around 4kb), and a number of these blocks are allocated in memory. When they want to access data from a fake pointer address they check if the block is loaded into memory and if it is they access it from there. If it is not then they find an empty slot and copy it in, then return the address. If there are no slots free then a piece of data will be written back out to disk (if it's been modified) and that slot will be reused.

The real beauty of this is that if your system has enough RAM then the operating system will cache much more than 2GB of this data in RAM at any point in time, and when you feel like you are actually reading and writing from disk the operating system will probably just be copying data around in memory. This, of course, requires a 32-bit operating system that support more than 3GB of physical memory, such as Linux or Windows Server with PAE.

SQLite has a nice self-contained implementation of this, which you could probably make use of with little effort.

If you do not wish to do this then your only alternatives are to either use a 64-bit operating system or to work with less data at any given point in time.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for the extensive answer!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.