Memory Mapped FIle is slow

Question

I am trying to read from a memory mapped file, but access to the file is taking a long time. I am mapping the whole file to my program, and initial access to fast, but then it begins to drastically slow down

The file is ~47gb and I have 16gb of RAM. I am running a 64-bit application on windows 7 using Visual Studios as my IDE. Below is snippet of my code

 hFile = CreateFile( "Valid Path to file", // name of the write GENERIC_READ , // open for reading 0, // do not share NULL, // default security OPEN_EXISTING, // existing file only FILE_ATTRIBUTE_NORMAL, // normal file NULL); // no attr. template if (hFile == INVALID_HANDLE_VALUE) { cout << "Unable to open vals" << endl; exit(1); } hMapFile = CreateFileMapping( hFile, NULL, // default security PAGE_READONLY, // read/write access 0, // maximum object size (high-order DWORD) 0, // maximum object size (low-order DWORD) NULL); // name of mapping object if (hMapFile == NULL) { cout<< "Error code " << GetLastError() << endl; exit(1); } data = (float*) MapViewOfFile( hMapFile, FILE_MAP_READ, 0, 0, 0); if (data == NULL) { cout << "Error code " << GetLastError() << endl; CloseHandle(hFile); exit(1); }

Is this just because the file is so large that continually swapping chunks of the file takes long, or is it some other parameter I need for faster access?

EDIT: I tried using read only instead of using read, write, execute as seen above, but the speed is still slow. I understand the concepts of memory mapping and switch swap space, but I thought I may have been doing something else wrong with was hindering the speed

I definitely would not use PAGE_EXECUTE_READWRITE or GENERIC_EXECUTE for a datafile. — Ben Voigt
– Ben Voigt, Commented Jul 14, 2015 at 15:28
Once you've used enough of the file to fill your available RAM, accessing a new block requires swapping another block out. Of course it's going to slow down. — Mark Ransom
– Mark Ransom, Commented Jul 14, 2015 at 15:30
What's your typical access pattern? Are you accessing the file in random order or sequentially? Are you touching all of the file or just portions of it? Your view is read only, but you open everything else with read/write an execute(!) .. Don't ask for more access than you need, as that can create constraints that slow things down. — Adrian McCarthy
– Adrian McCarthy, Commented Jul 14, 2015 at 15:55

Lawrence Aiello · Accepted Answer · 2015-07-14 15:31:51Z

This is because of paging. What is happening is that your RAM can only hold 16GB of the file (in reality it is less because of other programs running on your computer, but let's just use this for simplicity).

So if you access a part of the file in your program that is not in RAM (let's say, a part of the file that is in the 20GB segment) your RAM needs to talk to the disk and transfer a whole new segment of the file to RAM. This takes a lot of time.

But then how do other applications accessing large files still remain mostly responsive. Is it through threading so each thread looks at different portions of the file?
@user2521432 disks are optimized for serial access (even SSDs) and that's the typical access pattern, so things are speedy. Whenever you do intense random access to a file, it's going to be slow, no matter if you're reading it directly or letting the OS do it through memory mapping.

Collectives™ on Stack Overflow

Memory Mapped FIle is slow

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related