1

I am having run-time memory allocation errors with a C++ application. I have eliminated memory leaks, invalid pointer references and out-of-bounds vector assignments as the source of the issue - I am pretty sure it's something to do with memory fragmentation and I am hoping to get help with how to further diagnose and correct the problem.

My code is too large to post (about 15,000 lines - I know not huge but clearly too large to put online), so I am going to describe things with a few relevant snippets of code.

Basically, my program takes a bunch of string and numerical data sets as inputs (class objects with vector variables of type double, string, int and bool), performs a series of calculations, and then spits out the resulting numbers. I have tested and re-tested the calculations and outputs - everything is calculating as it should, and on smaller datasets things run perfectly.

However, when I scale things up, I start getting memory allocation errors, but I don't think I am even close to approaching the memory limits of my system - please see the two graphs below...my program cycles through a series of scenarios (performing identical calculations under a different set of parameters for each scenario) - in the first graph, I run 7 scenarios on a dataset of about 200 entries. As the graph shows, each "cycle" results in memory swinging up and back down to its baseline, and the overall memory usage is tiny (see the seven small blips on the right half of the bottom graph). On the second graph, I am now running a dataset of about 10,000 entries (see notes on dataset below). In this case, I only get through 2 full cycles before getting my error (as it is trying to resize a class object for the third scenario). You can see the first two scenarios in the bottom right-half graph; a lot more memory usage than before, but still only a small fraction of available memory. And as with the smaller dataset, usage increases while my scenario runs, and then decreases back to it's initial level before reaching the next scenario.

This pattern, along with other tests I have done, lead me to believe it's some sort of fragmentation problem. The error always occurs when I am attempting to resize a vector, although the particular resize operation that causes the error varies based on the dataset size. Can anyone help me understand what's going on here and how I might fix it? I can describe things in much greater detail but already felt like my post was getting long...please ask questions if you need to and I will respond/edit promptly.

Clarification on the data set The numbers 200 and 10,000 represent the number of unique records I am analyzing. Each record contains somewhere between 75 and 200 elements / variables, many of which are then being manipulated. Further, each variable is being manipulated over time and across multiple iterations (both dimensions variable). As a result, for an average "record" (the 200 to 10,000 referenced above), there could be easily as many as 200,000 values associated with it - a sample calculation:

1 Record * 75 Variables * 150 periods * 20 iterations = 225,000 unique values per record.

SmallDataRun

LargeDataSetRun

Offending Code (in this specific instance):

vector<LoanOverrides> LO; LO.resize(NumOverrides + 1); // Error is occuring here. I am certain that NumOverrides is a valid numerical entry = 2985 // Sample class definition class LoanOverrides { public: string IntexDealName; string LoanID; string UniqueID; string PrepayRate; string PrepayUnits; double DefaultRate; string DefaultUnits; double SeverityRate; string SeverityUnits; double DefaultAdvP; double DefaultAdvI; double RecoveryLag; string RateModRate; string RateModUnits; string BalanceForgivenessRate; string BalanceForgivenessRateUnits; string ForbearanceRate; string ForbearanceRateUnits; double ForbearanceRecoveryRate; string ForbearanceRecoveryUnits; double BalloonExtension; double ExtendPctOfPrincipal; double CouponStepUp; };
Improve this question
10
  • Where are the graphs? Commented Sep 5, 2013 at 13:50
  • How large are the objects you're putting in the vectors? 32-bit or 64-bit? Commented Sep 5, 2013 at 13:53
  • @Peter I am not sure how to answer that - forgive my ignorance - they are mostly simple data types like int, bool and double, and then a fair number of std:string. I posted a sample class definition if that helps. Commented Sep 5, 2013 at 13:56
  • 2
    What compiler do you use? I suggest you search for known-bugs regarding this, if none are found, try to narrow it down until you get a program with no more than a 30 or so lines, then ask about it. Though it's a lot more likely that you've got some undefined behaviour caused by something else, than having a compiler bug. Commented Sep 5, 2013 at 14:19
  • 1
    I think @Dukeling is right. Undefined behavior (= you're stepping on some memory somewhere) is sometimes detected by the allocator as a corrupted heap. Your structures (that you've shown to us) aren't big enough to cause fragmentation issues. Commented Sep 5, 2013 at 14:33

1 Answer 1

Reset to default
3

You have a 64-bit operating system capable of allocating large quantities of memory, but have built your application as a 32-bit application, which can only allocate a maximum of about 3GB of memory. You are trying to allocate more than that.

  • Try compiling as a 64-bit application. This may enable you to reach your goals. You may have to increase your pagefile size.
  • See if you can dispose of intermediate results earlier than you are currently doing so.
  • Try calculating how much memory is being used/would be used by your algorithm, and try reworking your algorithm to use less.
  • Try avoiding duplicating data by reworking your algorithm. I see you have a lot of reference data, which by the looks of it isn't going to change during the application run. You could put all of that into a single vector, which you allocate once, then refer to them via integer indexes everywhere else, rather than copying them. (Just guessing that you are copying them).
  • Try avoiding loading all the data at once by reworking your algorithm to work on batches.

Without knowing more about your application, it is impossible to offer better advice. But basically you are running out of memory because you are allocating a huge, huge amount of it, and based on your application and the snippets you have posted I think you can probably avoid doing so with a little thought. Good luck.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Very helpful response - I will try each of these and reply with what I find. Thank you
The 64-bit change allowed me to run everything without error - I believe my large test case required about 3.5GB of memory to run without errors. As you also pointed out, there are clearly other ways to save on space by only retaining data as long as its needed, which I will work in implementing, but for now, problem solved!
If your application needed 3.5 GB, you could still get it to run as a 32-bit application, but you need to compile it as "large-address aware" which is not a default setting in Visual Studio. You may also need to modify the OS configuration as well depending on the version of Windows.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.