memory allocation of 100 bytes failed

Question

I have some code that needs to create a large array. On my local computer (OSX) the program runs ok. However when I try to run the program on Ubuntu DigitalOcean droplet I get the following error:

memory allocation of 100 bytes failedAborted

There isn't any other information provided in the output, but I think it has to do with initializing the vector.

fn example() { let n = 25; let mut dp: Vec<Vec<f32>> = vec![vec![-1.0; n]; 2i32.pow(n as u32) as usize]; }

The size of that vector can get quite large in some instances. Is there a better way to create this large vector or is this caused by a system limit of memory?

for what value of n, what are the spec of your machine, "is this caused by a system limit of memory" obviously - - — Stargateur
– Stargateur, Commented Jul 28, 2019 at 15:21
Suppose n = 25, so your inner vectors require 100 bytes each. The outer vector has length 2**25, so the total size is (2**25)*100 which is more than 3 gigabytes. How much memory do you have in your DO droplet? — Warren Weckesser
– Warren Weckesser, Commented Jul 28, 2019 at 15:48
Aside from the memory issues, you should use 1 << n, as the outer size. Aside from being more efficient (it is a shift rather than a general power), no casts are needed, as everything works as a usize. — David Brown
– David Brown, Commented Jul 28, 2019 at 18:49
@DavidBrown: Actually, I find pow more readable. It could be written better though, since usize also has pow: 2_usize.pow(n as u32). I guess the OP following the compiler suggestion to add _i32 and then was stuck with a cast. — Matthieu M.
– Matthieu M., Commented Jul 29, 2019 at 13:47
@ThomasC: Nothing wrong in following the suggestions, they're specifically here to be followed. I am more disappointed in the poor suggestion here :/ — Matthieu M.
– Matthieu M., Commented Jul 29, 2019 at 14:00

Kornel · Accepted Answer · 2022-07-07 14:39:34Z

You've requested a Vec that's too large and ran out of memory. When this happens, Rust will abort the program.

The failure to allocate 100 bytes, rather than >3GB is surprising, but it's probably because behavior of memory allocation on Linux is very unintuitive. Linux pretends to have an infinite amount of memory available, and will allow overly large allocations (overcommit) until it can't bluff any more.

You can mitigate this somewhat by using try_reserve:

let mut vec = Vec::new(); let size = 2.pow(25); // this may fail, but won't hard abort vec.try_reserve(size)?; // now you can safely add up to the `size` to the vec vec.resize(size, -1.0);

There is fallible_collections crate that has more OOM-handling methods for Vec and others.

Another thing to consider is setting your own hard limit for memory usage for your Rust program using the cap allocator wrapper. This lets you avoid wrath of Linux's OOM killer, and limit your program's allocations before they become a machine-wide problem.

How can I check (not necessarily programically) how much memory does my machine have for such allocations in Rust?
@AnatolyBugakov On most systems the top command will show you memory stats at the top. However, note that Linux and most OSes with virtual memory can't give you a straight answer to how much "free" memory there is, because they use very complex tricks with memory reuse and lazy allocation. They don't know!

Collectives™ on Stack Overflow

memory allocation of 100 bytes failed

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related