Can not load large image with opencv

Question

I try to load an image with opencv that has the size 100.000 * 15.000 pixels and a file size of 4.305 kb. If i load this image with the following code:

 cv::Mat reallyBigImage = cv::imread("reallyBigImage.png");

i get the following error:

I have a pc with 32 gb ram and compile this program in 64 bit. This should be enough for an image of this size. Is it possible to load such an large image as whole to opencv?

Here is where my program breaks in assembler. The arrow indicates the specific location. This snippet is from the memcpy.asm.

 CopyUp: cmp r8, 128 jbe XmmCopySmall bt __favor, __FAVOR_ENFSTRG ; check for ENFSTRG (enhanced fast strings) jnc XmmCopyUp ; If Enhanced Fast String not available, use XMM ; use Enhanced Fast Strings ; but first align the destination dst to 16 byte alignment mov rax, r11 ; return original destination pointer mov r11, rdi ; save rdi in r11 mov rdi, rcx ; move destination pointer to rdi mov rcx, r8 ; move length to rcx mov r8, rsi ; save rsi in r8 mov rsi, r10 ; move source pointer to rsi -->rep movsb ; copy source to destination buffer mov rsi, r8 ; restore rsi mov rdi, r11 ; restore rdi ret

I can load images that are smaller than the one i try to load. And yes i can write to this folder. — Freewister
– Freewister, Commented Oct 9, 2017 at 14:22
I use the prebuilt opencv library from nuget version 310.3.0 for VS2015/2017. opencv.win.native and opencv.win.native.redist — Freewister
– Freewister, Commented Oct 9, 2017 at 15:13

Dan Mašek · Accepted Answer · 2017-10-09 19:59:27Z

Running some local tests, this can be reproduced with OpenCV 3.1, but not with 3.2 or 3.3. That suggests it was a bug (along with the fact that you're getting an access violation in the first place).

Specifically issue #6317.

This was fixed prior to release of OpenCV 3.2, which would match the above observations. Hence, the best option would be to upgrade your copy of OpenCV.

I updated to version 3.3 from 3.1 and i could load the image. Thanks for the help.

Scott Mudge · Accepted Answer · 2017-10-09 14:30:48Z

That image is 6 GB raw data (100,000 * 15,000 * 4 Bytes). Typically Windows limits the maximum heap size of a process much below that. But even if you increased that limit, you're going to encounter memory fragmentation (being unable to write to a single, 6GB contiguous block of memory), which OpenCV's allocator likely cannot handle.

Which is why you are getting access violation error during writing the image data to memory. It's attempting to write into a memory location it is not allowed/supposed to be writing in.

You're either going to have to split the image up into multiple ROIs/sub-images to avoid fragmentation, or write your own allocator that allocates the image in memory in smaller chunks.

But i can do this cv::Mat image = Mat::zeros(15000, 100000, CV_8UC3); cv::imwrite("reallyreallyBigImage2.png", image); I can create this image and write it to my file system.
Mat::zeros does not use Xmm extensions when setting allocated memory to 0, as cv::imread does. Writing to disk is also an entirely different process than reading into memory. Not only does OpenCV have to load the PNG data into memory, it must decode it as well, which takes up more memory.
Also, try using CV_8UC4, which takes up 33% more memory than CV_8UC3. PNG files are generally loaded with an alpha channel.

Geordie · Accepted Answer · 2020-02-11 05:43:29Z

May I commend libvips to you if you are dealing with large images? It is available for Linux, macOS and Windows - for free, from here.

If you have a 100,000 x 15,000 PNG image and you want to extract a region 2000x1000 starting at a point (100,100), you would do:

time vips crop test.png excerpt.png 2000 1000 100 100 --vips-leak

Sample Output

memory: high-water mark 192.29 MB real 0m1.771s user 0m0.826s sys 0m0.082s

As you can see, it only used 200MB of RAM and took around 1s. The time and --vips-leak were for demonstration purposes only - you can omit them normally.

Collectives™ on Stack Overflow

Can not load large image with opencv

3 Answers 3

1 Comment

3 Comments

Comments

Linked

Hot Network Questions