11

I've been trying to get OpenCV into an S3 bucket and then assign it to a lambda layer.

Theres very little about this online and what I have seen hasn't worked.

I've managed to use docker with the amazon linux environment, and followed this tutorial. https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

I've added setuptools, wheel and opencv-python==4.4.0.42 to the requirements.txt file.

setuptools and wheel because of an earlier error where the recommendation was to include these as they need updating, even though I have updated them. But it works with them, so who knows.

Created the docker image which I've zipped and put in an S3 bucket.

I keep getting { "errorMessage": "Unable to import module 'lambda_function': libGL.so.1: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError" } when I run it though.

I can't seem to figure out what is wrong.

Any ideas?

Improve this question

4 Answers 4

Reset to default
29

You will need to add a bunch of dependencies to your layer. Below are the steps that I've used for opencv_python on lambda.

1. On local workstation (terminal window 1)

mkdir /tmp/mylayer && cd /tmp/mylayer echo opencv-python==4.4.0.42 > ./requirements.txt

2. On local workstation (terminal window 2)

 docker run -it -v /tmp/mylayer:/mylayer lambci/lambda:build-python3.8 bash

The above command will put you into the docker container.

Inside the container:

cd /mylayer pip install --no-deps -t python/lib/python3.8/site-packages/ -r requirements.txt yum install -y mesa-libGL cp -v /usr/lib64/libGL.so.1 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGL.so.1.7.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libgthread-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libgthread-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libglib-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGLX.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libX11.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libXext.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGLdispatch.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGLESv1_CM.so.1.2.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGLX_mesa.so.0.0.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libGLESv2.so.2.1.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libxcb.so.1 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libXau.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /usr/lib64/libXau.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/ cp -v /lib64/libGLdispatch.so.0.0.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/

3. On local workstation again (terminal window 1)

Pack the python folder into mylayer.zip.

zip -r -9 mylayer.zip python

In AWS console

  1. Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

  2. Add AWS provide SciPy layer AWSLambda-Python38-SciPy1x and your own layer with cv2 into your function.

So you will have two layers in your function.

  1. Perform basic test of the layer in lambda using the following lambda function:
import cv2 def lambda_handler(event, context): print(dir(csv))

The function executes correctly (partial printout shown).

slation3D', 'exp', 'extractChannel', 'fastAtan2', 'fastNlMeansDenoising', 'fastNlMeansDenoisingColored', 'fastNlMeansDenoisingColoredMulti', 'fastNlMeansDenoisingMulti', 'fillConvexPoly', 'fillPoly', 'filter2D', 'filterHomographyDecompByVisibleRefpoints', 'filterSpeckles', 'find4QuadCornerSubpix', 'findChessboardCorners', 'findChessboardCornersSB', 'findChessboardCornersSBWithMeta', 'findCirclesGrid', 'findContours', 'findEssentialMat', 'findFundamentalMat', 'findHomography', 'findNonZero', 'findTransformECC', 'fisheye', 'fitEllipse', 'fitEllipseAMS', 'fitEllipseDirect', 'fitLine', 'flann', 'flann_Index', 'flip', 'floodFill', 'gemm', 'getAffineTransform', 'getBuildInformation', 'getCPUFeaturesLine', 'getCPUTickCount', 'getDefaultNewCameraMatrix', 'getDerivKernels', 'getFontScaleFromHeight', 'getGaborKernel', 'getGaussianKernel', 'getHardwareFeatureName', 'getNumThreads', 'g
Improve this answer
Sign up to request clarification or add additional context in comments.

13 Comments

Marcin, I love you!! You have no idea how many days I've been struggling with this. Just used this and it worked without any errors. Thank you!
OMG you are my new best friend!
The libGLES*.so.* files didn't get installed for me, but it seems to work without them.
Thank you so much!! I spent days trying to figure this out!!!
I spent three hours on the three garbage tutorials at the top of the Google search rankings and this worked instantly. Thanks so much! You should turn this into a blog post so that it drowns out the other nonsense out there wasting countless hours of people's lives.
|
4

Although this solution does work (adding dependency layers), this doesn't get around the hard size limit of 250MB. Luckily AWS have added support for Lamda's to run container images - see my answer here: How to increase the maximum size of the AWS lambda deployment package (RequestEntityTooLargeException)?

Improve this answer

Comments

2

To avoid the error

Unable to import module 'lambda_function': libGL.so.1: cannot open shared object file: No such file or directory" error,

Install the

opencv-python-headless

instead of

opencv-python

The lambda is throwing this error about LibGL because lambda functions does not support any UI and opencv comes with some UI stuff. The "headless" module does not include the GUI that comes with opencv, which resolves the error you see.

Check out these videos:

  1. how to create opencv layers with a ubuntu ec2 instance https://www.youtube.com/watch?v=FQBT8vVRkAg&ab_channel=SrceCde

  2. how to solve the LibGL problem. https://www.youtube.com/watch?v=Y2rWypy8OaM&ab_channel=SrceCde

Improve this answer

1 Comment

This worked for me, installing opencv on a Lamda function with Docker and Python 3.10
0

using lambci/lambda:build-python3.6 instead of lambci/lambda:build-python3.8 does fix the libGL.so.1: cannot open shared object file error for me since build-python3.8 reduce its size by removing some native dependencies

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.