3

I was in the need to move files with a aws-lambda from a SFTP server to my AWS account, then I've found this article:

https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/

Talking about paramiko as a SSHclient candidate to move files over ssh.

Then I've written this calss wrapper in python to be used from my serverless handler file:

import paramiko import sys class FTPClient(object): def __init__(self, hostname, username, password): """ creates ftp connection Args: hostname (string): endpoint of the ftp server username (string): username for logging in on the ftp server password (string): password for logging in on the ftp server """ try: self._host = hostname self._port = 22 #lets you save results of the download into a log file. #paramiko.util.log_to_file("path/to/log/file.txt") self._sftpTransport = paramiko.Transport((self._host, self._port)) self._sftpTransport.connect(username=username, password=password) self._sftp = paramiko.SFTPClient.from_transport(self._sftpTransport) except: print ("Unexpected error" , sys.exc_info()) raise def get(self, sftpPath): """ creates ftp connection Args: sftpPath = "path/to/file/on/sftp/to/be/downloaded" """ localPath="/tmp/temp-download.txt" self._sftp.get(sftpPath, localPath) self._sftp.close() tmpfile = open(localPath, 'r') return tmpfile.read() def close(self): self._sftpTransport.close()

On my local machine it works as expected (test.py):

import ftp_client sftp = ftp_client.FTPClient( "host", "myuser", "password") file = sftp.get('/testFile.txt') print(file)

But when I deploy it with serverless and run the handler.py function (same as the test.py above) I get back the error:

 Unable to import module 'handler': No module named 'paramiko'

Looks like the deploy is unable to import paramiko (by the article above it seems like it should be available for lambda python 3 on AWS) isn't it?

If not what's the best practice for this case? Should I include the library into my local project and package/deploy it to aws?

Improve this question
1
  • 1
    Last I saw paramiko is not included in Lambda. I refer this link for the list of available packages: gist.github.com/gene1wood/4a052f39490fae00e0c3. Also, step # 5 indicates that you need to create a virtualenv, package all the dependencies (inluding paramiko) and upload it to the lambda function. Commented Aug 22, 2018 at 18:36

6 Answers 6

Reset to default
3

A comprehensive guide tutorial exists at :

https://serverless.com/blog/serverless-python-packaging/

Using the serverless-python-requirements package as serverless node plugin.

Creating a virtual env and Docker Deamon will be required to packup your serverless project before deploying on AWS lambda

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

3

In the case you use 

custom: pythonRequirements: zip: true

in your serverless.yml, you have to use this code snippet at the start of your handler

try: import unzip_requirements except ImportError: pass

all details possible to find in Serverless Python Requirements documentation

Improve this answer

Comments

1

You have to create a virtualenv, install your dependencies and then zip all files under sites-packages/

sudo pip install virtualenv virtualenv -p python3 myvirtualenv source myvirtualenv/bin/activate pip install paramiko cp handler.py myvirtualenv/lib/python zip -r myvirtualenv/lib/python3.6/site-packages/ -O package.zip

then upload package.zip to lambda

Improve this answer

1 Comment

I like this example, it just needs some more info for the use-case and machine. For me, I use macOS Catalina 10.15.7, and the command works like this: zip -r package.zip /Users/me/.local/share/virtualenvs/lambda-dGk8CRLc/lib/python3.8/site-packages/
0

You have to provide all dependencies that are not installed in AWS' Python runtime.

Take a look at Step 7 in the tutorial. Looks like he is adding the dependencies from the virtual environment to the zip file. So I'd assume your ZIP file to contain the following:

  • your worker_function.py on top level
  • a folder paramico with the files installed in virtual env

Please let me know if this helps.

Improve this answer

Comments

0

I tried various blogs and guides like:

  1. web scraping with lambda
  2. AWS Layers for Pandas

spending hours of trying out things. Facing SIZE issues like that or being unable to import modules etc.

.. and I nearly reached the end (that is to invoke LOCALLY my handler function), but then my function even though it was fully deployed correctly and even invoked LOCALLY with no problems, then it was impossible to invoke it on AWS.

The most comprehensive and best by far guide or example that is ACTUALLY working is the above mentioned by @koalaok ! Thanks buddy!

actual link

Improve this answer

1 Comment

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review
0

The same happened to me as well. Paramiko is not working for higher versions of Python. This is only working for version 3.7. Try to edit your lambda run-time configuration to 3.7 and it will work.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.