1

I have a program, which collects various log-files from multiple computers into a single ZIP-file on the client machine. It is currently using Fabric and I'd like to rewrite it with AsyncSSH to make all the servers send their files in parallel.

What I'm trying to figure out is getting the files over -- as streams. My current Fabric-based code is:

 result = io.BytesIO() connection.get(remote = path, local = result) result.seek(0) return result

What's the equivalent for when the connection is a result of asyncssh.connect()?

Improve this question
3
  • From what I see, asyncssh.connect returns an object which allows you to execute a command, e.g. echo in the example. So just execute a cat of the log file. You can fire multiple requests in parallel using asyncio.gathet or .wait (see here) Commented Dec 27, 2020 at 6:50
  • 1
    I mean, for each source host you can do result = io.StringIO(await conn.run('cat logfile"), but in parallel using gather (assuming your logfiles are text and aren't huge) Commented Dec 27, 2020 at 6:58
  • I was hoping for some channel/stream kind of semantics -- so that, if a particular file is large, it does not need to be read fully into client's RAM, but can be read -- and compressed by zipfile -- in chunks... Commented Dec 27, 2020 at 19:58

1 Answer 1

Reset to default
2

I think you're looking for create_process() instead of run(). That will return you an SSHClientProcess object that has stdin/stdout/stderr classes which are SSHReader and SSHWriter objects that behave similar to asyncio's StreamReader and StreamWriter.

Alternately, if "connection.get" there is actually doing an SFTP file transfer, I think you want to look into use open() on the SFTPClient object, rather than get(). That would give you a file object you could call read() on. With SFTP you normally want to schedule several reads from the same file in parallel to improve speed, though, and so the read() calls take an offset and a size rather than treating it like a stream that you sequentially access.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks. Could you give some sample code for both options?
You can find an example of interactive input using create_process() at asyncssh.readthedocs.io/en/latest/#interactive-input. I'm still a bit unclear on exactly what you are trying to do, though, so it's hard to be more specific. I'm not really familiar with Fabric -- what exactly is the "connection.get" call doing?
If what you're doing is more file transfer related, asyncssh.readthedocs.io/en/latest/#sftp-client provides an example of how to retrieve a remove file into a local file in a single call. If you want to read data from that file interactively rather than writing it to disk on the local machine, you'd replace the "sftp.get" call with an "sftp.open" call and then call read(), seek(), or other such calls on the file object which open() returns.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.