5

I have provided my own answer below.

We have a number of bash scripts that work fine using SFTP, what I would like to do is to simply re-direct an ls or dir of a folder to a file on our SFTP Server..

We can run this as a cron job or I can run it manually.. I can view the list of files from the remote server, but I want to generate the list of files from the remote server to the local server in a text file..

Here is what I have for the bash script so far.. The fourth echo listed below is the line in question.

#!/bin/bash localpath=/home/localacct/somepath remotelocation=/home/account/logs/archive remotehost=' [email protected]' echo > $localpath/list.sftp echo "cd $remotelocation " >> $localpath/list.sftp echo "dir *.* " >> $localpath/list.sftp echo "dir *.* > $localpath/dirlist.txt " >> $localpath/list.sftp echo "bye " >> $localpath/list.sftp sftp -b $localpath/list.sftp $remotehost exit
Improve this question
5
  • So what is this script not doing right? Commented Jul 9, 2015 at 20:45
  • It is not writing the list (ls) of files to the text file. There is something wrong with the following: echo "dir . > $localpath/dirlist.txt " Commented Jul 9, 2015 at 20:46
  • 1
    Just curious and probably stating the obvious: you must use SFTP, right? This will not work for your use case? ssh $remotehost ls > $localpath/list.sftp Commented Jul 10, 2015 at 1:19
  • you should write you answer as separate answer and approve it. Commented Jul 13, 2015 at 14:04
  • I undeleted my previously posted Answer and is now visible. Thanks for the guidance. Commented Jul 13, 2015 at 14:14

8 Answers 8

Reset to default
6

The sftp command is very limited. If you can't make it do what you want, you can use another approach, which is to mount the remote directory with the SSHFS filesystem. SSHFS uses SFTP as the transport protocol, so the server side just sees an SFTP client. You need to be able to use FUSE on the client side.

With SSHFS, you mount the remote directory onto an existing, empty directory and use ordinary commands.

mkdir remote sshfs "$remotehost:$remotelocation" remote cd remote echo *.* >"$localpath/dirlist.txt" fusermount -u remote rmdir remote
Improve this answer
6

This works by far the best:

echo 'ls' | sftp hostname

You can forward the output into a file by

echo 'ls' | sftp hostname > /tmp/mylist.txt
Improve this answer
1
  • 4
    Returning one file per row should make the output easier to handle: echo 'ls -1a' | sftp hostname Commented May 6, 2019 at 21:22
2

The dir command within the sftp client does not support redirection. Example below, showing how it does nothing.

sftp> pwd Remote working directory: /var/tmp/foodir sftp> lcd /var/tmp/foodir sftp> dir *.* foo.txt sftp> dir *.* >dirlist.txt foo.txt sftp> dir foo.txt sftp>

Man page for sftp confirms.

ls [-1afhlnrSt] [path]

    Display a remote directory listing of either path or the current directory if path is not specified.  path may contain glob(3) characters and may match multiple files.

    The following flags are recognized and alter the behaviour of ls accordingly:

    -1

        Produce single columnar output.
    -a
        List files beginning with a dot (‘.’).
    -f
        Do not sort the listing. The default sort order is lexicographical.
    -h
        When used with a long format option, use unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte, Petabyte, and Exabyte in order to reduce the number of digits to four or fewer using powers of 2 for sizes (K=1024, M=1048576, etc.).
    -l
        Display additional details including permissions and ownership information.
    -n
        Produce a long listing with user and group information presented numerically.
    -r
        Reverse the sort order of the listing.
    -S
        Sort the listing by file size.
    -t
        Sort the listing by last modification time.

Improve this answer
1

in case anyone finds rsync doesn't work (for reports.paypal.com in my case) I found the following script using expect works:

#!/usr/bin/expect spawn sftp [email protected]:/ppreports/outgoing expect "password:" send "XXXXXXXXXX\n" expect "sftp>" log_file -noappend RemoteFileList.txt send "ls -1\n" expect "sftp>" log_file send "!sed -i '' '/ls -1/d' ./RemoteFileList.txt\n" expect "sftp>" send "!sed -i '' '/sftp>/d' ./RemoteFileList.txt\n" expect "sftp>" send "bye\n" interact

This is on Mac. On other systems the sed lines might do without the first '' arguments.

Improve this answer
1
  • I know this is old, but on the lines with sed why is there exclamations at the beginning of the command? I don't understand why that is necessary in this case? Commented Aug 24, 2023 at 17:17
1

You can also use lftp:

lftp -c 'connect sftp://user:password@host/dir; ls' > list.txt

Or

lftp -e 'ls;quit' sftp://user:password@host/dir > list.txt

lftp also has a find command for recursive listing.

Passing passwords on the command line is generally a bad idea. Alternative is to use ssh key authentication if possible (the password above will be ignored if key authentication is available, so you make it the empty string). Or you can pass it via the environment:

 LFTP_PASSWORD=password lftp --env-password -e 'ls;quit' \ sftp://user:password@host/dir > file.list

Or pass the commands via stdin, here using a here document (implemented with deleted tempfiles or pipes depending on the shell):

lftp <<'EOF' > file.list connect sftp://user:password@host/dir ls EOF
Improve this answer
0

There is another way which I wound up using.. Try using rsync.

We find the following:

  • If you specify no local destination then a listing of the specified files on the remote server is provided.

The BASH script then becomes:

#!/bin/bash localpath=/home/local-acct/path remotelocation=/home/account/logs/archive remotehost=' [email protected]' rsync -avz $remotehost:/$remotelocation > $localpath/dirlist.txt exit

and works great!

Improve this answer
1
  • 2
    Note that rsync uses ssh (so assumes the remote user has shell access), not sftp (a subsystem of ssh that does not require the user having shell access) and assumes rsync is installed on the remote machine. So you might as well do ssh user@host ls /dir > file Commented Jun 9, 2016 at 8:32
0

So you can use this method too (with different example) :

#!/bin/bash +x fct() { a=`sftp user@host<< EOL ls -la .. EOL` echo $a echo $a > tsftpres.000 } fct fct > tsftpres.001 fct_2() { sftp user@host << EOL ls -la .. EOL } fct_2 > tsftpres.002
Improve this answer
1
  • (1) I guess you’re trying to show variations on a theme, but this is a bit incoherent.  You demonstrate creating three files (one of them gets written twice).  Are they identical?  Is there any reason a user should prefer one of these commands over the others? (2) Why are you listing .. instead of remotelocation? (3) Have you tried this?  Does it really produce reasonable output?  I ask because I expect that echo "$a" would work much better than echo $a. (4) Does the name fct signify anything? Commented Apr 25, 2019 at 16:28
0

I can't add a comment but i post a new answer comment about my last answer.

Yes i know but it just an example of the method but i write this so quickly normally i prefer writing echo "${a}",i use ls -la .. because i wanted to test my sshd_config (ChrootDirectory) but i rewrite this better :

#!/bin/bash +x # with a varaiable or directly in file # Replace "user", "host" and filename with your value function list_sftp() { list=`sftp user@host << EOL ls -la EOL` echo "${list}" > tsftpres-000.txt } list_sftp function other_lsftp() { sftp user@host << EOL ls -la EOL } other_lsftp > tsftpres-001.txt
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.