1

I need to scp the files from machineB and machineC to machineA. I am running my below shell script from machineA. I have setup the ssh keys properly.

If the files are not there in machineB, then it should be there in machineC.

#!/bin/bash readonly PRIMARY=/data01/primary/. readonly FILERS_LOCATION=(machineB machineC) readonly MAPPED_LOCATION=/bat/data/snapshot PARTITION=(0 3 5 7 9) dir1=20131222 dir2=20131222 scp david@${FILERS_LOCATION[0]}:$dir1/weekly_1980_[$el]_200003_5.data $PRIMARY || scp david@${FILERS_LOCATION[1]}:$dir2/weekly_1980_[$el]_200003_5.data $PRIMARY

Now my question is if you take a look into my above scp command, I have [$el], I need to replace this with PARTITION, which means I need to iterate PARTITION and replace [$el] with each number in PARTITION.

So my scp command should look like this if I iterate PARTITION - 

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/. scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/. scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/. scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/. scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/.
  1. How do I iterate PARTITION in such a way, so that I can make the above SCP command?
  2. And as you can see, I am copying the files one by one into machineA /data01/primary/ folder. Is there any better way of doing this? Meaning, can I copy all the files together in one shot instead of doing it one by one?
Improve this question

3 Answers 3

Reset to default
1

To copy all the partitions, use a wildcard:

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_*_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_*_200003_5.data /data01/primary/.

If you to copy only partitions 0, 3, 5, 7 , 9, then use:

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_[03579]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_[03579]_200003_5.data /data01/primary/.

If the partition list is specified by a variable, the simplest would be:

PARTITION=03579 scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_[$PARTITION]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_[$PARTITION]_200003_5.data /data01/primary/.

If it is specified in an array:

PARTITION=(0 3 5 7 9) oldIFS="$IFS" IFS="" scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_["${PARTITION[*]}"]_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_["${PARTITION[*]}"]_200003_5.data /data01/primary/. IFS="$oldIFS"
Improve this answer
7
  • Thanks John, I cannot use wild card here because I need to copy only those files that are there in the PARTITION list only. In that case, what change I am supposed to make in my above shell script? Commented Dec 24, 2013 at 6:42
  • @TechGeeky Do you need to specify PARTITION as a bash array? Could you specify it just as easily as PARTITION=03579 ? Commented Dec 24, 2013 at 6:48
  • Actually that partition has to be passed by Python script. So currently I am in the impression that I will be passing in the form of bash arrays and I haven't decided yet.. And in general number of elements in the bash arrays will be 420... And also if I need to do it one by one, then how I should do that as well? Commented Dec 24, 2013 at 6:56
  • Python has much better looping and string handling constructs than bash. If this is all being controlled by a python script, I would have python create the scp command and feed it to subprocess for execution. Commented Dec 24, 2013 at 7:04
  • Thanks a lot John. I will try to see whether I should do all these things in Python or not. And also, can you explain little bit on your solution for array? You are using IFS something which I am not aware of.. Commented Dec 24, 2013 at 7:12
1 
#!/bin/bash readonly PRIMARY=/data01/primary/. readonly FILERS_LOCATION=(machineB machineC) readonly MAPPED_LOCATION=/bat/data/snapshot for i in 0 3 5 7 9 do scp david@${FILERS_LOCATION[0]}:$dir1/weekly_1980_${i}_200003_5.data PRIMARY || scp david@${FILERS_LOCATION[1]}:$dir2/weekly_1980_${i}_200003_5.data $PRIMARY done
Improve this answer
1

To efficiently copy many files at once, use rsync instead of scp. Rsync uses the SSH connection to transfer files remotely.

rsync_filter=() for el in "${PARTITION[@]}"; do rsync_filter+=(--include="weekly_1980_${el}_200003_5.data") done for machine in "${FILERS_LOCATION[@]}"; do rsync -a "${rsync_filter[@]}" --exclude='*' "david@$machine:$dir1/" "$PRIMARY/" done

Rsync will skip copying if the files are already present with the same size and modification time (you can configure this).

Alternatively, forget all about the fact that they're remote copies. Mount the remote filesystem with SSHFS, and proceed as you would copy local files.

mkdir machineB machine C sshfs david@machineB: machineB sshfs david@machineC: machineC for el in "${PARTITION[@]}"; do cp -p "machineB/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/" || cp -p "machineC/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/" done

You can even do one better and make a union mount of the two machines. A union mount joins two directory trees into one, such that accessing a file from the union tree tries inside one tree first, and if the file is not present tries the same location in the other tree instead.

mkdir union mount -t aufs -o dirs=$PWD/machineB:$PWD/machineC machineB+machineC union for el in "${PARTITION[@]}"; do cp -p "union/$dir1/weekly_1980_${el}_200003_5.data" "$PRIMARY/" done
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.