1

I have a complex command I am passing via ssh to a remote server. I am trying to unzip a file and then change its naming structure and extension in a second ssh command. The command I have is:

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print $1$3".log"}'"

Obviously the " around the .log portion of the print statement are failing me. The idea is that I would strip the .out portion from the filename and end up with file20171119.log as an ending result. I am just a bit confused on the syntax or on how to escape that properly so bash interprets the .log appropriately.

Improve this question
12
  • Do you even need to expand the archive? If all you are doing is working with the file name, bash parameter expansion can handle that. If f=/tmp/file.out-20171119.gz, then g=${f/.out-/}; g=${g#/tmp/}; g=${g/%.gz/.log} should do it. Commented Nov 29, 2017 at 17:49
  • There is a .out file in the archive. That is the file I am trying to rename basically. Would that still apply? Commented Nov 29, 2017 at 17:51
  • Same approach, yes. Different value of f, of course, and you may need to adjust the exact patterns I used the the assignments to g. Commented Nov 29, 2017 at 17:52
  • Is there a reason you're passing this on the command line rather than stdin? Commented Nov 29, 2017 at 17:54
  • BTW, are you actually wanting to cat that file into awk? Because that's not what your script does at all. echo file* is echoing the names of files into awk. Commented Nov 29, 2017 at 17:55

3 Answers 3

Reset to default
5

The easiest way to deal with this problem is to avoid it. Don't bother trying to escape your script to go on a command line: Pass it on stdin instead.

ssh root@server1 bash -s <<'EOF' gzip -d /tmp/file.out-20171119.gz # note that (particularly w/o a cd /tmp) this doesn't do anything at all related to the # line above; thus, probably buggy as given in the original question. echo file* | awk -F'[.-]' '{print $1$3".log"}' EOF

A quoted heredoc -- one with <<'EOF' or <<\EOF instead of <<EOF -- is passed literally, without any shell expansions; thus, $1 or $3 will not be replaced by the calling shell as they would with an unquoted heredoc.

If you don't want to go the avoidance route, you can have the shell do the quoting for you itself. For example:

external_function() { gzip -d /tmp/file.out-20171119.gz echo file* | awk -F'[.-]' '{print $1$3".log"}' } ssh root@server1 "$(declare -f external_function); external_function"

declare -f prints a definition of a function. Putting that function literally into your SSH command ensures that it's run remotely.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

This seems ideal, but for some reason bash is complaining about that EOF. It says wanted `EOF' followed by an unexpected end of file.
I'd need to see the code exactly as it's run to diagnose that. If the final EOF (terminating the heredoc) is indented, for example, there's your problem; it has to be at the very beginning of the line that it's on.
@user1943674, ...which is to say, please verify that you get a bug with my code exactly as it's given here, without making any changes whatsoever (except for the name of server1).
Definitely no bug. It was a spacing issue. Is it possible to also pass in a variable into the shell command? I would like to have the gzip command be run as gzip -d /tmp/$file
Yes, it is, and we have questions elsewhere in the knowledgebase asking and answering that question; let me find one...
|
1

You need to escape the " to prevent them from closing your quoted string early, and you need to escape the $ in the awk script to prevent local parameter expansion.

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
Improve this answer

3 Comments

I get an error now: bash -c unexpected EOF while looking for matching `'' and another error: syntax error: unexpected end of file
Double-check your quoting; works for me (at least, when I replace gzip with a variable assignment and echo that value instead of file* to awk).
probably you left an open quote, that made to skip over the proper EOF string.
0

The most probable reason (as you don't show the contents of the root home directory in the server) is that you are uncompressing the file in the /tmp directory, but feeding to awk filenames that should exist in the root home directory.

" allows escaping sequences with \. so the correct way to do is 

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"

(like you wrote in your question) this means the following command is executed with a shell in the server machine.

gzip -d /tmp/file.out-20171119.gz; echo file* | awk - F'[.-]' '{print $1$3".log"}'

You are executing two commands, the first to gunzip /tmp/file.out-2017119.gz (beware, as it will be gunzipped in /tmp). And the second can be the source for the problem. It is echoing all the files in the local directory (this is, the root user home directory, probably /root in the server) that begin with file in the name (probably none), and feeding that to the next awk command.

As a general rule.... test your command locally, and when it works locally, just escape all special characters that will go unescaped, after being parsed by the first shell.

another way to solve the problem is to use gzip(1) as a filter... so you can decide the name of the output file

ssh root@server1 "gzip -d </tmp/file.out-20171119.gz >file20171119.log"

this way you save an awk(1) execution just to format the output file. Or if you have the date from an environment variable.

DATE=`date +%Y%m%d` ssh root@server1 "gzip -d </tmp/file.out-${DATE}.gz >file${DATE}.log"

Finally, let me give some advice: Don't use /tmp to uncompress files. /tmp is used by several distributions as a high speed temporary dir. It is normally ram based, too quick, but limited space, so uncompressing a log file there can fill up the memory of the kernel used for the ram based filesystem, which is not a good idea. Also, a log file normally expands a lot and /tmp is a local system general directory, where other users can store files named file<something> and you can clash with those files (in case you do searches with wildcard patterns, like you do in your command) Also, it is common once you know the name of the file to assign it to environment variables and use those variables, so case you need to change the format of the filename, you do it in only one place.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.