3

Summary

Is xargs -I s printf s more compatible than xargs -n 1 printf?

Background

To handle binary data that may include 0x00. I know how to convert binary data to text, like this:

# make sure that you have done this: export LC_ALL=C od -A n -t x1 -v | # or -t o1 or -t u1 or whatever tr ABCDEF abcdef | # because POSIX doesn't specify in which case tr -d ' \t\n' | # because POSIX says they are delimiters fold -w 2 | grep . # to make sure to terminate final line with LF

... and here is how to convert back to binary:

# input: for each line, /^[0-9a-f]\{2\}$/ # also make sure export LC_ALL=C before awk -v _maxlen="$(getconf ARG_MAX 2>/dev/null)" ' BEGIN{ # (1) make a table # assume that every non-null byte can be converted easily # actually not portable in Termux; LC_ALL=C does not work and # awk is gawk by default, which depends on locale. # to deal with it, here is alternative: # for(i=0;i<256;i++){ # xc[sprintf("%02x",i)]=sprintf("\\\\%03o",i); # xl[sprintf("%02x",i)]=5; # } # # and skip to (2) # but why not just env -i awk to force one true awk, if so. # also is not it pretty rare that C locale is not available? for(i=1;i<256;i++){ xc[sprintf("%02x",i)]=sprintf("%c",i); xl[sprintf("%02x",i)]=1; } # now for chars that requires special converting. # numbers; for previous char is \\ooo. for(i=48;i<58;i++){ xc[sprintf("%02x",i)]=sprintf("\\\\%03o",i); xl[sprintf("%02x",i)]=5; } # and what cannot be easily passed to xargs -n 1 printf # null xc["00"]="\\\\000"; xl["00"]=5; # <space> xc["09"]="\\\\t"; xl["09"]=3; xc["0a"]="\\\\n"; xl["0a"]=3; xc["0b"]="\\\\v"; xl["0b"]=3; xc["0c"]="\\\\f"; xl["0c"]=3; xc["0d"]="\\\\r"; xl["0d"]=3; xc["20"]="\\\\040"; xl["20"]=5; # meta chars for printf xc["25"]="%%"; xl["25"]=2; xc["5c"]="\\\\\\\\";xl["5c"]=4; # hyphen; to prevent to be treated as if it were an option xc["2d"]="\\\\055"; xl["2d"]=5; # chars for quotation xc["22"]="\\\""; xl["22"]=2; xc["27"]="\\'\''"; xl["27"]=2; # (2) preparation # reason why 4096: _POSIX_ARG_MAX # reason why length("printf "): because of ARG_MAX # reason why 4096/2 and _maxlen/2: because some xargs such as GNU specifies buffer length less than ARG_MAX if(_maxlen==""){ maxlen=(4096/2)-length("printf "); }else{ maxlen=int(_maxlen/2)-length("printf "); } ORS=""; LF=sprintf("\n"); arglen=0; } { # (3) actual conversion here. # XXX. not sure why arglen+4>maxlen. # but I think maximum value for xl[$0] is 5. # and maybe final LF is 1. if(arglen+4>maxlen){ print LF; arglen=0; } print xc[$0]; arglen+=xl[$0]; } END{ # for some xargs who hates input w/o LF termination if(NR>0)print LF; } ' | xargs -n 1 printf

I found an issue for null input: in GNU/Linux, it fails, like this:

$ xargs -n 1 printf </dev/null printf: missing operand Try 'printf --help' for more information.

Then I found xargs -n 1 printf 2>/dev/null || :, adding if(NR==0)printf"\"\"\n"; on END block, and xargs -I s printf s are alternatives. I have seen only the first one is actually used on ShellShoccar-jpn's programs, but I think it's kinda forceful. The second one is also less clean than the last one. Can the third one be also an alternative on not only GNU/Linux, but also every other (or most of the other) environment? Since I have GNU/Linux only, I have no ideas how to validate my idea on every other environment. The easiest way is to obtain their source and refer them, or refer to their manuals. If it is impossible to validate at all, then I have to give up.

My knowledge

  • It seems that printf requires at least one argument, as POSIX says so.
  • Some xargs ignores input without LF termination; grep ^ | xargs something here is more portable than xargs something here for input that may not have LF termination.
  • xargs is not portable for input without non-blank lines; printf ' \n\n' | xargs echo foo outputs nothing on FreeBSD and foo on GNU/Linux. In this case, you have to make the command for xargs safe for such input or let the command ignore the error.
  • FreeBSD's xargs receives its arguments as if they were $@ while GNU/Linux's as if they were "$@".
  • Escaping by backslash works for xargs, like printf '\\\\\\'"'" | sed "$(printf 's/[\047\042\\]/\\\\&/g')" | xargs printf to obtain \' as output.

PS

I found out that xargs -E '' is more compatible than without the option, as some xargs defaults -E _.

Improve this question
5
  • 1
    Also note that some xargs implementations have a very low limit on the size of a single argument. IIRC 255 bytes as allowed by POSIX. Commented Feb 27, 2021 at 7:50
  • What do you mean by in Termux; LC_ALL=C does not work? Commented Feb 27, 2021 at 8:00
  • Note that uuencode/uudecode are non-optional POSIX utilities (though maybe not available on some of the embedded systems you're trying to target). Commented Feb 27, 2021 at 8:04
  • 1
    @Stéphane POSIX locale is not provided; only en_US.UTF-8 is available there, as in this: <<github.com/termux/termux-packages/issues/5845>> Commented Feb 27, 2021 at 8:04
  • The C/POSIX locale is meant to be the default. LC_ALL=BONKERS is also meant to give on the C locale if the BONKERS locale doesn't exist. So in your termux case, it looks more like the C locale's charset is UTF-8 (there is an ongoing discussion on the POSIX (austing-group-l) mailing list as whether that's allowed and the general consensus is that it shouldn't) Commented Feb 27, 2021 at 8:11

1 Answer 1

Reset to default
5

xargs is probably the worst POSIX utility when it comes to portability (and interface design). I would stay away from it. How about:

<file.hex awk -v q="'" -v ORS= ' BEGIN{ for (i=0; i<256; i++) c[sprintf("%02x", i)] = sprintf("\\%o", i) } NR % 50 == 1 {print sep"printf "q; sep = q"\n"} {print c[$0]} END {if (sep) print q"\n"} ' | sh

instead for instance?

The awk part outputs something like:

printf '\61\12\62\12\63\12\64\12\65\12\66\12\67\12\70\12\71\12\61\60\12\61\61\12\61\62\12\61\63\12\61\64\12\61\65\12\61\66\12\61\67\12\61\70\12\61\71\12\62\60' printf '\12\62\61\12\62\62\12\62\63\12\62\64\12\62\65\12\62\66\12\62\67\12\62\70\12\62\71\12\63\60\12\63\61\12\63\62\12\63\63\12\63\64\12\63\65\12\63\66\12\63' printf '\67\12\63\70\12\63\71\12\64\60\12\64\61\12\64\62\12\64\63\12\64\64\12\64\65\12\64\66\12\64\67\12\64\70\12\64\71\12\65\60\12\65\61\12\65\62\12\65\63\12' printf '\65\64\12\65\65\12\65\66\12\65\67\12\65\70\12\65\71\12\66\60\12\66\61\12\66\62\12\66\63\12\66\64\12\66\65\12\66\66\12\66\67\12\66\70\12\66\71\12\67\60' printf '\12\67\61\12\67\62\12\67\63\12\67\64\12\67\65\12\67\66\12\67\67\12\67\70\12\67\71\12\70\60\12\70\61\12\70\62\12\70\63\12\70\64\12\70\65\12\70\66\12\70' printf '\67\12\70\70\12\70\71\12\71\60\12\71\61\12\71\62\12\71\63\12\71\64\12\71\65\12\71\66\12\71\67\12\71\70\12\71\71\12\61\60\60\12'

For sh to interpret. In sh implementations where printf is builtin, that would not fork extra processes. In those where it's not, those lines should be more than short enough to avoid the ARG_MAX limit but still not run more than one printf for every 50 bytes.

Note that you can't really determine the max length of a command line based on the value of ARG_MAX alone. How that limit is reached and handled depends greatly on the system and version thereof. On many, ARG_MAX is on the limit of a cumulative size including the argv[] and envp[] list of pointers (so typically 8 byte per argument/envvar on a 64bit system) plus the size of each arg/env strings in bytes (including the NUL delimiter). Linux also has an independent limit on the size of a single argument.

Also note that replacing \12 with \n for instance is only valid on ASCII-based systems. POSIX doesn't specify the encoding of characters (other than NUL). There are still POSIX systems using some variants of EBCDIC instead of ASCII.

Improve this answer
0

You must log in to answer this question.