How do I stop sed from adding extra newline characters

Question

I am running the following 2 sed commands. The first one adds newline characters where I want them, the second also adds newline characters where I want them, BUT also adds an extra one at the end of the file where there wasn't one before.

sed -e 's|\<LIST_G_STATEMENT>|&\ |g' ${XMLDIR}/statement_tmp_1.xml > ${XMLDIR}/statement_tmp_2.xml sed -e 's|\</LIST_G_STATEMENT>|&\ |g' ${XMLDIR}/statement_tmp_2.xml > ${XMLDIR}/statement_tmp_3.xml

Using od -c on all 3 of the files gives the following output.

statement_tmp_1.xml (no \n at end of file)

1314700 T A T E M E N T > < / L I S T _ 1314720 G _ S T A T E M E N T > < / G _ 1314740 S E T U P > < / L I S T _ G _ S 1314760 E T U P > < / A R X S G P O > 1314777

statement_tmp_2.xml (no \n at end of file)

1314700 S T A T E M E N T > < / L I S T 1314720 _ G _ S T A T E M E N T > < / G 1314740 _ S E T U P > < / L I S T _ G _ 1314760 S E T U P > < / A R X S G P O > 1315000

statement_tmp_3.xml (\n at end of file - where did it come from?)

1314700 S T A T E M E N T > < / L I S T 1314720 _ G _ S T A T E M E N T > \n < / 1314740 G _ S E T U P > < / L I S T _ G 1314760 _ S E T U P > < / A R X S G P O 1315000 > \n 1315002

I am running AIX 5.3

Basically, I either want it to stop adding the extra \n, or find a way of removing it.

Just a question: why are you using a literal newline in your substitution pattern when you could have used s|...|&\n| just as well? — Joseph R.
– Joseph R., Commented Nov 4, 2013 at 10:59
A file which doesn't end in a newline character is not a text file, so the behaviour with text utilities on them is unspecified. Use perl or other tool that can deal with binary data. — Stéphane Chazelas
– Stéphane Chazelas, Commented Nov 4, 2013 at 11:06
@JosephR. No, \<LF> is the traditional and POSIX way to add a LF character. \n would typically substitute a n character in anything but GNU sed. — Stéphane Chazelas
– Stéphane Chazelas, Commented Nov 4, 2013 at 11:07

Stéphane Chazelas · Accepted Answer · 2013-11-04 13:50:27Z

You should consider yourself lucky that AIX sed added that missing newline characters.

A non-empty file that doesn't end in a newline character is not a text file (at least as per the POSIX definition of a text file) as a text file is meant to contain lines and lines are a (not-too-long) sequence of characters terminated by a newline character, so the behaviour of text utilities like sed on it is unspecified and in practice varies from implementation to implementation.

Some sed implementation would have dismissed those spurious character after the last line.

AFAIK, xml files are meant to be text files, so that means sed just fixed it for you.

If you do need that file not to end in a newline character, then you could use perl or other tools that can cope with non-text data.

perl -pe 's|<LIST_G_STATEMENT>|$&\n|g'

The terminating newline is helpful, if you expect to pipe your sed output into any other standard Unix utility. Honestly, I didn't notice sed did this for years, since Bourne shell command substitutions like $(sed 's/bas/replac/' <<<'basement') furtively trim the final newline, if there is one. But there are times when you definitely don't want it; e.g., manipulating X clipboard text with sed. FYI, GNU sed, if available, does not add a terminating newline if you use p it with the -n option, as described in this SE answer. — Kevin E
– Kevin E, Commented Dec 13, 2019 at 22:23

chan · Accepted Answer · 2013-11-29 14:31:57Z

Here's a way to remove the final newline from a file using dd:

printf "" | dd of='/path/to/file' seek=<filesize_in_bytes - 1> bs=1 count=1

To test whether a file ends with a newline you could use:

tail -c 1 /path/to/file | tr -dc '\n' | wc -c

And to get the file size in bytes use:

wc -c < /path/to/file

mikeserv · Accepted Answer · 2014-05-21 00:52:07Z

According to this AIX manual IBM's tail does -reverse - which looks pretty cool. So long as your file is under 20KB the following should work:

tail -r <file | dd bs=1 skip=1 | tail -r >file.new

Stack Exchange Network

How do I stop sed from adding extra newline characters

3 Answers 3

You must log in to answer this question.

Linked

Hot Network Questions

How do I stop sed from adding extra newline characters

3 Answers 3

You must log in to answer this question.

Linked

Related

Hot Network Questions