30

I have files that end in one or more newlines and should end in only one newline. How can I do that with Bash/Unix/GNU tools?

Example bad file:

1\n \n 2\n \n \n 3\n \n \n \n

Example corrected file:

1\n \n 2\n \n \n 3\n

In other words: There should be exactly one newline between the EOF and the last non-newline character of the file.

Reference Implementation

Read file contents, chop off a single newline till there no further two newlines at the end, write it back:

#! /bin/python import sys with open(sys.argv[1]) as infile: lines = infile.read() while lines.endswith("\n\n"): lines = lines[:-1] with open(sys.argv[2], 'w') as outfile: for line in lines: outfile.write(line)

Clarification: Of course, piping is allowed, if that is more elegant.

Improve this question

9 Answers 9

Reset to default
32

From useful one-line scripts for sed.

# Delete all trailing blank lines at end of file (only). sed -e :a -e '/^\n*$/{$d;N;};/\n$/ba' file
Improve this answer
3
  • 6
    Thanks, I used the following to do it in place for multiple files: find . -type f -name '*.js' -exec sed --in-place -e :a -e '/^\n*$/{$d;N;};/\n$/ba' {} \; Commented Nov 22, 2013 at 9:48
  • @jakub.g in place and recursive is exactly what I needed. thank you. Commented Dec 13, 2015 at 10:41
  • 1
    To add to the excellent comment from @jakub.g you can invoke the command like this on OS X: find . -type f -name '*.js' -exec sed -i '' -e :a -e '/^\n*$/{$d;N;};/\n$/ba' {} \; Commented Feb 19, 2018 at 18:35
21 
awk '/^$/ {nlstack=nlstack "\n";next;} {printf "%s",nlstack; nlstack=""; print;}' file
Improve this answer
6
  • 4
    +1: awk's solutions are (almost) always elegant and readable! Commented Jul 4, 2013 at 9:37
  • @OlivierDulac Indeed. When I saw the sed proposal I just thought OMG... Commented Jul 4, 2013 at 11:32
  • 1
    this doesn't work on OSX Mavericks using the latest available awk from Homebrew. It errors with awk: illegal statement. brew install mawk and changing the command to mawk works though. Commented May 9, 2014 at 5:02
  • @noname I don't even understand the question... Commented Oct 16, 2018 at 20:22
  • 1
    +1. Add BEGINFILE {nlstack=""} if operating on multiple files. Commented Mar 27, 2022 at 20:49
19

Since you already have answers with the more suitable tools sed and awk; you could take advantage of the fact that $(< file) strips off trailing blank lines.

a=$(<file); printf '%s\n' "$a" > file

That cheap hack wouldn't work to remove trailing blank lines which may contain spaces or other non-printing characters, only to remove trailing empty lines. It also won't work if the file contains null bytes.

In shells other than bash and zsh, use $(cat file) instead of $(<file).

Improve this answer
5
  • +1 to point out what looks like a bug to me : $(<file) isn't really reading the file? why does it discard trailing newlines? (it does, i just tested, thanks for pointing it out!) Commented Jul 4, 2013 at 9:23
  • 3
    @OlivierDulac $() discards trailing newlines. That's a design decision. I assume that this shall make the integration in other strings easier: echo "On $(date ...) we will meet." would be evil with the newline that nearly every shell command outputs at the end. Commented Jul 4, 2013 at 11:31
  • @HaukeLaging: good point, it's probably the source of that behaviour Commented Jul 4, 2013 at 12:13
  • I added a special case to avoid appending "\n" to empty files: [[ $a == '' ]] || printf '%s\n' "$a" >"$file". Commented Apr 14, 2014 at 19:11
  • To strip multiple newlines off the start of a file, insert tac into the process (I use gnu coreutils on Mac, so gtac for me) : a=$(gtac file.txt); printf '%s\n' "$a" | gtac > file.txt Commented Oct 3, 2018 at 12:53
5

You can use this trick with cat & printf:

$ printf '%s\n' "`cat file`"

For example

$ printf '%s\n' "`cat ifile`" > ofile $ cat -e ofile 1$ $ 2$ $ $ 3$

The $ denotes the end of a line.

References

Improve this answer
0
5

This question is tagged with , but nobody has proposed an ed solution.

Here's one:

ed -s file <<'ED_END' a . ?.?+1,$d w ED_END

or, equivalently,

printf '%s\n' a '' . '?.?+1,$d' w | ed -s file

ed will place you at the last line of the editing buffer by default upon startup.

The first command (a) adds an empty line to the end of the buffer (the empty line in the editing script is this line, and the dot (.) is just for coming back into command mode).

The address of the second command (?.?) looks for the nearest previous line that contains something (even white-space characters), and then deletes (d) everything to the end of the buffer from the next line on.

The third command (w) writes the file back to disk.

The added empty line protects the rest of the file from being deleted in the case that there aren't any empty lines at the end of the original file.

Improve this answer
0
4

Here's a Perl solution that doesn't require reading more than one line into memory at a time:

my $n = 0; while (<>) { if (/./) { print "\n" x $n, $_; $n = 0; } else { $n++; } }

or, as a one-liner:

perl -ne 'if (/./) { print "\n" x $n, $_; $n = 0 } else { $n++ }'

This reads the file a line at a time and checks each line to see if contains a non-newline character. If it doesn't, it increments a counter; if it does, it prints the number of newlines indicated by the counter, followed by the line itself, and then resets the counter.

Technically, even buffering a single line in memory is unnecessary; it would be possible to solve this problem using a constant amount of memory by reading the file in fixed-length chunks and processing it character by character using a state machine. However, I suspect that would be needlessly complicated for the typical use case.

Improve this answer
2

If your file is small enough to slurp into memory, you can use this

perl -e 'local($/);$f=<>; $f=~s/\n*$/\n/;print $f;' file
Improve this answer
0

In python (I know it is not what you want, but it is much better as it is optimized, and a prelude to the bash version) without rewriting the file and without reading all the file (which is a good thing if the file is very large):

#!/bin/python import sys infile = open(sys.argv[1], 'r+') infile.seek(-1, 2) while infile.read(1) == '\n': infile.seek(-2, 1) infile.seek(1, 1) infile.truncate() infile.close()

Note that it does not work on files where the EOL character is not '\n'.

Improve this answer
0

A bash version, implementing the python algorithm, but less efficient as it needs many processes:

#!/bin/bash n=1 while test "$(tail -n $n "$1")" == ""; do ((n++)) done ((n--)) truncate -s $(($(stat -c "%s" "$1") - $n)) "$1"
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.