5

TARGET

How to delete all lines in a text file before a matching one, including this one?

Input file example:

apple pear banana HI_THERE lemon coconut orange

Desired output:

lemon coconut orange

The aim is to do it in sed to use the "-i" option (direct editing).

CLEAN SOLUTION ?

Most answers for similar problems propose something like:

sed -n '/HI_THERE/,$p' input_file

But the matched line is not deleted:

HI_THERE lemon coconut orange

Then, knowing this will delete all from matched line (including it) to end of file:

sed '/HI_THERE/,$d' input_file

I tried something like this:

sed '^,/HI_THERE/d' input_file

But then sed complains:

sed: -e expression #1, char 1: unknown command: `^'

DIRTY SOLUTION

The last (dirty) solution is using pipeline:

sed -n '/HI_THERE/,$p' input_file | tail -n +2

but then, direct edit of the file doesn't work:

sed -n '/HI_THERE/,$p' input_file | tail -n +2 > input_file cat input_file # returns nothing

and one must use a temporary file like that...

sed -n '/HI_THERE/,$p' input_file | tail -n +2 > tmp_file mv tmp_file input_file
Improve this question
2
  • 2
    I think you wanted 1, not ^. Does that work for you? Commented Aug 2, 2019 at 8:00
  • Ach.... it works :) (please, publish this as an answer so that I can validate it :-) ) Commented Aug 2, 2019 at 8:02

6 Answers 6

Reset to default
6

Similar to your "clean solution":

sed -e '1,/HI_THERE/d' input_file

The first line in the file is line 1 - there's no special ^ address because you always know that, while $ is needed for the end because you don't (necessarily) know which line that is.

This does fall over if the matching line is the first line of the file. With GNU sed you can use 0 instead of 1 to deal with that. For POSIX sed and for portability (which seem to be different in this case) it's more complex (see comments below and this follow-up question).

Improve this answer
13
  • 2
    IIRC this will fail to match (hence deleting to the end of the file) if HI_THERE occurs in line 1 (in GNU sed, one can use 0,/HI_THERE/ to cover this special case) Commented Aug 2, 2019 at 8:08
  • POSIX-compatibly you can use sed -e '1{/HI_THERE/d;};1,//d' if that is an issue. Commented Aug 2, 2019 at 8:19
  • @MichaelHomer This POSIX version will not help, I'm afraid. All lines will get killed by the 1,//d anyhow. Commented Aug 2, 2019 at 8:43
  • 1
    I'm still not sure which is wrong, but as a minimal case printf 'a\nb\n'|sed -e '1{d;};1,/a/p' behaves how I expect on BSD/macOS sed and Solaris' POSIX sed (one output line), and not on GNU sed (two output lines), including in its POSIX mode. Busybox matches GNU sed. I will try to have a prod at more exotic systems. There is an argument for either interpretation of the specification text and I'm no longer certain which one I expect from it. Presumably Solaris and macOS have passed the conformance suites, though, so that ought to be correct. Thanks for the comments! I've just edited yours in. Commented Aug 2, 2019 at 9:13
  • 1
    A sed -e '1,/HI/d' file fails if the first line match the regex. A sed -e '1{/HI/d;};1,//d' file will remove up to the second regex match or the whole file if no second match exists. A sed '/HI/,$!d;//d' file will remove all lines where the regex match. Commented Aug 8, 2019 at 21:15
1

If you want to stick to Posix sed you could use this:

sed -ne ' /HI_THERE/!d :loop n p bloop ' inp.file

Or, written in a condensed manner:

sed -n '/HI_THERE/!d;:a;n;p;ba' inp.file
Improve this answer
0 
$ perl -ne 'print if 1 <(/HI_THERE/...eof)' input_file

Where we make use of the range operator ... to form the proper range and further constrain it to reject the first element within the selected range.

Improve this answer
0

sed

There is no simple way in sed to do what you ask for.
The simplest portable POSIXly solution for sed is something like:

sed -ne '/HI/{:1' -e 'n;p;b1' -e '}'

Other simple solutions are:

sed '0,/HI/d' ./file # GNU sed awk 'f; /HI/{f=1}' ./file

ed

The closest POSIXly non-sed solution is with ed. Just remove the range from the first line of the file to the regex /HI/, even if HI is on the same first line.

printf '%s\n' 1,/HI/d ,p Q | ed -s file

Or

ed -Gs imfile2 <<-\edscript 1,/HI/d ,p Q edscript

Which means:

  • printing no additional information (number of lines read) with option -s
  • remove (delete) all lines from the first line of the file to the regex /HI/ (1,/HI/d).
  • then print the whole file (,p).
  • and exit even if the file has been modified (Q).

If you want to modify the file replace ,p Q with just w (write to file).

Why sed fails on 1,/HI/d while ed works ?

Because sed expects the regex to match on the next line.

In ed, either 3,3 3,/3/ /3/,3 /3/,/3/ will print only one 3:

$ printf '%s\n' 3,3p 3,/3/p /3/3p /3/,/3/p Q | ed -s <(seq 5) 3 3 3 3

While sed will do this:

$ sed -ne 3,3p <(seq 5) 3 $ sed -ne 3,/3/p <(seq 5) 3 4 5 $ sed -ne /3/,3p <(seq 5) 3 $ sed -ne /3/,/3/p <(seq 5) 3 4 5

A regex at the end of a range is expected to match a line following the line matched on the start of the range (the left of the ,). There is no 3 following line numbered 3, so sed prints all following lines (4 and 5).

That is why GNU sed solves the issue with 0,/HI/.

Why :1;n;p;b1 ?

A way to print all lines (except the first) of a range is to use a loop that first asks for the next line and then prints it:

$ sed -n '5{:1;n;p;b1}' <(seq 8) # GNU syntax 6 7 8

So, simply match the regex you need /HI/ and get inside such loop.

sed -n '/HI/{:1;n;p;b1}' file # GNU syntax

Which needs to be expanded to a more complex script because some older seds don't allow labels to be finished by the ;.

sed -n -e '/HI/{:1' -e 'n;p;b1' -e '}' file # portable syntax
Improve this answer
0

grep -nw HI_THERE file.txt |awk -F":" '{print $1}' | xargs -I % sed '1,%d' file.txt

Explanation: I grep using exact word w, get the line number n
Next, I pull the line number using awk, the separator is :
Further I pipe this using xargs and delete until then.

Improve this answer
-1

sed is for doing s/old/new on individual strings that is all. For anything else you should use awk, e.g. with any awk in any shell on every UNIX box and handling "HI_THERE" appearing on any line:

$ awk 'f; /HI_THERE/{f=1}' file lemon coconut orange

If you want GNU sed for -i then use GNU awk for -i inplace instead. See https://stackoverflow.com/a/17914105/1745001 for other scripts to select sections of files.

Improve this answer
5
  • Hi, Ed, why not better <file instead of file for really odd named files? Commented Aug 8, 2019 at 18:28
  • @Isaac Because <file strips you of the ability to access FILENAME and isn't extensible to use with multiple input files which is a very common situation when using awk. If your file names contain odd characters just quote them appropriately in either case and it makes no difference which approach you use in the call to awk. The only mild benefit to using <file is in the error case - if file doesn't exist or can't be read then if you write awk '1' foo > bar then bar is created while with awk '1' <foo > bar it isn't and you get an error message from the shell instead of from awk. Commented Aug 8, 2019 at 18:31
  • 1
    That makes sense. However, if a file is named var=value file, awk will set a variable inside awk in either awk '...' "var=value file" or in awk '...' 'var=value file' even in awk '...' "$file" or the should be avoided in any case awk '...' $file. How to quote that?. Thanks in advance. Commented Aug 8, 2019 at 18:45
  • That's just the well-documented awk 'script' './var=value file' Commented Aug 8, 2019 at 18:53
  • @Isaac Hmm, I may have been too quick to say "well-documented". I thought it was but I just googled and looked through the POSIX spec and the gawk manual and couldn't find it so I emailed the gawk maintainers and asked them to add it to the next revision of the manual. Commented Aug 8, 2019 at 19:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.