116

I have a file something like:

# ID 1 blah blah blah blah $ description 1 blah blah # ID 2 blah $ description 2 blah blah blah blah 

How can I use a sed command to delete all lines between the # and $ line? So the result will become:

# ID 1 $ description 1 blah blah # ID 2 $ description 2 blah blah blah blah 

Can you please kindly give an explanation as well?

8 Answers 8

117

Use this sed command to achieve that:

sed '/^#/,/^\$/{/^#/!{/^\$/!d}}' file.txt 

Mac users (to prevent extra characters at the end of d command error) need to add semicolons before the closing brackets

sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt 

OUTPUT

# ID 1 $ description 1 blah blah # ID 2 $ description 2 blah blah blah blah 

Explanation:

  • /^#/,/^\$/ will match all the text between lines starting with # to lines starting with $. ^ is used for start of line character. $ is a special character so needs to be escaped.
  • /^#/! means do following if start of line is not #
  • /^$/! means do following if start of line is not $
  • d means delete

So overall it is first matching all the lines from ^# to ^\$ then from those matched lines finding lines that don't match ^# and don't match ^\$ and deleting them using d.

Sign up to request clarification or add additional context in comments.

6 Comments

For Mac users: To prevent extra characters at the end of d command error you need to add semicolons before the closing brackets sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt
How would you do that if you want to include the # and $ lines for deletion? If you want to find $ at the end of a line, you can do $\$,
Then just use: sed '/^#/,/^\$/d' file
I used sed '/^====/,/^>>>>/d' file-with-git-merge-conflicts.xml > file-ok.xml to remove git merge conflict lines between '=======' and '>>>>>>> branch-name, and sed '/^<<<</d' to remove '<<<<<<< HEAD' lines (my real file was a Dia software file), but mergetool like meld should be useful !-)
Is there a way to limit this to "between last instance of pattern 1 and the (only) instance of pattern 2"? Also, trying an adapted version of the command here is still removing text after pattern 2, which doesn't seem right.
|
67
$ cat test 1 start 2 end 3 $ sed -n '1,/start/p;/end/,$p' test 1 start end 3 $ sed '/start/,/end/d' test 1 3 

6 Comments

The speed at which this works on 300mb files is impressive. I'm talking split second on a SSD.
I was a little confused since I'm not familiar with the sed syntax. It was not clear that the first and second sed commands did not have a dependency - i.e. the difference between the two is whether you want to preserve the match token or not. Until I tested it, I had assumed the first command removed everything between the tokens and the second removed the tokens themselves. If you're trying to strip out a chunk between tokens you only need to use the second command.
No idea why, but '1,/start/p;/end/,$p' completely screwed up my workflow, as I was relying on this working. It does not work at all for me.
github.com/theAkito/akito-libbash/blob/… is the line in question. Did I miss something? Because to me it looks like it is exactly as you have shown in your answer @Lri.
The solution that actually works is the following: sed '/PATTERN-1/,/PATTERN-2/{//!d}' input.txt
|
32

In general form, if you have a file with contents of form abcde, where section a precedes pattern b, then section c precedes pattern d, then section e follows, and you apply the following sed commands, you get the following results.

In this demonstration, the output is represented by => abcde, where the letters show which sections would be in the output. Thus, ae shows an output of only sections a and e, ace would be sections a, c, and e, etc.

Note that if b or d appear in the output, those are the patterns appearing (i.e., they're treated as if they're sections in the output).

Also don't confuse the /d/ pattern with the command d. The command is always at the end in these demonstrations. The pattern is always between the //.

  • sed -n -e '/b/,/d/!p' abcde => ae
  • sed -n -e '/b/,/d/p' abcde => bcd
  • sed -n -e '/b/,/d/{//!p}' abcde => c
  • sed -n -e '/b/,/d/{//p}' abcde => bd
  • sed -e '/b/,/d/!d' abcde => bcd
  • sed -e '/b/,/d/d' abcde => ae
  • sed -e '/b/,/d/{//!d}' abcde => abde
  • sed -e '/b/,/d/{//d}' abcde => ace

4 Comments

Is there any way to get ade?
@TrippKinetics See my answer to a question about this case (first line of the range to be removed, last line of the range to be output). This also works in a more complex case where a line that matches the stop pattern (= new block) also matches the start pattern (= block to ignore).
@vinc17 Excellent. Could you do me a favor and add that case to the text of this answer? It makes it easier for me to refer to.
@TrippKinetics Unfortunately the above answer is underspecified, for instance in cases where lines may contain both b and d and you may have contiguous start-end blocks. For the ade case, perhaps you may prefer this solution to be in the same spirit as the above answer: printf "%s\n" 1a 2bd 3c 4db 5e | sed -e '/b/,/d/{//!d;x;/./!d;s/.*//;x}' (the s/.*// is optional here, but allows several start-end blocks). But in practice, I think that this is less useful.
27

Another approach with sed:

sed '/^#/,/^\$/{//!d;};' file 
  • /^#/,/^\$/: from line starting with # up to next line starting with $
  • //!d: delete all lines except those matching the address patterns

4 Comments

how do you do this inclusive of the patterns?
Try this: sed '/^#/,/^\$/d;' file.
How to delete including only the first or only the last pattern?
Compact and crystal clear. The most elegant answer.
8

I did something like this long time ago and it was something like:

sed -n -e "1,/# ID 1/ p" -e "/\$ description 1/,$ p" 

Which is something like:

  • -n suppress all output
  • -e "1,/# ID 1/ p" execute from the first line until your pattern and p (print)
  • -e "/\$ description 1/,$ p" execute from the second pattern until the end and p (print).

I might be wrong with some of the escaping on the strings, so please double check.

Comments

1

This might work for you (GNU sed):

sed '/^#/{:a;N;/^\$/M!ba;s/\n.*\n/\n/}' file 

On encountering a line beginning # gather up further lines until one beginning $, then replace everything between newlines with a newline.

Otherwise print the result.

N.B. Ranges have a flaw in that they expect both a start and end regexp. If the second regexp is not forthcoming the action of the range is still invoked i.e. if the action of the range is to delete lines, this will continue deleting lines until the end of the file.

Comments

1

sed is great for doing s/old/new/ on individual lines but for anything else, just use awk for clarity, simplicity, portability, robustness, etc.

Using any awk in any shell on every Unix box:

$ awk '/\$/{f=0} !f{print} /#/{f=1}' file # ID 1 $ description 1 blah blah # ID 2 $ description 2 blah blah blah blah 

Comments

-2

The example below removes lines between "if" and "end if".

All files are scanned, and lines between the two matching patterns are removed ( including them ).

IFS=' ' PATTERN_1="^if" PATTERN_2="end if" # Search for the 1st pattern in all files under the current directory. GREP_RESULTS=(`grep -nRi "$PATTERN_1" .`) # Go through each result for line in "${GREP_RESULTS[@]}"; do # Save the file and line number where the match was found. FILE=${line%%:*} START_LINE=`echo "$line" | cut -f2 -d:` # Search on the same file for a match of the 2nd pattern. The search # starts from the line where the 1st pattern was matched. GREP_RESULT=(`tail -n +${START_LINE} $FILE | grep -in "$PATTERN_2" | head -n1`) END_LINE="$(( $START_LINE + `echo "$GREP_RESULT" | cut -f1 -d:` - 1 ))" # Remove lines between first and second match from file sed -e "${START_LINE},${END_LINE}d;" $FILE > $FILE done 

1 Comment

I would not recommend your approach. While it demonstrates a couple of cool things like how to parse the output of grep in a clever way, your script runs several external programs and calls the shell functions several times when it can all be done internallly by sed, much more efficiently. Thus the downvote.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.