Why is this sed command only working on every other match?

/PATTERN/,/^>/ will match from a line containing PATTERN to a line starting with > (which can be a line containing PATTERN). You should instead match an empty line, like so:

$ sed '/PATTERN/,/^$/{/PATTERN/!d}' ip.txt >PATTERN >PATTERN >PATTERN >PATTERN >asdf 1 2 3 >asdf a b c 

Your aside isn't very clear to me, but if you want to delete the line with PATTERN as well, you can simplify it to:

$ sed '/PATTERN/,/^$/d' ip.txt >asdf 1 2 3 >asdf a b c 

You can also use:

awk -v RS= -v ORS='\n\n' '!/PATTERN/' 

but it will have an extra empty line at the end of the output. The advantage is that instead of your for loop, you can do this:

awk 'BEGIN{FS="\n"; ORS="\n\n"} NR==FNR{a[">" $0]; next} !($1 in a)' bad_results.txt RS= 16S.fasta 

The above code stores each line of bad_results.txt in an associative array, with > character prefixed. And then, contents of 16S.fasta will be printed only if entire line starting with > isn't present in bad_results.txt.

If you want a partial match:

awk 'BEGIN{FS="\n"; ORS="\n\n"} NR==FNR{a[$0]; next} {for (k in a) if(index($1, k)) next; print}' bad_results.txt RS= 16S.fasta 

In your range pattern match, the second element 'consumes' the line so that the start of the range no longer sees that block as a match. This is why you apparently have 'skipping.' This can be fixed by using a lookahead that does not consume characters to match. Unfortunately, sed lacks lookaheads.

Perl is really a better choice than sed for complex multi line matches involving lookaheads.

Here is a Perl that reads the file and applies the regex /(?:^>PATTERN)|(?:^>[\s\S]*?)(?=\v?^>|\z)/ (Demo) to it:

$ perl -0777 -lnE 'while(/(?:^>PATTERN)|(?:^>[\s\S]*?)(?=\v?^>|\z)/gm) { say $& }' file >PATTERN >PATTERN >PATTERN >PATTERN >asdf 1 2 3 >asdf a b c 

Aside: Please read Looping through the content of a file in Bash. The way you are doing it is not idea. Specifically, read here on the side effects of using cat in a Bash loop.

This might work for you (GNU sed):

sed -E '/PATTERN/{p;:a;$!{N;/\n>/!s/\n//;ta};D}' file 

As has been already stated, the range operator matches from PATTERN to a line beginning >. The latter line may also contain PATTERN but is not matched, hence the alternating pattern.

The solution above, does not use the range operator but instead gathers the lines from the first containing PATTERN to the line before a line beginning >.

If a line contains PATTERN it is printed, then subsequent lines are collected until the end-of-file or a line begins >.

Within this collection, newlines are removed - essentially making the first line in the pattern space the concatenation of one or more lines.

On a match (or end-of-file) this long line is removed and any line still in the pattern space is processed as if it had been read in as part of the normal sed cycle.

N.B. The difference between the d and the D commands is the d command deletes the pattern space and immediately begins the next sed cycle which involves reading in the next line of input. Whereas the D command removes everything up to and including the first newline in the pattern space and then begins the next sed cycle. However if the pattern space is not empty, the reading in of the next line from the input is forgone and then the sed cycle resumed.

An alternative:

sed '/^>/{h;/^>PATTERN/p};G;/\n>PATTERN/!P;d' file