How to cut a file starting from the line in which a certain pattern occurs?

You should be able to do it by just truncating the file in place without having to write a new copy of the file like sed -i/perl -i/ed/gawk -i /usr/share/awk/inplace.awk¹ would do. With perl:

find . -name '*.txt' -type f -exec perl -ne ' BEGIN{@ARGV=map{"+<$_"}@ARGV} # open files in read+write mode in the # while(<>) loop implied by -n if (/END DATA/) { seek ARGV,-length,1; # back to beginning of matching line print ARGV "NEW END\n"; truncate ARGV, tell ARGV; close ARGV; # skip to next file }' {} + 

That minimises the I/O as perl stops reading as soon as it finds a match, and NEW END\n is the only thing it writes. It also writes in place, so the files metadata (ownership, permission, acls, sparseness...) are preserved and hard links are not broken.

With -exec {} + we also minimise the number of perl invocations.

^{¹ do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'}

With GNU grep and GNU sed

grep -lZ 'END DATA' *.txt | xargs -0 sed -i -e '/END DATA/,${//i foo' -e 'd}' 

where *.txt assumes all your files are in current directory ending with .txt extension. If you need to recursively search for files, GNU grep also supports -r/-R options.

/END DATA/,$ range of lines to operate

//i foo here // will match the previously used regex, i.e. /END DATA/ and i command will add the new ending marker as needed

as i command has to be separated by newline, -e option is used to separate the d command to delete all lines matched by the range

as an alternate, you can also use this, but only one file will be passed at a time to sed:

grep -lZ 'END DATA' *.txt | xargs -0 -n1 sed -i -e '/END DATA/{i foo' -e 'Q}' 

Keep it simple and just use awk for the file manipulation part, e.g., with GNU find, awk, grep, and xargs:

find . -type f -exec grep -lZ 'END DATA' {} + | xargs -r0 awk -i inplace '/END DATA/{nextfile} 1' 

or to print your own end tag at the end of every file:

... | xargs -r0 awk -i inplace '/END DATA/{print "NEW END"; nextfile} 1' 

This python 3.8 solution is loosely based on Stephane's in-place truncate solution with a couple of differences 1. The code doesn't rely on external utilities for directory traversal 2. Files are memory-mapped for ease of locating the END DATA string

Place the code in a .py file and pass the directory name as a parameter

import mmap import os import sys from contextlib import closing def yield_all_files(dir_): for root, dir_, files in os.walk(dir_): yield from (os.path.join(root, file_) for file_ in files if file_.endswith('.txt')) if __name__ == '__main__': old = b'END DATA' new = b'NEW END\n' dir_ = sys.argv[1] for file_ in yield_all_files(dir_): with open(file_, mode='r+b') as f: with closing(mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_WRITE)) as mm: if (loc := mm.find(old)) > -1: mm.seek(loc) mm.write(new) mm.resize(mm.tell()) 

Combining Sundeep’s answer and Ed Morton’s answer, but without xargs:

find . -type f -name '*.txt' …(other criteria)… -exec grep -q 'END DATA' {} ';' -a \ -exec sed -i -e '/END DATA/,${//i NEW END' -e 'd}' {} +

• find, of course, selects files.  By default, it searches the specified director(ies) recursively.  To search only the current directory, add -maxdepth 1 after the ..
• grep -q quietly exits with a “success” exit status if a file contains the pattern being searched for (END DATA) and “false” otherwise.
• -a means “AND”, like && on a shell command line.  It means “do the following thing if (only if) the previous thing succeeded.”  Actually, it’s the default conjunction operator between find predicates (tests/actions), so you can leave it out.  I included it only for clarity.
• So the sed command, which is copied verbatim from Sundeep’s answer (but with foo changed to NEW END), is executed only on files that contain the END DATA string and satisfy the other find tests.
• -exec … + causes sed to be invoked once with multiple files, as xargs does it.
  Note that we can’t use -exec … + with the grep command because it doesn’t allow testing exit status.

Using awk to find the offset of the pattern, and dd to truncate the file at that point and append the new trailer:

# usage truncatoo pattern new_end find_args ... truncatoo(){ pat=$1; shift; tail=$1; shift LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{ gsub(q,q"\\"q q,FILENAME); system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null"); exit } {l+=length()+1} ' {} \; } truncatoo '/END DATA/' 'NEW END\n' file.txt truncatoo '/END DATA/' 'NEW END\n' . -type f -name '*.txt' 

With an awk implementation which supports nextfile (gawk, bwk, some versions of mawk [1]), that could be done more efficiently by passing batches of files to awk:

# usage truncatoo pattern new_end find_args ... truncatoo(){ pat=$1; shift; tail=$1; shift LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{ gsub(q,q"\\"q q,FILENAME); system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null"); l=0; nextfile } {l+=length()+1} ' {} + } 

$ file="a'b\$q * r"; seq 1 100 >"$file" $ truncatoo /7/ 'CUT\n' "$file"; cat "$file" 1 2 3 4 5 6 CUT 

Instead of the icky 2>/dev/null status=noxfer could be used with dd implementations which support it.

The quoting kludge and environment variables passing is a mess, it could use some improvement.

[1]: according to the GNU awk manual, it should be supported in mawk too. However, the older version of mawk from Debian 10 does not support it.

If this is a one time task, it is convenient to use vi for this:

First, locate the line with the needed content (by using search / or ?)

To delete all lines following the current line to the end of file, press d G.

To delete all lines from the start of the file to the current line, press d gg.

Save and quit :wq