Find pattern: sort and delete older files

Question

I want to delete old files and keep only the most recent from each file pattern based on the first five characters (aa_te, bb_co, cc_co, dd_xy) and the date and time in the file name.

Note:

All files have .err.mm_dd_yyyy hr_mm_ss am/pm.csv for the sorting. Filename length may vary

Example: I have the below CSV files in my folder c/test

aa_te_bbc_207_mc_ghghgh.err.05_31_2021 7_15_04 am.csv aa_te_ccd.err.05_30_2021 7_55_35 pm.csv bb_co_ddee_354_m.err.06_15_2021 5_20_00 am.csv bb_co_eeffff.err.06_15_2021 5_20_01 am.csv cc_co_eeffyy_cc.err.04_10-2020 12_00_00 am.csv dd_xy_eeffyyvv_cc.err.04_10-2020 12_00_00 am.csv dd_xy_eeffyyvv_cc.err.04_10-2020 12_10_25 am.csv

What I expect remains of those afterwards:

aa_te_bbc_207_mc_ghghgh.err.05_31_2021 7_15_04 am.csv bb_co_eeffff.err.06_15_2021 5_20_01 am.csv cc_co_eeffyy_cc.err.04_10-2020 12_00_00 am.csv dd_xy_eeffyyvv_cc.err.04_10-2020 12_10_25 am.csv

This would be a lot easier if you used YYYY-mm-dd date format, and 2-digit hours — glenn jackman
– glenn jackman, Commented Nov 16, 2021 at 21:10
I changed the file directory dir= c/test and, removed the echo rm and kept && " $search" at last and run the code. There is no error message but the files are not removed at all. Am I missing something? Was it working for you? Will this code work for hundreds of files with out any change? — Promise
– Promise, Commented Nov 19, 2021 at 2:49

alecxs · Accepted Answer · 2021-11-19 19:30:22Z

delete "echo" rm in last line when okay for you

#!/bin/bash # search dir dir=./ # function conv PATTERN conv() { # extract date from file name: 06_15_2021 5_20_00 am date=${1%.*} date=${date##*.} # sanitize date string: 06/15/2021 5:20:00-am date=${date//[^0-9AaPpMm]/-} date=${date/-//} date=${date/-//} date=${date/-/ } date=${date/-/:} date=${date/-/:} # print epoch time date --date="$date" +%s || return 1 return 0 } # search: *m.csv for file in "$dir"*[Mm].[Cc][Ss][Vv] do [ -f "$file" ] || continue # file name: bb_co_ddee_354_m.err.06_15_2021 5_20_00 am.csv file=${file##*/} # base name: bb_co_ddee_354_m name=${file%%.*} # suffix: ddee_354_m del=${name#*_} del=${del#*_} # short name = base name - _suffix: bb_co name=${name%_$del} # convert date date=$(conv "$file") || continue # search short name: bb_co_*m.csv for search in "${dir}${name}"_*[Mm].[Cc][Ss][Vv] do # delete older file date2=$(conv "${search##*/}") || continue [ "$date" -gt "$date2" ] && echo rm "$search" done done

Comments are not for extended discussion; this conversation has been moved to chat. — terdon
– terdon ♦, Commented Nov 22, 2021 at 10:33

Stéphane Chazelas · Accepted Answer · 2021-11-22 10:33:11Z

With zsh instead of bash, you could do:

#! /bin/zsh - zmodload zsh/datetime typeset -A latest=() all=( *.err.*.csv(Nnoe['LC_ALL=C strftime -rs REPLY %m_%d_%Y_%I_%M_%S_%p ${${${REPLY:r}##*.err.}//[- ]/_}']) ) for file in $all; do latest[$file[1,5]]=$file done keep=($latest) echo rm -f -- ${all:|keep}

(to be run within the directory where the files are; remove echo (for dry-run) if that looks alright)

We sort the list of files by timestamp into $all using zsh's oe glob qualifier which sorts (numerically (with n)) based on the value of $REPLY as returned by the code in [...].

That code uses strftime -r (strftime in reverse which in effect is strptime) to convert the time stamp in the file name to the epoch time in seconds.

Then we loop on that list to record the latest file for each 5-character prefix in to the $latest associative array.

Then we use the ${a:|b} array subtraction operator to remove the elements of $all that are not in $keep (itself containing the values of the $latest associative array).

That assumes all the *.err.*.csv files in the current directory are in the format you specified, though it will also accommodate filename where date elements are separated with -s instead of _s as seen in some of your samples (by converting the -s to _s using the ksh93-style ${var//pattern/replacement} operator).

@alecxs, oh. I hadn't seen that. But then again, it contradicts the OP's specification. — Stéphane Chazelas
– Stéphane Chazelas, Commented Nov 22, 2021 at 10:14
@alecxs, I've updated the code so it converts the -s to _s prior to calling strftime -r in case it wasn't just typos in the OP's sample. — Stéphane Chazelas
– Stéphane Chazelas, Commented Nov 22, 2021 at 10:18

Stack Exchange Network

Find pattern: sort and delete older files

2 Answers 2

You must log in to answer this question.

Linked

Hot Network Questions

Find pattern: sort and delete older files

2 Answers 2

You must log in to answer this question.

Linked

Related

Hot Network Questions