0

I have a file with below content that is output from a ls -lrt command. It's a big file that contains ls -lrt output for few months.

sh> ls -lrt /vdat/sfrdir/mkbs/daily/2020*/*/*{GNMDCONS,GMFDAILY,GMFDLYMNI,FHLSEC1D,FNMSEC1D,FHLSUP1D,FNMSUP1D,FNMDUSDN,GNMHECD,FNMSEC1M,FHLSEC1M,FHLSEC2M,FHLSEC3M,FHLPSEC4M,FNMSUP1M,FHLSUP1M,FHLSUP2M,FHLSUP3M,FHLPSUP4M,FNMDUSMBS,GNMISS2,G11FCTR,G21FCTR,GNMHECM,GMFMON,GNMMCONS,G12FCTR,G22FCTR,FNMGMEGA,FHLGGNT,GNMPDSCL,FNMDUSDC,FNMDUSMC}.* 

A file has two parts: a ZIP file and a SIG file. They get downloaded with different time stamps.

sh> head /tmp/jan_files -n14 -rw-r--r--. 1 ctlr sftg 1541 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 191881 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 23185 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDLYMNI.ZIP -rw-r--r--. 1 ctlr sftg 30 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDLYMNI.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 7726 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSEC1D.ZIP.11:39:35 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSEC1D.SIG.11:40:18 -rw-r--r--. 1 ctlr sftg 185206 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSUP1D.ZIP.11:40:38 -rw-r--r--. 1 ctlr sftg 11179 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSEC1D.ZIP.11:44:54 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSEC1D.SIG.11:45:37 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSUP1D.SIG.11:41:21 -rw-r--r--. 1 ctlr sftg 282086 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSUP1D.ZIP.11:45:59 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSUP1D.SIG.11:46:44 sh> 

I am trying to create an output that pairs the ZIP and SIG file together based on the time they arrived like below.

sh> head /tmp/jan_files -n14 -rw-r--r--. 1 ctlr sftg 1541 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 191881 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 23185 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDLYMNI.ZIP -rw-r--r--. 1 ctlr sftg 30 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDLYMNI.SIG -rw-r--r--. 1 ctlr sftg 7726 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSEC1D.ZIP.11:39:35 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSEC1D.SIG.11:40:18 -rw-r--r--. 1 ctlr sftg 185206 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSUP1D.ZIP.11:40:38 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSUP1D.SIG.11:41:21 -rw-r--r--. 1 ctlr sftg 11179 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSEC1D.ZIP.11:44:54 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSEC1D.SIG.11:45:37 -rw-r--r--. 1 ctlr sftg 282086 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSUP1D.ZIP.11:45:59 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSUP1D.SIG.11:46:44 sh> 

I have created the below sort command which is close but its not what I am looking for.

sh> sort -s -k6M -k7n -b -k8.1,8.2 -k9.42,9.43 -k9.51,9.52 /tmp/jan_files | head -n14 -rw-r--r--. 1 ctlr sftg 1541 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 23185 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDLYMNI.ZIP -rw-r--r--. 1 ctlr sftg 191881 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 30 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDLYMNI.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 7726 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSEC1D.ZIP.11:39:35 -rw-r--r--. 1 ctlr sftg 185206 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSUP1D.ZIP.11:40:38 -rw-r--r--. 1 ctlr sftg 11179 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSEC1D.ZIP.11:44:54 -rw-r--r--. 1 ctlr sftg 282086 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSUP1D.ZIP.11:45:59 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSEC1D.SIG.11:40:18 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSUP1D.SIG.11:41:21 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSEC1D.SIG.11:45:37 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSUP1D.SIG.11:46:44 sh> 

Update:

Another case of input files

sh-4.1$ head /tmp/jan_files -rw-r--r--. 1 ctlr sftg 2163 Jan 3 05:23 /vdat/sfrdir/mkbs/daily/20200103/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:24 /vdat/sfrdir/mkbs/daily/20200103/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 239840 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 13560 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.11:43:28 -rw-r--r--. 1 ctlr sftg 207324 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.11:44:23 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.11:44:58 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.11:44:04 -rw-r--r--. 1 ctlr sftg 8525 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.11:38:56 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.11:39:31 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.11:39:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.11:40:25 -rw-r--r--. 1 ctlr sftg 8537 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.15:39:11 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.15:40:10 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.15:40:47 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.15:39:51 -rw-r--r--. 1 ctlr sftg 13958 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.15:43:55 -rw-r--r--. 1 ctlr sftg 212215 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.15:44:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.15:44:32 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.15:45:28 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 16340 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG -rw-r--r--. 1 ctlr sftg 8537 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG -rw-r--r--. 1 ctlr sftg 239708 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG -rw-r--r--. 1 ctlr sftg 2513 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP.20:00:56 -rw-r--r--. 1 ctlr sftg 5286 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP.20:01:54 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG.20:02:30 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG.20:01:35 -rw-r--r--. 1 ctlr sftg 3249 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP -rw-r--r--. 1 ctlr sftg 5951 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG sh-4.1$ 

A vendor creates certain files everyday. Below are the list of files. {GNMDCONS,GMFDAILY,GMFDLYMNI,FHLSEC1D,FNMSEC1D,FHLSUP1D,FNMSUP1D,FNMDUSDN,GNMHECD,FNMSEC1M,FHLSEC1M,FHLSEC2M,FHLSEC3M,FHLPSEC4M,FNMSUP1M,FHLSUP1M,FHLSUP2M,FHLSUP3M,FHLPSUP4M,FNMDUSMBS,GNMISS2,G11FCTR,G21FCTR,GNMHECM,GMFMON,GNMMCONS,G12FCTR,G22FCTR,FNMGMEGA,FHLGGNT,GNMPDSCL,FNMDUSDC,FNMDUSMC} Each file has two parts: a ZIP file and a SIG file. The problem is that they get downloaded with different timestamps. I am trying to pair them together based on the time that they get downloaded. I will then use that list for another operation that will load these file into a database. The load operation requires both the ZIP file and it's associated SIG file. Also, when the same file arrives multiple times a day, it gets renamed with timestamp appended to the file name.

I am looking for output something like this.

-rw-r--r--. 1 ctlr sftg 2163 Jan 3 05:23 /vdat/sfrdir/mkbs/daily/20200103/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:24 /vdat/sfrdir/mkbs/daily/20200103/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 239840 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 8525 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.11:38:56 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.11:39:31 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.11:39:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.11:40:25 -rw-r--r--. 1 ctlr sftg 13560 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.11:43:28 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.11:44:04 -rw-r--r--. 1 ctlr sftg 207324 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.11:44:23 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.11:44:58 -rw-r--r--. 1 ctlr sftg 8537 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.15:39:11 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.15:39:51 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.15:40:10 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.15:40:47 -rw-r--r--. 1 ctlr sftg 13958 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.15:43:55 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.15:44:32 -rw-r--r--. 1 ctlr sftg 212215 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.15:44:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.15:45:28 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG -rw-r--r--. 1 ctlr sftg 8537 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG -rw-r--r--. 1 ctlr sftg 16340 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG -rw-r--r--. 1 ctlr sftg 239708 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG -rw-r--r--. 1 ctlr sftg 2513 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP.20:00:56 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG.20:01:35 -rw-r--r--. 1 ctlr sftg 5286 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP.20:01:54 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG.20:02:30 -rw-r--r--. 1 ctlr sftg 3249 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG -rw-r--r--. 1 ctlr sftg 5951 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG 

This is what my sort outputs

sh-4.1$ sort -s -k6M -k7n -b -k8.1,8.2 -k9.42,9.43 -k9.51,9.52 /tmp/jan_files -rw-r--r--. 1 ctlr sftg 2163 Jan 3 05:23 /vdat/sfrdir/mkbs/daily/20200103/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 239840 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:24 /vdat/sfrdir/mkbs/daily/20200103/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 8525 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.11:38:56 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.11:39:50 -rw-r--r--. 1 ctlr sftg 13560 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.11:43:28 -rw-r--r--. 1 ctlr sftg 207324 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.11:44:23 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.11:39:31 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.11:40:25 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.11:44:58 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.11:44:04 -rw-r--r--. 1 ctlr sftg 8537 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.15:39:11 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.15:40:10 -rw-r--r--. 1 ctlr sftg 13958 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.15:43:55 -rw-r--r--. 1 ctlr sftg 212215 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.15:44:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.15:40:47 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.15:39:51 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.15:44:32 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.15:45:28 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 8537 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 16340 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 239708 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 2513 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP.20:00:56 -rw-r--r--. 1 ctlr sftg 5286 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP.20:01:54 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG.20:02:30 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG.20:01:35 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG -rw-r--r--. 1 ctlr sftg 3249 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP -rw-r--r--. 1 ctlr sftg 5951 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG sh-4.1$ 

And this is what your solution outputs

sh-4.1$ awk -F"[/.]" '{print $2,$9,$8" |"$0}' /tmp/jan_files | \ > sort -k5,5M -k6,6n -k7,7 -k8,9 | \ > awk -F"|" '{print $2}' -rw-r--r--. 1 ctlr sftg 2163 Jan 3 05:23 /vdat/sfrdir/mkbs/daily/20200103/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:24 /vdat/sfrdir/mkbs/daily/20200103/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 239840 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 05:34 /vdat/sfrdir/mkbs/daily/20200103/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 13560 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.11:43:28 -rw-r--r--. 1 ctlr sftg 207324 Jan 3 06:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.11:44:23 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.11:44:04 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.11:44:58 -rw-r--r--. 1 ctlr sftg 8525 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.11:38:56 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.11:39:31 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.11:39:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 06:33 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.11:40:25 -rw-r--r--. 1 ctlr sftg 8537 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP.15:39:11 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 11:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP.15:40:10 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG.15:39:51 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG.15:40:47 -rw-r--r--. 1 ctlr sftg 13958 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP.15:43:55 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG.15:44:32 -rw-r--r--. 1 ctlr sftg 212215 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP.15:44:50 -rw-r--r--. 1 ctlr sftg 29 Jan 3 11:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG.15:45:28 -rw-r--r--. 1 ctlr sftg 174536 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 16340 Jan 3 15:30 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 8537 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Products/FHLSEC1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSEC1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FHLSUP1D.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:31 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSEC1D.SIG -rw-r--r--. 1 ctlr sftg 239708 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMSUP1D.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:32 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMSUP1D.SIG -rw-r--r--. 1 ctlr sftg 2513 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP.20:00:56 -rw-r--r--. 1 ctlr sftg 5286 Jan 3 15:55 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP.20:01:54 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG.20:01:35 -rw-r--r--. 1 ctlr sftg 29 Jan 3 15:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG.20:02:30 -rw-r--r--. 1 ctlr sftg 3249 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSDN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSDN.SIG -rw-r--r--. 1 ctlr sftg 5951 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Products/FNMDUSMN.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 3 19:56 /vdat/sfrdir/mkbs/daily/20200103/Signal/FNMDUSMN.SIG sh-4.1$ 
5
  • what are you expecting for output when a pair of 'matching' files have OS timestamps that differ by 1+ second(s), eg: 05:23 and 05:25 or 05:59 and 06:00? Commented Feb 5, 2020 at 3:46
  • the 4x files @ 05:39 are not sorted alphabetically, while the 6x files a@ 06:33 are sorted alphabetically ... typo? or some other/odd sorting logic? Commented Feb 5, 2020 at 4:25
  • Why not parse ls? Commented Feb 5, 2020 at 6:08
  • @markp I have made an update to the question which I hope answers your questions. Commented Feb 6, 2020 at 1:30
  • you need to define (in text) the rules you're trying to implement in your sort; I spent too much time last night trying to guess at what you're looking for; maybe someone else will come along who has some free time on his/her hands to try and glean a pattern from the wall of sample ls listings ... Commented Feb 6, 2020 at 3:18

1 Answer 1

3

Assumptions:

  • time they arrived is referring to the OS timestamp on the file (eg, Jan 2 05:23), and not the timestamp suffix that some files have
  • the OS timestamp has precedence when sorting files which means ...
  • 'matching' files will have the same OS timestamp (eg, 05:23 == 05:23) otherwise the proposed sorting method may not group matching files together [NOTE: this assumption is pretty flimsy since there are going to be 'matching' files with OS timestamps that differ by 1+ seconds which in turn causes a 1-minute difference, eg: 05:23:59+1 second => 05:24]
  • for a set of files with the same OS timestamp the sorting preference is by parent directory ({Products,Signal}) and then filename, both alphabetically (which means the files in the sample output with OS timestamp of 05:39 are not sorted properly)

Sample data:

$ cat jan_files -rw-r--r--. 1 ctlr sftg 1541 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 191881 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 23185 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDLYMNI.ZIP -rw-r--r--. 1 ctlr sftg 30 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDLYMNI.SIG -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 7726 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSEC1D.ZIP.11:39:35 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSEC1D.SIG.11:40:18 -rw-r--r--. 1 ctlr sftg 185206 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSUP1D.ZIP.11:40:38 -rw-r--r--. 1 ctlr sftg 11179 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSEC1D.ZIP.11:44:54 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSEC1D.SIG.11:45:37 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSUP1D.SIG.11:41:21 -rw-r--r--. 1 ctlr sftg 282086 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSUP1D.ZIP.11:45:59 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSUP1D.SIG.11:46:44 

The general idea is to pull a copy of the sortable fields to the front of the line, with a copy of the original line appended on the end; once we perform the desired sorts we strip off the 'sort fields' and we're left with our original lines in the desired sort order.

Here's one awk/sort/awk combo:

awk -F"[/.]" '{print $2,$9,$8" |"$0}' jan_files | \ sort -k5,5M -k6,6n -k7,7 -k8,9 | \ awk -F"|" '{print $2}' 

Where:

  • first awk command uses dual delimiters (\ and .) and reprints fields 2 (owner/group/timestamp) and the last 2 fields ({Products,Signal} and the base/filename), followed by a new delimiter (|) and the entire line; this allows us to (more) easily sort by all the desired fields and still keep our entire line intact ...
  • the sort command allows us to sort by Month, Day, HH:MM, {Products,Signal} and filename; with appropriate flags to allow us to sort as per the desired output
  • second awk makes use of our new delimiter (|) to allow us to strip off our 'sort fields' and display the original line

Running the above on the sample data file

-rw-r--r--. 1 ctlr sftg 1541 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDAILY.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:23 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDAILY.SIG -rw-r--r--. 1 ctlr sftg 23185 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GMFDLYMNI.ZIP -rw-r--r--. 1 ctlr sftg 30 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GMFDLYMNI.SIG -rw-r--r--. 1 ctlr sftg 191881 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Products/GNMDCONS.ZIP -rw-r--r--. 1 ctlr sftg 29 Jan 2 05:39 /vdat/sfrdir/mkbs/daily/20200102/Signal/GNMDCONS.SIG -rw-r--r--. 1 ctlr sftg 7726 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSEC1D.ZIP.11:39:35 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:32 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSEC1D.SIG.11:40:18 -rw-r--r--. 1 ctlr sftg 185206 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FHLSUP1D.ZIP.11:40:38 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FHLSUP1D.SIG.11:41:21 -rw-r--r--. 1 ctlr sftg 11179 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSEC1D.ZIP.11:44:54 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSEC1D.SIG.11:45:37 -rw-r--r--. 1 ctlr sftg 282086 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Products/FNMSUP1D.ZIP.11:45:59 -rw-r--r--. 1 ctlr sftg 29 Jan 2 06:33 /vdat/sfrdir/mkbs/daily/20200102/Signal/FNMSUP1D.SIG.11:46:44 
Sign up to request clarification or add additional context in comments.

2 Comments

What a marvelous answer!
You don't need a \ after |. If | is the last character in the line, then newlines (and lines with a comment) are ignored.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.