1

I have a C++ executable which scan some of my files and those files has some user id in it. After scanning is completed it generates below log file (abc.log) like this - 

INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_32_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_32_200003_5.data" WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 45576752 with value badge_leaf_cat and status -2 WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 870206432 with value badge_leaf_cat and status -2 INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_32_200003_5.data is valid INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_13_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_13_200003_5.data" WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 876269533 with value badge_leaf_cat and status -2 WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 17256973 with value badge_leaf_cat and status -2 WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 830173693 with value badge_leaf_cat and status -2 INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_13_200003_5.data is valid INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_0_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_0_200003_5.data" ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073135142816 num attributes: 0 seeing 1 bad records from 365 records ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073698151136 num attributes: 0 seeing 2 bad records from 595 records ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744072929739296 num attributes: 0 seeing 3 bad records from 1214 records ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:117) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupt

Now I need to grep the above logs and find out how many files it has been scanned, how many user id failed, and which files are corrupt.

  • For how many files it has scanned, I need to look for checking file words in each line and basis on that increase the count for number of files it has scanned.
  • For how many unique user id failed, I need to look for failed reading user id words in a line and basis on that increase the count and then provide the list of user id's which failed.
  • If any user id failed, then it means the file will be corrupt which was holding that user id, so I need to look for is corrupt word in each line and find the file name which is corrupt. In general this file database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupted.

And below is the response I wanted to see after scanning the above logs -

Total Number of Files Scanned - 1000 Total Number of Unique User ID failed - 10000 Total Number of Files Corrupted - 5 List of Unique User Id's which are corrupt - UserId-A UserId-B Files which are corrupted - FileName-A FileName-B

How would I go ahead and get the above result after scanning the logs using grep?

Improve this question

1 Answer 1

Reset to default
2

Try the following script:

#!/bin/bash logfile="$1" nfiles=$(grep -c 'checking file' "$logfile") failed_userid=($(grep -oP 'failed reading user id: \K[^ ]*' "$logfile")) corrupted_files=($(grep -oP '[^ ]*(?= is corrupt)' "$logfile")) echo "Total Number of Files Scanned - $nfiles" echo "Total Number of Unique User ID failed - ${#failed_userid[@]}" echo "Total Number of Files Corrupted - ${#corrupted_files[@]}" echo echo "List of Unique User Id's which are corrupt - " for uid in "${failed_userid[@]}"; do echo "$uid" done echo echo "Files which are corrupted - " for corf in "${corrupted_files[@]}"; do echo "$corf" done

Run it with

$ ./script file.log

The result for input from your question looks like

Total Number of Files Scanned - 3 Total Number of Unique User ID failed - 3 Total Number of Files Corrupted - 1 List of Unique User Id's which are corrupt - 18446744073135142816 18446744073698151136 18446744072929739296 Files which are corrupted - /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data

Short explanation:

  • -c option of grep counts the matching lines
  • -P enables perl regular expresions syntax
  • -o matches only part of lines
  • (?= construct is the so called positive look-ahead (take it as pattern, but do not include to the output)
  • \K is look-behind assertion (take whole pattern, but throw away from result everything up to this point)

The rest should be obvious. Be aware however that I've assumed there are no whitespaces in file names!

Improve this answer
1
  • Thanks a lot. Works like a charm. learned bunch of things with grep. appreciated your help. Commented Oct 28, 2014 at 0:10

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.