2

I have a 6 GB application log file. The loglines have the following format (shortened)

[...] timestamp;hostname;sessionid-ABC;type=m timestamp;hostname;sessionid-ABC;set_to_TRUE [...] timestamp;hostname;sessionid-HHH;type=m timestamp;hostname;sessionid-HHH;set_to_FALSE [...] timestamp;hostname;sessionid-ZZZ;type=m timestamp;hostname;sessionid-ZZZ;set_to_FALSE [...] timestamp;hostname;sessionid-WWW;type=s timestamp;hostname;sessionid-WWW;set_to_TRUE

I have a lot of session with more then these 2 lines. I need to find out all sessions with type=m and set_to_TRUE

My first attempt was to grep all sessionIDs with type=m and write it into a file. Then looping with every line from the file (1 sessionID per line) trough the big logfile and grep for sessionID;set_to_TRUE

This method takes a loooot of time. Can anyone give me a hint to solve this in a much better and faster way?

Improve this question
7
  • Do you just want the session names or all lines pertaining to those sessions? In the latter case: is there any way to know when no more lines will appear for a session? And do you know that "type" will appear before "set_to_(TRUE|FALSE)"? Commented Jul 28, 2017 at 10:58
  • in short words: i need to count all type=m, which were set to TRUE or FALSE. in case these information are spread over 2 different lines, i only have the session-ID to get the information connected. Commented Jul 28, 2017 at 11:04
  • I figured that out from you sample data, but that doesn't answer what you need the output to be, or whether you know if the order will be as shown. Commented Jul 28, 2017 at 11:23
  • As a general rule you should always post a relevant input sample. You've already got two answers that assume those lines are consecutive... which is not the case. Commented Jul 28, 2017 at 11:53
  • 1
    @xMaNuu Also are the [...] actual lines or just a representation meaning continuation lines? Can you post a realistic data sample Commented Jul 28, 2017 at 12:07

5 Answers 5

Reset to default
1

If each session has a corresponding type and is set_to either TRUE or FALSE then you could use sed and ranges to do it:

sed '/type=m/,/set_to_/!d;/set_to_TRUE$/!d;s/.*\(sessionid-.*\);.*/\1/' infile

This removes all lines that are not in the /type=m/,/set_to_/ ranges. It also deletes the lines in those ranges that don't end in set_to_TRUE. The sessionid is then extracted from the remaining line (if any).
Alternatively,

sed -n '/type=/h;/set_to_TRUE$/{ x;s/.*\(sessionid-.*\);type=m$/\1/p }' infile

should print the same.
The latter works by overwriting the hold buffer on each line matching type=
Then, on each line matching set_to_TRUE, the buffers are exchanged and a substitution is attempted - namely, extract sessionid from a line ending in type=m - and if successful, the result is printed. Otherwise, nothing happens since autoprinting is disabled via -n.
The above assumes there are no trailing blanks on your lines.

Improve this answer
1

Use this awk command:

awk -F";" '/type=m/{flag=$3;next} /set_to_TRUE/ && ($3==flag)' infile.txt

it will match if both sessionIDs were same and required conditions also seen.

timestamp;hostname;sessionid-ABC;set_to_TRUE

above will print entire line, you can print only desired column by adding print $3 in order to have only sessionIDs, like below:

awk -F";" '/type=m/{flag=$3;next} /set_to_TRUE/ && ($3==flag){print $3}' infile.txt
Improve this answer
0
1

Using grep it can be done pretty easy:

grep -E "(type=|set_to_)" file.txt | grep -A 1 "type=m" | grep -B 1 "set_to_TRUE" > file1.txt &

Put it in the background get a coffee and let it finish. Not sure if 'awk' nor 'sed' would be quicker. 6GB are alot for only text, that's why it will take a long time anyway you try to do it.

You'd see when it is finished in the console anyway once you press enter or type in another command as:

[1]+ Done grep --color=auto -E '(type=|set_to_)' file.txt | grep --color=auto -A 1 "type=m" | grep --color=auto -B 1 "set_to_TRUE" > file1.txt
Improve this answer
3
  • @don_crissti Hmm, why doesn't he edit his question then? Does this mean there can be more then one session opened aswell? So there can be two type=m and afterwards 2 sessions_set_to some time later? That's kinda not clear from his example given. Commented Jul 28, 2017 at 11:53
  • I adjusted it so it fits the only one session at a time. Commented Jul 28, 2017 at 12:02
  • When it comes to searching, grep is faster than sed and awk (it's optimized for search). In this particular case (OP is looking for fixed strings, not regex) you can make it even faster using the -F switch with all three grep invocations. However, how much faster this could be depends on the number of matches (if 80% of the lines contained type= or set_to then you could say you're almost processing the file twice etc). Then there's the problem of sessionid. If the OP needs only that information (not the entire lines) you'll have to do some post-processing. Commented Jul 28, 2017 at 16:40
0

Assumptions:

  • you only want the session names
  • the "type=" line will come before the "set_to_" line
  • you actually meant that you only want sessions where "set_to_TRUE", as written in the question (which you contradicted in a comment)

If you need to handle multiple open sessions at once (it's not clear), you can do something like (untested):

perl -ne 'if (m/<timestamp>;<hostname>;(sessionid-[a-z]+);type=(.)/) { $type{$1}=$2 }; if (m/<timestamp>;<hostname>;(sessionid-[a-z]+);set_to_TRUE/) { if ($type{$1} eq "m") { print $1 } }'
Improve this answer
0 
awk -F';' '/type=m/ { seen[$3]=1 }; /set_to_TRUE/ && seen[$3] { print $3 ; delete seen[$3] }; /set_to_FALSE/ && seen[$3] { delete seen[$3] }' logfile.txt

This is similar to αғsнιη's answer, but instead of using just a single variable (flag), it uses an array (seen) to keep track of which session-ids it has seen that match type=m.

This means that it works even when there are other type=m lines for different sessions between the sighting of any session's type=m line and sighting that session's set_to_ line.

To minimise memory requirements for the script, it deletes the seen array element for a given session-id as soon as it sees either a matching set_to_TRUE or set_to_FALSE.

NOTE: The script above prints only the matching session-ids. If you want to print the actual type=m and set_to_TRUE lines, the awk script would be:

awk -F';' '/type=m/ { seen[$3]=$0 }; /set_to_TRUE/ && seen[$3] { print seen[$3]"\n"$0 ; delete seen[$3] }; /set_to_FALSE/ && seen[$3] { delete seen[$3] }' logfile.txt

This has the potential to use a LOT more memory. Storing entire input lines in an array takes a lot more RAM than storing integers.

NOTE2: this script assumes the session-ids are unique. If, for example, it is possible that different hostnames (field 2) may generate the same session-id, try this instead:

awk -F';' '/type=m/ { seen[$3$2]=1 }; /set_to_TRUE/ && seen[$3$2] { print $3 ; delete seen[$3$2] }; /set_to_FALSE/ && seen[$3$2] { delete seen[$3$2] }' logfile.txt

or

awk -F';' '/type=m/ { seen[$3$2]=$0 }; /set_to_TRUE/ && seen[$3$2] {print seen[$3$2]"\n"$0;delete seen[$3$2]}; /set_to_FALSE/ && seen[$3$2] { delete seen[$3$2] }' logfile.txt
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.