Cloudwatch Logs Insights working with multiple @messages

Question

I have the following query with the following output:

Query: filter @message like /A:|B:/ Output: [INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }] [INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':0}

I would like to print ONLY the A message when in the B message 'MyValue' = 0. For the above example, I would have to have the following output

Output: [INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]

For the next example

[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }] [INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':12}

The output should be empty

I can't do something like this because I miss the A message:

filter @message like /A:|B:/ filter MyValue = 0

Any ideas?

No, I didn't. You should export your data and process it in another platform like excel — Andres Gardiol
– Andres Gardiol, Commented Aug 3, 2021 at 19:05
@OmkarKulkarni I posted a possible solution below, hopefully it will help in your case — YoniXw
– YoniXw, Commented Sep 14, 2021 at 13:07

YoniXw · Accepted Answer · 2021-09-14 12:57:37Z

If anyone still interested, there IS ways to get the first and last from grouping by a field. So if you can fit your data into pairs of messages, it might help.

For example, given API Gateway access log (each row is a @message):

2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Extended Request Id: *************** 2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Verifying Usage Plan for request: 01c53288-5d25-*******. API Key: API Stage: **************/dev 2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) API Key authorized because method 'ANY /path/{proxy+}' does not require API Key. Request will not contribute to throttle or quota limits 2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Usage Plan check succeeded for API Key and API Stage **************/dev 2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Starting execution for request: 01c53288-5d25-******* 2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) HTTP Method: GET, Resource Path: /path/json.json 2021-09-14T14:09:00.468+03:00 (01c53288-5d25-*******) Method completed with status: 304

We can get method, uri and return code from the last 2 rows. To do this, I parse the relevant data into params, and then get them by doing aggregation by request id (that i also parse)

The magic is: using stats likesortsFirst() and sortsLast() and grouping by @reqid. (AWS Docs

Note: IMO, don't use earliest() and latest() as they depend on built-in @timestamp and worked weird for me where 2 sequential messages had the same timestamp

So, for example, using this query:

filter @message like "Method" | parse @message /\((?<@reqid>.*?)\) (.*?) (Method: (?<@method>.*?), )?(.*?:)* (?<@data>[^\ ]*)/ | sort @timestamp desc | stats sortsFirst(@method) as @reqMethod, sortsFirst(@data) as @reqPath, sortsLast(@data) as @reqCode by @reqid | limit 20

We would get the following desired output:

@reqid @reqMethod @reqPath @reqCode f42e2b44-b858-45cb-***************** GET /path-******.json 304 fecddb03-3804-4ff5-***************** OPTIONS /path-******.json 200 e8e47185-6280-4e1e-***************** GET /path-******.json 304 e4fa9a0c-6d75-4e26-***************** GET /path-******.json 304

Another note: Unfortunately, since we use aggregations, if one would like to add another aggregation like count(*), it does not seem currently possible in aws logs insight, as it doesn't support 2 level aggregation.

Collectives™ on Stack Overflow

Cloudwatch Logs Insights working with multiple @messages

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related