![gopherbadger-tag-do-not-edit]

Parser is in charge of turning raw log lines into objects that can be manipulated by heuristics. Parsing has several stages represented by directories on config/stage. The alphabetical order dictates the order in which the stages/parsers are processed.

The runtime representation of a line being parsed (or an overflow) is an Event, and has fields that can be manipulated by user :

• Parsed : a string dict containing parser outputs
• Meta : a string dict containing meta information about the event
• Line : a raw line representation
• Overflow : a representation of the overflow if applicable

The Event structure goes through the stages, being altered with each parsing step. It's the same object that will be later poured into buckets.

A parser configuration is a Node object, that can contain grok patterns, enrichement instructions.

For example :

filter: "evt.Line.Labels.type == 'testlog'" debug: true onsuccess: next_stage name: tests/base-grok pattern_syntax: MYCAP: ".*" nodes: - grok: pattern: ^xxheader %{MYCAP:extracted_value} trailing stuff$ apply_on: Line.Raw statics: - meta: log_type value: parsed_testlog

optional if present and prometheus or profiling are activated, stats will be generated for this node.

filter: "Line.Src endsWith '/foobar'"

• optional filter : an expression that will be evaluated against the runtime of a line (Event)
  • if the filter is present and returns false, node is not evaluated
  • if filter is absent or present and returns true, node is evaluated

debug: true

• optional debug : a bool that sets debug of the node to true (applies at runtime and configuration parsing)

onsuccess: next_stage|continue

• mandatory indicates the behavior to follow if the node succeeds. next_stage make the line go to the next stage, while continue will continue processing the current stage.

statics: - meta: service value: tcp - meta: source_ip expression: "Event['source_ip']" - parsed: "new_connection" expression: "Event['tcpflags'] contains 'S' ? 'true' : 'false'" - target: Parsed.this_is_a_test value: foobar

Statics apply when a node is considered successful, and are used to alter the Event structure. An empty node, a node with a grok pattern that succeeded or an enrichment directive that worked are successful nodes. Statics can :

• meta: add/alter an entry in the Meta dict
• parsed: add/alter an entry in the Parsed dict
• target: indicate a destination field by name, such as Meta.my_key The source of data can be :
• value: a static value
• expr_result : the result of an expression

Grok patterns are used to parse one field of Event into one or several others :

grok: name: "TCPDUMP_OUTPUT" apply_on: message

name is the name of a pattern loaded from patterns/. Base patterns can be seen on the repo : https://github.com/crowdsecurity/grokky/blob/master/base.go

grok: pattern: "^%{GREEDYDATA:request}\\?%{GREEDYDATA:http_args}$" apply_on: request

pattern which is a valid pattern, optionally with an apply_on that indicates to which field it should be applied

Present at the Event level, the pattern_syntax is a list of subgroks to be declared.

pattern_syntax: DIR: "^.*/" FILE: "[^/].*$"

The Enrichment mechanism is exposed via statics :

statics: - method: GeoIpCity expression: Meta.source_ip - meta: IsoCode expression: Enriched.IsoCode - meta: IsInEU expression: Enriched.IsInEU

The GeoIpCity method is called with the value of Meta.source_ip. Enrichment plugins can output one or more key:values in the Enriched map, and it's up to the user to copy the relevant values to Meta or such.

The Node object allows as well a nodes entry, which is a list of Node entries, allowing you to build trees.

filter: "Event['program'] == 'nginx'" #A nodes: #A' - grok: #B name: "NGINXACCESS" # this statics will apply only if the above grok pattern matched statics: #B' - meta: log_type value: "http_access-log" - grok: #C name: "NGINXERROR" statics: - meta: log_type value: "http_error-log" statics: #D - meta: service value: http

The evaluation process of a node is as follows:

• apply the filter (A), if it doesn't match, exit
• iterate over the list of nodes (A') and apply the node process to each.
• if a grok entry is present, process it
  • if the grok entry returned data, apply the local statics of the node (if the grok 'B' was successful, apply B' statics)
• if any of the nodes or the grok was successful, apply the statics (D)

Main structs :

• Node (config.go) : the runtime representation of parser configuration
• Event (runtime.go) : the runtime representation of the line being parsed

Main funcs :

• CompileNode : turns YAML into runtime-ready tree (Node)
• ProcessNode : process the raw line against the parser tree, and produces ready-for-buckets data