[tcp/udp] Add option to parse syslog

[taylor-swanson]

What does this PR do?

• Add option to parse syslog
• Fix sample logs to be properly formatted RFC 5424 messages
• Add syslog ECS field references
• Expose TLS in TCP integration
• Add system test for TCP/TLS
• Add system tests for syslog

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

cd packages/tcp && elastic-package test cd packages/udp && elastic-package test 

Related issues

• Closes TCP/UDP Packages | Syslog Processor #3212

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-29T13:09:08.495+0000

• Duration: 18 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9
Skipped	0
Total	9

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 3.521
Classes	100.0% (0/0)	💚 3.521
Methods	100.0% (6/6)	💚 10.749
Lines	100.0% (0/0)	💚 9.17
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

It looks like you will need to add a stub ingest pipeline to ensure that the types are correctly mapped. This looks like it comes from .../reader/syslog which maps the version as an int, which disagrees with ECS. Nope, scrub that. I don't see why this is happening, that is doing exactly what it should be doing. This appears to be due to this.

FAILURE DETAILS: tcp/generic syslog: [0] parsing field value failed: field "log.syslog.version"'s Go type, float64, does not match the expected field type: keyword (field value: 1) ╭─────────┬─────────────┬───────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├─────────┼─────────────┼───────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤ │ tcp │ generic │ system │ tcp │ PASS │ 32.395726252s │ │ tcp │ generic │ system │ syslog │ FAIL: one or more errors found in documents stored in logs-tcp.generic-ep data stream │ 30.910389558s │ │ tcp │ generic │ system │ tls │ PASS │ 30.009228121s │ ╰─────────┴─────────────┴───────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────╯ 

The system test event gets renders so...

{ <snip> "input": { "type": "tcp" }, "log": { "source": { "address": "192.168.48.4:51130" }, "syslog": { "appname": "testproductname", "facility": { "code": 16, "name": "local0" }, "hostname": "testhostname", "priority": 134, "procid": "1930", "severity": { "code": 6, "name": "Informational" }, "version": 1 ← rendered as a number } }, "message": "some longer testmessage. - a {2:2}." } 

The udp package has the same behaviour (omitted the event which looks essentially the same, as expected).

FAILURE DETAILS: udp/generic syslog: [0] parsing field value failed: field "log.syslog.version"'s Go type, float64, does not match the expected field type: keyword (field value: 1) ╭─────────┬─────────────┬───────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├─────────┼─────────────┼───────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤ │ udp │ generic │ system │ udp │ PASS │ 39.218767859s │ │ udp │ generic │ system │ syslog │ FAIL: one or more errors found in documents stored in logs-udp.generic-ep data stream │ 31.048586593s │ ╰─────────┴─────────────┴───────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────╯ 

[taylor-swanson]

It looks like you will need to add a stub ingest pipeline to ensure that the types are correctly mapped. This looks like it comes from .../reader/syslog which maps the version as an int, which disagrees with ECS. Nope, scrub that. I don't see why this is happening, that is doing exactly what it should be doing. This appears to be due to this.

FAILURE DETAILS: tcp/generic syslog: [0] parsing field value failed: field "log.syslog.version"'s Go type, float64, does not match the expected field type: keyword (field value: 1) ╭─────────┬─────────────┬───────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├─────────┼─────────────┼───────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤ │ tcp │ generic │ system │ tcp │ PASS │ 32.395726252s │ │ tcp │ generic │ system │ syslog │ FAIL: one or more errors found in documents stored in logs-tcp.generic-ep data stream │ 30.910389558s │ │ tcp │ generic │ system │ tls │ PASS │ 30.009228121s │ ╰─────────┴─────────────┴───────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────╯ 

The system test event gets renders so...

{ <snip> "input": { "type": "tcp" }, "log": { "source": { "address": "192.168.48.4:51130" }, "syslog": { "appname": "testproductname", "facility": { "code": 16, "name": "local0" }, "hostname": "testhostname", "priority": 134, "procid": "1930", "severity": { "code": 6, "name": "Informational" }, "version": 1 ← rendered as a number } }, "message": "some longer testmessage. - a {2:2}." } 

The udp package has the same behaviour (omitted the event which looks essentially the same, as expected).

FAILURE DETAILS: udp/generic syslog: [0] parsing field value failed: field "log.syslog.version"'s Go type, float64, does not match the expected field type: keyword (field value: 1) ╭─────────┬─────────────┬───────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├─────────┼─────────────┼───────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤ │ udp │ generic │ system │ udp │ PASS │ 39.218767859s │ │ udp │ generic │ system │ syslog │ FAIL: one or more errors found in documents stored in logs-udp.generic-ep data stream │ 31.048586593s │ ╰─────────┴─────────────┴───────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────╯ 

What version of the stack are you running? This was an issue in 8.2.0 that was fixed in 8.2.1 (which is why the minimum kibana version is at 8.2.1). I think this only happens when using elastic-package test? elastic-package must ignore package constraints, then. This shouldn't happen with 8.2.1 and above. I suppose we could add convert processors into affected pipelines to get around this, but I'm not convinced it's worth the effort.

[efd6]

What version of the stack are you running? This was an issue in 8.2.0 that was fixed in 8.2.1 (which is why the minimum kibana version is at 8.2.1).

That would explain it. (confirmed)