Skip to content

azure: add related.entity field to auditlogs and signinlogs default ingest pipeline #11344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Nov 5, 2024
Merged

azure: add related.entity field to auditlogs and signinlogs default ingest pipeline #11344

merged 7 commits into from
Nov 5, 2024

Conversation

@orouz
Copy link
Contributor

@orouz orouz commented Oct 7, 2024 •
edited
Loading

Proposed commit message

adds processors to auditlogs and signinlogs ingest pipelines that appends all entity identifiers from the log event to related.entity in order to facilitate pivoting around a piece of data

Context

this PR is part of the cloud security CDR epic. it populates related.entity which is an upcoming ECS field

benchmark (100 runs)

  1. auditlogs
Metric     
Description Main append + painless script processor
Commit 63854f0 01e0f3c
Average EPS 16409.48978 15400.45392
Stddev 1273.233006 1197.630202
Min 8196.721311 10416.66667
Max 17857.14286 16949.15254
EPS Change to baseline   -6.15%
  1. signinlogs
Metric     
Description Main append processor
Commit 63854f0 de9827c
Average EPS 9641.707196 9561.806718
Stddev 701.7646604 861.931866
Min 7142.857143 5405.405405
Max 11363.63636 10989.01099
EPS Change to baseline   -0.83%

Related issues
@orouz orouz added enhancement New feature or request Integration:azure Azure Logs Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Oct 7, 2024
@orouz orouz force-pushed the azure_entra_related_entity branch from 5c5457a to 62557f7 Compare October 7, 2024 11:23
@orouz orouz self-assigned this Oct 7, 2024
@orouz orouz changed the title Add related.entity field to azure auditlogs and signinlogs default ingest pipeline azure: add related.entity field to auditlogs and signinlogs default ingest pipeline Oct 20, 2024
@orouz orouz force-pushed the azure_entra_related_entity branch 2 times, most recently from c0a0273 to 30a8c39 Compare October 20, 2024 12:03
@orouz orouz force-pushed the azure_entra_related_entity branch from 30a8c39 to 06cb8f6 Compare October 20, 2024 12:05
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Oct 20, 2024 •
edited
Loading

🚀 Benchmarks report

Package azure 👍(4) 💚(4) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
signinlogs 2392.34 1937.98 -454.36 (-18.99%) 💔
eventhub 500000 333333.33 -166666.67 (-33.33%) 💔
firewall_logs 1869.16 1406.47 -462.69 (-24.75%) 💔

To see the full report comment with /test benchmark fullreport
@orouz
Copy link
Contributor Author

orouz commented Oct 20, 2024

/test benchmark fullreport
@orouz orouz marked this pull request as ready for review October 20, 2024 13:07
@orouz orouz requested review from a team as code owners October 20, 2024 13:07
efd6
efd6 reviewed Oct 20, 2024
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orouz Please revise the proposed commit message to be a stand-alone text that explains what is being done and why. It should preferably not refer to internal issues since it will be read by people who may not have access to those documents. Also note that git commit messages are not markdown, so do not use markdown features.
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Oct 21, 2024

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
81.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
@elasticmachine
Copy link

elasticmachine commented Oct 21, 2024

💚 Build Succeeded

History

cc @orouz
@orouz orouz requested a review from efd6 October 27, 2024 09:32
@orouz orouz merged commit 705c44f into elastic:main Nov 5, 2024
5 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Nov 5, 2024

Package azure - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=azure
@nicpenning
Copy link
Contributor

nicpenning commented Nov 6, 2024

related.entity doesn't appear to be a field that will be in compliance with ECS.

https://www.elastic.co/guide/en/ecs/current/ecs-related.html

I have an integration going through the works right now that would like to use something like related.file.name but was directed to creating an RFC for adopting that field.

Will this cause issues down the road or those that use the Data Quality dashboard to reveal actual ECS fields? Is there already a precedent set here that any field can live beneath related.? Thoughts?
@orouz
Copy link
Contributor Author

orouz commented Jan 7, 2025

related.entity doesn't appear to be a field that will be in compliance with ECS.

it's currently being worked on at elastic/ecs#2360

Will this cause issues down the road or those that use the Data Quality dashboard to reveal actual ECS fields? Is there already a precedent set here that any field can live beneath related.? Thoughts?

@tinnytintin10 can comment more on that.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@orouz
azure: add related.entity field to auditlogs and signinlogs default i…
f2f0f01 
…ngest pipeline (elastic#11344)
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@orouz
azure: add related.entity field to auditlogs and signinlogs default i…
65732c7 
…ngest pipeline (elastic#11344)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:azure Azure Logs Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

7 participants

@orouz @elasticmachine @nicpenning @zmoog @kaiyan-sheng @efd6 @muthu-mps