Add health_status field to status change logs data stream

[MichelLosier]

Proposed commit message

• Adds a new health_status field to the status change logs data stream that uses rollup health statuses as seen in the Fleet UI and agent status metrics: https://www.elastic.co/docs/reference/fleet/monitor-elastic-agent#view-agent-status
• Improves system metrics alerting rules by sourcing from a more specific index, removing RLIKE usage and using the system.process.cpu.total.normalized.pct field for CPU usage.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Test the pipeline:

Run ES

elastic-package stack up -d --services=elasticsearch 

From the elastic_agent package dir run:

elastic-package test pipeline 

Try out the package:

From the elastic_package dir:

elastic-package build 

• Upload the zip as an integration in kibana
• Start and stop an agent
• Review the status_change logs
• Review the new alert rules in Observability > Alerts > Rule management

Related issues

• Closes: https://github.com/elastic/ingest-dev/issues/6319

Other notes

An alternative approach to this task would've been to update the data stream here, and also update the background task in Kibana that provides the logs for this data stream. Using a pipeline processor instead allows us to avoid version constraints on the package to a kibana version for what is a derived field

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[nchaulet]

LGTM 🚀

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: cbe82fb

History

• 💚 Build #33847 succeeded df983a8
• 💚 Build #33663 succeeded 067ae45
• 💚 Build #33626 succeeded a0c2937
• 💚 Build #33621 succeeded ecb6075
• 💔 Build #33618 failed 3d4919f
• 💔 Build #33615 failed 425b6a3

[elastic-vault-github-plugin-prod]

Package elastic_agent - 2.6.8 containing this change is available at https://epr.elastic.co/package/elastic_agent/2.6.8/