[aws] [cloudwatch_metrics] Map aws.dimensions field as object (backport of #11883)

[zmoog]

Proposed commit message

Change the mapping type for the aws.dimensions field from flattened to object.

Currently, all *_metrics data streams but one use the object mapping. The cloudwatch_metrics data stream uses the flattened type instead.

We need to unify the mapping of aws.dimensions across all metrics-related data streams in the AWS integration.
If all data streams use the exact mapping for aws.dimensions, users will be able to query and build a dashboard that correlates data across different data streams.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Run the test scenario on 8.15.4

Related issues

• relates Bring uniformity in aws.dimensions.* fields mapping #11806
• backports [aws] [cloudwatch_metrics] Map aws.dimensions field as object #11883

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[zmoog]

I tested the upgrade from AWS integration 2.30.2 to 2.30.3 (the unreleased changes from this PR) with the following steps:

• Started a brand new local stack (8.15.4)
• Installed AWS integration 2.30.2 (with aws.dimensions as flattened)
• Started sending 1 document every 5 secs, including a field containing a sequence number
• Upgraded the AWS integration to 2.30.3 (unreleased, this PR)
• Waited for the rollout to take effect (checking settings.index.time_series | .start_time, .end_time)
• Checked that the the data stream didn't lose any sequence number

More details on selected steps.

Started sending 1 document every 5 secs

• create an API key on the local stack
• export env vars for the es tool.

Set up the es tool config:

export ELASTICSEARCH_ENDPOINTS="https://localhost:9200" export ELASTICSEARCH_API_KEY="[redacted, event if it is not need since it's local]"

I used the following shell script:

sequence=0 while true do cat > metrics.json <<EOF {  "@timestamp": "$(date '+%Y-%m-%dT%H:%M:%S%z')",  "aws": {  "dimensions": {  "name": "Maurizio Branca",  "AutoScalingGroupName": "whatever"  },  "metric": {  "cpu": 10,  "sequence": $sequence  }  } }  EOF ((sequence++)) cat metrics.json | jq -c | es docs bulk -f - -i metrics-aws.cloudwatch_metrics-sdh5390 sleep 5 done

Results in:

2025/01/07 12:54:11 adding a new document: {"@timestamp":"2025-01-07T12:54:11+0100","aws":{"dimensions":{"name":"Maurizio Branca","AutoScalingGroupName":"whatever"},"metric":{"cpu":10,"sequence":0}}} 2025/01/07 12:54:11 closing bulk indexer 2025/01/07 12:54:11 Successfully indexed document 2025/01/07 12:54:11 bulk indexer closed 2025/01/07 12:54:11 getting bulk indexer stats 2025/01/07 12:54:11 Stats: {NumAdded:1 NumFlushed:1 NumFailed:0 NumIndexed:0 NumCreated:1 NumUpdated:0 NumDeleted:0 NumRequests:1}

The scripts sends a document like the following every 5 secs:

{ "@timestamp": "2024-12-31T00:14:58+0100", "aws": { "dimensions": { "name": "Maurizio Branca", "AutoScalingGroupName": "whatever" }, "metric": { "cpu": 10, "sequence": 270 } } }

Upgraded the AWS integration to 2.30.3 (unreleased, this PR)

Upgrade the AWS integration package from 2.30.2 to 2.30.3.

Waited for the rollout to take effect

Right after the upgrade, Fleet/ES creates a new -000002 index, but keeps writing to the -000001 index until the settings.index.time_series.end_time elapses.

Old index -000001:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings { ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000001": { "settings": { "index": { "mapping": { "total_fields": { "limit": "1000", "ignore_dynamic_beyond_limit": "true" } }, "hidden": "true", "time_series": { "end_time": "2025-01-07T12:33:16.000Z", "start_time": "2025-01-07T09:54:11.000Z" },

New index -000002:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings { ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000002": { "settings": { "index": { "mapping": { "total_fields": { "limit": "1000", "ignore_dynamic_beyond_limit": "true" } }, "hidden": "true", "time_series": { "end_time": "2025-01-07T13:03:16.000Z", "start_time": "2025-01-07T12:33:16.000Z" },

Now I need to wait until 2025-01-07T12:33:16.000Z (UTC) to see if the ES smoothly transitions from index -000001 to -000002.

Checked that the the data stream didn't lose any sequence number

At 2025-01-07T12:33:16.000Z, ES successfully transitioned from index -000001 to -000002 and from flattened to object field mapping.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
20.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 235d140

History

• 💔 Build #20092 failed 10f269f
• 💚 Build #20076 succeeded 63ae257

cc @zmoog

[muthu-mps]

LGTM!

[elastic-vault-github-plugin-prod]

Package aws - 2.30.3 containing this change is available at https://epr.elastic.co/package/aws/2.30.3/