[cisco_duo] Avoid obsolete cursor data in activity, telephony_v2

[chrisberkhout]

Proposed commit message

[cisco_duo] Avoid obsolete cursor data in activity, telephony_v2 In an earlier version of the CEL code for `activity`[1] and `telephony_v2`[2], `cursor.last_published` was set to a UNIX timestamp value. This was changed to use RFC3339 formatted times in later PRs[3][4] (with the corresponding read-time parsing added in [5]). Users who didn't create a new policy may have the current parsing logic fail when it encounters an old UNIX timestamp value in `cursor.last_published`. This PR addresses that issue by renaming `cursor.last_published` to `cursor.last_response_ts`. That effectively clears the cursor so that obsolete values will not be seen. [1]: https://github.com/elastic/integrations/blob/2ea993/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs#L111-L114 [2]: https://github.com/elastic/integrations/blob/2ea993/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs#L112-L115 [3]: https://github.com/elastic/integrations/pull/11640 [4]: https://github.com/elastic/integrations/pull/11670 [5]: https://github.com/elastic/integrations/pull/11772 

Discussion

The API documentation's description of the ts field remains consistent with #11772.

An alternative to this PR would be to parse both legacy and new data, as follows:

diff --git a/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs b/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs index c03f4a1ed3..ffea8d42de 100644 --- a/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs +++ b/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs @@ -39,7 +39,10 @@ program: | state.with({ "mintime": string(1000 * int( state.?cursor.last_published.optMap(t, - t.parse_time(time_layout.RFC3339Nano) + !is_error(int(t)) ? + timestamp(int(t)/1000)+duration(string(int(t)%1000)+"ms") + : + t.parse_time(time_layout.RFC3339Nano) ).orValue( now - duration(state.initial_interval) ) diff --git a/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs b/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs index 553258248b..000c160d67 100644 --- a/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs +++ b/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs @@ -39,7 +39,10 @@ program: | state.with({ "mintime": string(1000 * int( state.?cursor.last_published.optMap(t, - t.parse_time(time_layout.RFC3339Nano) + !is_error(int(t)) ? + timestamp(int(t)/1000)+duration(string(int(t)%1000)+"ms") + : + t.parse_time(time_layout.RFC3339Nano) ).orValue( now - duration(state.initial_interval) )

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Relates [Cisco Duo] - Added fix for 400 and 401 errors in telephony v2 data stream #11640
• Relates cisco_duo: fix 400 and 401 errors in Auth and Activity #11670
• Relates cisco_duo: fix the parsing of last_published timestamp #11772

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

The naming here makes me feel a little icky; last_published_ts or last_response_ts?

[chrisberkhout]

The naming here makes me feel a little icky; last_published_ts or last_response_ts?

Sure. Now it's last_response_ts.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d933886246fb0e1ea76329abc9fad7f4854d0677

History

• 💚 Build #21054 succeeded 335ef0e7ceff3044d56e27997936c53c525c9880

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[efd6]

Thanks

[elastic-vault-github-plugin-prod]

Package cisco_duo - 2.3.2 containing this change is available at https://epr.elastic.co/package/cisco_duo/2.3.2/