Pipelines: Add created_date and modified_date #130847
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
szybia force-pushed the index-pipeline-tracking branch 3 times, most recently from 5efc252 to 1b30775 Compare 
szybia force-pushed the index-pipeline-tracking branch from 91e42f2 to 3f46277 Compare 
szybia commented Jul 11, 2025
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/simulate.ingest/10_basic.yml Outdated Show resolved Hide resolved
server/src/internalClusterTest/java/org/elasticsearch/ingest/PipelineConfigurationSyncIT.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/action/ingest/ReservedPipelineAction.java Outdated Show resolved Hide resolved
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/ingest/10_basic.yml Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/PipelineConfiguration.java Show resolved Hide resolved
szybia commented Jul 11, 2025
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
szybia added >enhancement :Data Management/Ingest Node 
elasticsearchmachine added the Team:Data Management 
Collaborator
| Pinging @elastic/es-data-management (Team:Data Management) |
Collaborator
| Hi @szybia, I've updated the changelog YAML for you. |
Member
PeteGillinElastic left a comment
PeteGillinElastic left a comment There was a problem hiding this comment.
Sorry, I don't think I'm going to finish this review before EOD, so I figured I'd publish the comments I've got so far.
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/ingest/10_basic.yml Outdated Show resolved Hide resolved
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/ingest/10_basic.yml Outdated Show resolved Hide resolved
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/ingest/10_basic.yml Outdated Show resolved Hide resolved
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/simulate.ingest/10_basic.yml Outdated Show resolved Hide resolved
server/src/internalClusterTest/java/org/elasticsearch/ingest/PipelineConfigurationSyncIT.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/action/ingest/SimulateExecutionService.java Outdated Show resolved Hide resolved
Member
I'll get back to this on Monday. |
…king * upstream/main: (33 commits) Allow both WithEntitlementsOnTestCode and EntitledTestPackages together (elastic#130826) Move streams status actions to cluster:monitor group (elastic#131015) Update JDK base image for OIDC fixture (elastic#131176) Mute org.elasticsearch.xpack.esql.ccq.MultiClustersIT testLookupJoinAliases elastic#131166 Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testEnqueuedMergeTasksAreUnblockedWhenEstimatedMergeSizeChanges elastic#131165 Mute org.elasticsearch.xpack.esql.ccq.MultiClustersIT testNotLikeListKeyword elastic#131155 Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test elastic#131154 Check file entitlements on the Lucene FilterFileSystem in tests (elastic#130825) Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {lookup-join.MvJoinKeyOnFromAfterStats ASYNC} elastic#131148 Move FrequencyCappedAction to common package (elastic#131060) Mute org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT testStopQueryLocal elastic#121672 Remove nesting from multi allocation decision (elastic#130844) Disable async search rest tests in release builds (elastic#131132) Fix testStopQueryLocal (elastic#131130) Fixes based on resharding disruption tests (elastic#130870) Remove inactive logger (elastic#131121) Add wait for remote start for the test (elastic#131124) Add existing shards allocator settings to failure store allowed list. (elastic#131056) Don't allow field caps to use semantic queries as index filters (elastic#131111) issue should be already fixed by elastic#121466 (elastic#130860) ...
Collaborator
| Hi @szybia, I've updated the changelog YAML for you. |
Member
PeteGillinElastic left a comment
PeteGillinElastic left a comment There was a problem hiding this comment.
Thanks, this is definitely in pretty decent shape, I just have a few suggested tweaks.
server/src/main/java/org/elasticsearch/action/ingest/SimulateExecutionService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/PipelineConfiguration.java Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/PipelineConfiguration.java Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/TrackingResultProcessor.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/rest/action/ingest/RestPutPipelineAction.java Show resolved Hide resolved
server/src/test/java/org/elasticsearch/ingest/IngestServiceTests.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
…king * upstream/main: (91 commits) Mute org.elasticsearch.packaging.test.DockerTests test130JavaHasCorrectOwnership elastic#131369 Add exception logging when interrupted (elastic#131153) Mute org.elasticsearch.packaging.test.DockerTests test140CgroupOsStatsAreAvailable elastic#131372 Mute org.elasticsearch.packaging.test.DockerTests test070BindMountCustomPathConfAndJvmOptions elastic#131366 Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/delete_expired_data/Test delete expired data with body parameters} elastic#131364 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectorAndField {functionName=v_cosine similarityFunction=COSINE} elastic#131363 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testDifferentDimensions {functionName=v_cosine similarityFunction=COSINE} elastic#131362 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectors {functionName=v_cosine similarityFunction=COSINE} elastic#131361 Check SCORE_FUNCTION capability in VerifierTests (elastic#131352) Replace deprecated routingTable table call in tests (elastic#131005) [DOCS] Remove misused applies_to tag (elastic#131349) Adj ivf postings list building (elastic#130843) [Transform] Read metadata from Project State (elastic#131205) Add note on o11y to architecture guide (elastic#131291) Upgrade AWS Java SDK to 2.31.78 (elastic#131050) Support Fields API in conditional ingest processors (elastic#121914) ESQL - KNN function uses prefilters when pushed down to Lucene (elastic#131004) Add docs for ES|QL query logs (elastic#131287) Simplify `expectedFinalRegisterValue` computation (elastic#131274) Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing, inner_hits and maxConcurrentGroupRequests} elastic#131348 ...
…king * upstream/main: Mark watcher NotMultiProjectCapable and replace deprecated multi-project methods (elastic#131313) Enable force inference endpoint deleting for invalid models and after stopping model deployment fails (elastic#129090) [ML] Remove SageMaker Elastic updates (elastic#131301) Refactor AsyncSearchErrorTraceIT to use assertBusy (elastic#131328)
szybia commented Jul 17, 2025
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
PeteGillinElastic approved these changes Jul 21, 2025
Member
PeteGillinElastic left a comment
PeteGillinElastic left a comment There was a problem hiding this comment.
Thanks @szybia . This LGTM to me, with just a few final nits. I'm happy for you to tidy those up before you merge. Or you can get another review if you'd like more feedback as a learning process.
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
…king * upstream/main: (100 commits) Term vector API on stateless search nodes (elastic#129902) TEST Fix ThreadPoolMergeSchedulerStressTestIT testMergingFallsBehindAndThenCatchesUp (elastic#131636) Add inference.put_custom rest-api-spec (elastic#131660) ESQL: Fewer serverless docs in tests (elastic#131651) Skip search on indices with INDEX_REFRESH_BLOCK (elastic#129132) Mute org.elasticsearch.indices.cluster.RemoteSearchForceConnectTimeoutIT testTimeoutSetting elastic#131656 [jdk] Resolve EA OpenJDK builds to our JDK archive (elastic#131237) Add optimized path for intermediate values aggregator (elastic#131390) Correctly handling download_database_on_pipeline_creation within a pipeline processor within a default or final pipeline (elastic#131236) Refresh potential lost connections at query start for `_search` (elastic#130463) Add template_id to patterned-text type (elastic#131401) Integrate LIKE/RLIKE LIST with ReplaceStringCasingWithInsensitiveRegexMatch rule (elastic#131531) [ES|QL] Add doc for the COMPLETION command (elastic#131010) ESQL: Add times to topn status (elastic#131555) ESQL: Add asynchronous pre-optimization step for logical plan (elastic#131440) ES|QL: Improve generative tests for FORK [130015] (elastic#131206) Update index mapping update privileges (elastic#130894) ESQL: Added Sample operator NamedWritable to plugin (elastic#131541) update `kibana_system` to grant it access to `.chat-*` system index (elastic#131419) Clarify heap size configuration (elastic#131607) ...
szybia commented Jul 22, 2025
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/ingest/20_tracking.yml Show resolved Hide resolved
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds system-managed created_date and modified_date properties to ingest pipelines to track when pipelines are created and last modified. The implementation includes validation to prevent users from manually setting these system properties, transport version compatibility for backwards compatibility, and comprehensive test coverage.
- Adds
created_dateandmodified_datefields to Pipeline class with automatic timestamp management - Implements validation to reject user-provided system properties in pipeline configurations
- Adds transport version compatibility handling to exclude new properties when communicating with older nodes
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Pipeline.java | Core implementation adding created/modified date fields and parsing logic |
| IngestService.java | Business logic for timestamp management, validation, and cluster state updates |
| PipelineConfiguration.java | Transport compatibility handling to exclude new properties for older nodes |
| RestPutPipelineAction.java | REST API capability declaration for pipeline tracking |
| TransportVersions.java | New transport version constant for feature compatibility |
| IngestServiceTests.java | Comprehensive test coverage for timestamp tracking functionality |
| PipelineProcessorTests.java | Test updates for new Pipeline constructor parameters |
| TrackingResultProcessor.java | Pipeline creation updates to include timestamp fields |
| SimulateExecutionService.java | Pipeline creation updates for simulation service |
| SimulateIngestService.java | Validation integration for pipeline substitutions |
| Test files | Integration tests for new tracking functionality and validation |
| Changelog | Documentation of the enhancement |
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
dakrone requested changes Jul 22, 2025
Member
dakrone left a comment
dakrone left a comment There was a problem hiding this comment.
Thanks for working on this Szymon, I think it looks pretty good. The only concern I have is around storing the time as a formatted string in the config rather than a number value.
server/src/main/java/org/elasticsearch/ingest/IngestService.java Outdated Show resolved Hide resolved
…king * upstream/main: (90 commits) Register a blob cache long counter metric for total evicted regions (elastic#131862) Move plan attribute resolution to its own component (elastic#131830) Make restore support multi-project (elastic#131661) Use logically more correct expression (elastic#131869) [ES|QL] Change equals and hashcode for ConstantNullBlock (elastic#131817) Update `TransportVersion` to support a new model (elastic#131488) Correct slow log user for RCS 2.0 (elastic#130140) Revert "Remove 8.17 from dev branches" Mute org.elasticsearch.compute.aggregation.ValuesBytesRefGroupingAggregatorFunctionTests testSomeFiltered elastic#131878 Remove 8.17 from dev branches Revert "CompressorFactory.compressor (elastic#131655)" (elastic#131866) Add fast path for single value in VALUES aggregator (elastic#130510) Resolve inference release tests failing due to missing feature flag (elastic#131841) [Docs] Replace placeholder URLs (elastic#131309) CompressorFactory.compressor (elastic#131655) add availability info for speed loading setting (elastic#131714) [Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. (elastic#131486) Migrate x-pack-enrich legacy rest tests to new test framework (elastic#131743) Fix plugin example test failures due to deprecation warning (elastic#131819) Remove deprecated function isNotNullAndFoldable (elastic#130944) ...
…king * upstream/main: Fix score computation in ES91Int4VectorsScorer (elastic#131905)
dakrone approved these changes Jul 25, 2025
Member
dakrone left a comment
dakrone left a comment There was a problem hiding this comment.
LGTM, I left one really minor comment about javadoc
| } | ||
| } | ||
| | ||
| private Map<String, Object> configForTransport(final TransportVersion transportVersion) { |
Member
There was a problem hiding this comment.
Can you add a javadoc to this method to explain what it does?
Contributor Author
There was a problem hiding this comment.
sure thing! let me just address in next PR so i don't have to wait for this build again...
szybia added a commit to szybia/elasticsearch that referenced this pull request Jul 25, 2025
…-tracking * upstream/main: (106 commits) Pipelines: Add `created_date` and `modified_date` (elastic#130847) add thread pool change availability (elastic#131734) Add failure store availability info / and port over privileges (elastic#131729) add availability information for ssl handshake timeout settings (elastic#131786) add availability information for rescore_vector (elastic#131710) add availability to oversample value of 0 (elastic#131707) clarify hnsw filter heuristic setting availability (elastic#131715) add availability info for default heap dump path change (elastic#131713) clarify default algorithms per stack version (elastic#131728) Refine error messages in `Fork` for correctness and clarity. (elastic#131701) [ES|QL] Replace RoundTo linear search evaluator with manual evaluators (elastic#131733) ESQL: Fix buildParams in tests with --configuration-cache (elastic#131826) Unmute `CrossClusterEsqlRCS2EnrichUnavailableRemotesIT#testEsqlEnrichWithSkipUnavailable` (elastic#131916) Allow templates for `.chat-*` index template (elastic#131914) ESQL: Fix NPE on empty to_lower/to_upper call (elastic#131917) Fix score computation in ES91Int4VectorsScorer (elastic#131905) Register a blob cache long counter metric for total evicted regions (elastic#131862) Move plan attribute resolution to its own component (elastic#131830) Make restore support multi-project (elastic#131661) Use logically more correct expression (elastic#131869) ...
flash1293 added a commit to elastic/kibana that referenced this pull request Jul 31, 2025
Fixes #229733 With elastic/elasticsearch#130847 , we track the creation and modification date of ingest pipelines. As part of adjusting processing through the streams API, the ingest pipeline object is loaded, modified and written back. This fails since it tries to set the modified/creation timestamps which isn't allowed. This PR is stripping out the offending keys
delanni pushed a commit to delanni/kibana that referenced this pull request Aug 5, 2025
…230053) Fixes elastic#229733 With elastic/elasticsearch#130847 , we track the creation and modification date of ingest pipelines. As part of adjusting processing through the streams API, the ingest pipeline object is loaded, modified and written back. This fails since it tries to set the modified/creation timestamps which isn't allowed. This PR is stripping out the offending keys
flash1293 added a commit to flash1293/kibana that referenced this pull request Sep 12, 2025
…230053) Fixes elastic#229733 With elastic/elasticsearch#130847 , we track the creation and modification date of ingest pipelines. As part of adjusting processing through the streams API, the ingest pipeline object is loaded, modified and written back. This fails since it tries to set the modified/creation timestamps which isn't allowed. This PR is stripping out the offending keys (cherry picked from commit 4561993) # Conflicts: # x-pack/platform/plugins/private/translations/translations/de-DE.json
flash1293 added a commit to elastic/kibana that referenced this pull request Sep 12, 2025
…230053) (#234916) Fixes #232445 # Backport This will backport the following commits from `main` to `8.19`: - [🌊 Streams: Fix classic streams ingest pipeline manipulation (#230053)](#230053) <!--- Backport version: 10.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Joe Reuter","email":"johannes.reuter@elastic.co"},"sourceCommit":{"committedDate":"2025-07-31T12:58:52Z","message":"🌊 Streams: Fix classic streams ingest pipeline manipulation (#230053)\n\nFixes https://github.com/elastic/kibana/issues/229733\n\nWith elastic/elasticsearch#130847 , we track the\ncreation and modification date of ingest pipelines.\nAs part of adjusting processing through the streams API, the ingest\npipeline object is loaded, modified and written back. This fails since\nit tries to set the modified/creation timestamps which isn't allowed.\n\nThis PR is stripping out the offending keys","sha":"4561993a2d8d50773ee1a2f1bf4ef7dca7908944","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","Team:obs-ux-logs","Feature:Streams","v9.2.0"],"title":"🌊 Streams: Fix classic streams ingest pipeline manipulation","number":230053,"url":"https://github.com/elastic/kibana/pull/230053","mergeCommit":{"message":"🌊 Streams: Fix classic streams ingest pipeline manipulation (#230053)\n\nFixes https://github.com/elastic/kibana/issues/229733\n\nWith elastic/elasticsearch#130847 , we track the\ncreation and modification date of ingest pipelines.\nAs part of adjusting processing through the streams API, the ingest\npipeline object is loaded, modified and written back. This fails since\nit tries to set the modified/creation timestamps which isn't allowed.\n\nThis PR is stripping out the offending keys","sha":"4561993a2d8d50773ee1a2f1bf4ef7dca7908944"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/230053","number":230053,"mergeCommit":{"message":"🌊 Streams: Fix classic streams ingest pipeline manipulation (#230053)\n\nFixes https://github.com/elastic/kibana/issues/229733\n\nWith elastic/elasticsearch#130847 , we track the\ncreation and modification date of ingest pipelines.\nAs part of adjusting processing through the streams API, the ingest\npipeline object is loaded, modified and written back. This fails since\nit tries to set the modified/creation timestamps which isn't allowed.\n\nThis PR is stripping out the offending keys","sha":"4561993a2d8d50773ee1a2f1bf4ef7dca7908944"}}]}] BACKPORT-->
8 tasks
juliaElastic added a commit to elastic/kibana that referenced this pull request Oct 30, 2025
## Summary Closes #230188 Omit system properties when creating ingest pipelines on remote clusters. Currently pipelines syncing fails with the error ``` [2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception Root causes: parse_exception: Provided a pipeline property which is managed by the system: created_date_millis. ``` It seems the new fields were introduced by elasticsearch in 9.2 elastic/elasticsearch#130847 To verify: - Start 2 clusters and set up syncing between them https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Create a custom ingest pipeline e.g. `logs-system.auth@custom` - Wait for sync to run - Verify that the sync is successful ``` # example pipeline { description: 'description', processors: [ { set: [Object] } ], version: 2, created_date_millis: 1761731006028, modified_date_millis: 1761732134830 } [2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom [2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 30, 2025
…#241096) ## Summary Closes elastic#230188 Omit system properties when creating ingest pipelines on remote clusters. Currently pipelines syncing fails with the error ``` [2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception Root causes: parse_exception: Provided a pipeline property which is managed by the system: created_date_millis. ``` It seems the new fields were introduced by elasticsearch in 9.2 elastic/elasticsearch#130847 To verify: - Start 2 clusters and set up syncing between them https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Create a custom ingest pipeline e.g. `logs-system.auth@custom` - Wait for sync to run - Verify that the sync is successful ``` # example pipeline { description: 'description', processors: [ { set: [Object] } ], version: 2, created_date_millis: 1761731006028, modified_date_millis: 1761732134830 } [2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom [2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. (cherry picked from commit 171e720)
kibanamachine added a commit to elastic/kibana that referenced this pull request Oct 30, 2025
…241096) (#241226) # Backport This will backport the following commits from `main` to `9.2`: - [[Fleet] omit system properties when syncing ingest pipelines (#241096)](#241096) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-10-30T07:17:54Z","message":"[Fleet] omit system properties when syncing ingest pipelines (#241096)\n\n## Summary\n\nCloses https://github.com/elastic/kibana/issues/230188\n\nOmit system properties when creating ingest pipelines on remote\nclusters.\nCurrently pipelines syncing fails with the error\n```\n[2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception\n Root causes:\n parse_exception: Provided a pipeline property which is managed by the system: created_date_millis.\n```\nIt seems the new fields were introduced by elasticsearch in 9.2\nhttps://github.com/elastic/elasticsearch/pull/130847\n\nTo verify:\n- Start 2 clusters and set up syncing between them\nhttps://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md\n- Create a custom ingest pipeline e.g. `logs-system.auth@custom`\n- Wait for sync to run\n- Verify that the sync is successful\n\n```\n# example pipeline\n{\n description: 'description',\n processors: [ { set: [Object] } ],\n version: 2,\n created_date_millis: 1761731006028,\n modified_date_millis: 1761732134830\n}\n[2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom\n[2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success\n```\n\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"171e720d805c7f5215c46ed231de302f73560e0d","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:version","v9.3.0","v9.2.1"],"title":"[Fleet] omit system properties when syncing ingest pipelines","number":241096,"url":"https://github.com/elastic/kibana/pull/241096","mergeCommit":{"message":"[Fleet] omit system properties when syncing ingest pipelines (#241096)\n\n## Summary\n\nCloses https://github.com/elastic/kibana/issues/230188\n\nOmit system properties when creating ingest pipelines on remote\nclusters.\nCurrently pipelines syncing fails with the error\n```\n[2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception\n Root causes:\n parse_exception: Provided a pipeline property which is managed by the system: created_date_millis.\n```\nIt seems the new fields were introduced by elasticsearch in 9.2\nhttps://github.com/elastic/elasticsearch/pull/130847\n\nTo verify:\n- Start 2 clusters and set up syncing between them\nhttps://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md\n- Create a custom ingest pipeline e.g. `logs-system.auth@custom`\n- Wait for sync to run\n- Verify that the sync is successful\n\n```\n# example pipeline\n{\n description: 'description',\n processors: [ { set: [Object] } ],\n version: 2,\n created_date_millis: 1761731006028,\n modified_date_millis: 1761732134830\n}\n[2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom\n[2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success\n```\n\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"171e720d805c7f5215c46ed231de302f73560e0d"}},"sourceBranch":"main","suggestedTargetBranches":["9.2"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/241096","number":241096,"mergeCommit":{"message":"[Fleet] omit system properties when syncing ingest pipelines (#241096)\n\n## Summary\n\nCloses https://github.com/elastic/kibana/issues/230188\n\nOmit system properties when creating ingest pipelines on remote\nclusters.\nCurrently pipelines syncing fails with the error\n```\n[2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception\n Root causes:\n parse_exception: Provided a pipeline property which is managed by the system: created_date_millis.\n```\nIt seems the new fields were introduced by elasticsearch in 9.2\nhttps://github.com/elastic/elasticsearch/pull/130847\n\nTo verify:\n- Start 2 clusters and set up syncing between them\nhttps://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md\n- Create a custom ingest pipeline e.g. `logs-system.auth@custom`\n- Wait for sync to run\n- Verify that the sync is successful\n\n```\n# example pipeline\n{\n description: 'description',\n processors: [ { set: [Object] } ],\n version: 2,\n created_date_millis: 1761731006028,\n modified_date_millis: 1761732134830\n}\n[2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom\n[2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success\n```\n\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [ ] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"171e720d805c7f5215c46ed231de302f73560e0d"}},{"branch":"9.2","label":"v9.2.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
sbelastic pushed a commit to sbelastic/kibana that referenced this pull request Oct 30, 2025
…#241096) ## Summary Closes elastic#230188 Omit system properties when creating ingest pipelines on remote clusters. Currently pipelines syncing fails with the error ``` [2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception Root causes: parse_exception: Provided a pipeline property which is managed by the system: created_date_millis. ``` It seems the new fields were introduced by elasticsearch in 9.2 elastic/elasticsearch#130847 To verify: - Start 2 clusters and set up syncing between them https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Create a custom ingest pipeline e.g. `logs-system.auth@custom` - Wait for sync to run - Verify that the sync is successful ``` # example pipeline { description: 'description', processors: [ { set: [Object] } ], version: 2, created_date_millis: 1761731006028, modified_date_millis: 1761732134830 } [2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom [2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.
ana-davydova pushed a commit to ana-davydova/kibana that referenced this pull request Nov 3, 2025
…#241096) ## Summary Closes elastic#230188 Omit system properties when creating ingest pipelines on remote clusters. Currently pipelines syncing fails with the error ``` [2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception Root causes: parse_exception: Provided a pipeline property which is managed by the system: created_date_millis. ``` It seems the new fields were introduced by elasticsearch in 9.2 elastic/elasticsearch#130847 To verify: - Start 2 clusters and set up syncing between them https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Create a custom ingest pipeline e.g. `logs-system.auth@custom` - Wait for sync to run - Verify that the sync is successful ``` # example pipeline { description: 'description', processors: [ { set: [Object] } ], version: 2, created_date_millis: 1761731006028, modified_date_millis: 1761732134830 } [2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom [2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.
albertoblaz pushed a commit to albertoblaz/kibana that referenced this pull request Nov 4, 2025
…#241096) ## Summary Closes elastic#230188 Omit system properties when creating ingest pipelines on remote clusters. Currently pipelines syncing fails with the error ``` [2025-10-29T10:50:06.429+01:00][ERROR][plugins.fleet.fleet:sync-integrations-task:1.0.5] Failed to install ingest_pipeline logs-system.auth@custom, error: ResponseError: parse_exception Root causes: parse_exception: Provided a pipeline property which is managed by the system: created_date_millis. ``` It seems the new fields were introduced by elasticsearch in 9.2 elastic/elasticsearch#130847 To verify: - Start 2 clusters and set up syncing between them https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Create a custom ingest pipeline e.g. `logs-system.auth@custom` - Wait for sync to run - Verify that the sync is successful ``` # example pipeline { description: 'description', processors: [ { set: [Object] } ], version: 2, created_date_millis: 1761731006028, modified_date_millis: 1761732134830 } [2025-10-29T11:03:38.922+01:00][DEBUG][plugins.fleet.fleet:sync-integrations-task:1.0.5] Updating ingest pipeline: logs-system.auth@custom [2025-10-29T11:11:36.343+01:00][INFO ][plugins.fleet.fleet:sync-integrations-task:1.0.5] [SyncIntegrationsTask] runTask ended: success ``` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Add new system-managed properties to pipelines:
created_date: when the pipeline with a given ID was createdmodified_date: when the pipeline was updatedRelates to: #108754