Simulate API: Return 400 on invalid processor(s) #130325
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
e.g. missing processor or invalid grok pattern Closes elastic#120731
elasticsearchmachine added v9.2.0 external-contributor 
mattc58 removed the external-contributor 
szybia commented Jul 2, 2025
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/simulate.ingest/10_basic.yml Show resolved Hide resolved
server/src/main/java/org/elasticsearch/ingest/SimulateIngestService.java Show resolved Hide resolved
server/src/test/java/org/elasticsearch/ingest/SimulateIngestServiceTests.java Outdated Show resolved Hide resolved
server/src/test/java/org/elasticsearch/ingest/SimulateIngestServiceTests.java Outdated Show resolved Hide resolved
szybia commented Jul 2, 2025
modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/120_grok.yml Show resolved Hide resolved
Member
PeteGillinElastic left a comment
PeteGillinElastic left a comment There was a problem hiding this comment.
This mostly LGTM. Not approving 'cos we want to wait for BCC.
server/src/test/java/org/elasticsearch/ingest/SimulateIngestServiceTests.java Outdated Show resolved Hide resolved
server/src/test/java/org/elasticsearch/ingest/SimulateIngestServiceTests.java Outdated Show resolved Hide resolved
* upstream/main: (822 commits) Improve Semantic Text Exists Query Tests (elastic#132283) Make hierarchical k-means over centroids cheaper (elastic#132316) Remove unnecessary listener.delegateFailure in IndexShard#ensureMutable (elastic#132294) Add missing release note (elastic#132319) Unmute elastic#131803 (elastic#132295) Include bytes for live docs in ShardFieldStats (elastic#132232) Fix default missing index sort value of data_nanos pre 7.14 (elastic#132162) [DiskBBQ] Quantize centroids using 7 bits instead of 4 bits (elastic#132261) Use panamized version for windows in Int7VectorScorer (elastic#132311) Mute org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT testTooManyByAndOverFields elastic#132310 Mute org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT testManyDistinctOverFields elastic#132308 Update 8.17 version to 8.17.10 (elastic#132303) Mute org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT test {p0=data_stream/10_basic/Create hidden data stream with match all template} elastic#132298 Add random queries to logsdb data generation tests (elastic#132109) ES|QL Inference runner refactoring (elastic#131986) Add basic example to linear-retriever.md (elastic#132196) Refactor RemoteClusterService to be multi-project aware (elastic#131894) ESQL: Mark csv-spec tests (elastic#132098) Mute org.elasticsearch.common.logging.JULBridgeTests testThrowable elastic#132280 Bump versions after 8.19.0 release ...
* upstream/main: (278 commits) ESQL - dense vector support cosine normalization (elastic#132721) [ML] Add support for dimensions in google vertex ai request (elastic#132689) ESQL - Add byte element support for dense_vector data type (elastic#131863) ESQL: Fix async operator warnings not always sent when blocking (elastic#132744) Method not needed anymore (elastic#132912) [Test] Excercise shutdown more reliably in snapshot stress IT (elastic#132909) Update Gradle shadow plugin to 9.0.1 (elastic#132637) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/410_named_queries/named_queries_with_score} elastic#132906 Update docker.elastic.co/wolfi/chainguard-base-fips:latest Docker digest to fa6cb69 (elastic#132735) Remove unnecessary calls to fold() (elastic#131870) Use consistent terminology for transport version resources/references (elastic#132882) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search.vectors/40_knn_search_cosine/kNN search only regular query} elastic#132890 Finalize release notes for v9.1.2 release (elastic#132745) Finalize release notes for v9.0.5 release (elastic#132718) Move inner records out of TransportVersionUtils (elastic#132872) Add support for Lookup Join on Multiple Fields (elastic#131559) Bootstrap PR-based benchmarks (elastic#132717) Refactor MetadataIndexTemplateService to use template maps instead of project metadata (elastic#132662) [Gradle] Update nebula ospackage plugin to 12.1.0 (elastic#132640) Mute org.elasticsearch.xpack.esql.CsvTests test {csv-spec:ip.CdirMatchEqualsInsOrs} elastic#132860 ...
* upstream/main: (32 commits) Speed up loading keyword fields with index sorts (elastic#132950) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testSyntheticSourceWithTranslogSnapshot elastic#132964 Simplify EsqlSession (elastic#132848) Implement WriteLoadConstraintDecider#canAllocate (elastic#132041) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/400_synthetic_source/_doc_count} elastic#132965 Switch to PR-based benchmark pipeline defined in ES repo (elastic#132941) Breakdown undesired allocations by shard routing role (elastic#132235) Implement v_magnitude function (elastic#132765) Introduce execution location marker for better handling of remote/local compatibility (elastic#132205) Mute org.elasticsearch.cluster.ClusterInfoServiceIT testMaxQueueLatenciesInClusterInfo elastic#132957 Unmuting simulate index data stream mapping overrides yaml rest test (elastic#132946) Remove CrossClusterCancellationIT.createLocalIndex() (elastic#132952) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testFetch elastic#132956 Fix failing UT by adding a required capability (elastic#132947) Precompute the BitsetCacheKey hashCode (elastic#132875) Adding simulate ingest effective mapping (elastic#132833) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testFetchMany elastic#132948 Rename skipping logic to remove hard link to skip_unavailable (elastic#132861) Store ignored source in unique stored fields per entry (elastic#132142) Add random tests with match_only_text multi-field (elastic#132380) ...
szybia added :Data Management/Ingest Node 
Collaborator
| Hi @szybia, I've updated the changelog YAML for you. |
* upstream/main: (58 commits) ESQL: mark LOOKUP JOIN as ExecutesOn.Any by default (elastic#133064) Fix 404s in REST API landing page (elastic#133086) Fix release tests for OptimizerVerificationTests (elastic#133100) Make Glob non-recursive (elastic#132798) Update ES|QL function list for release versions (elastic#133096) Split transport version func test into abstract base (elastic#133035) Omit project ID from snapshot metrics (elastic#133098) Mute org.elasticsearch.xpack.esql.analysis.AnalyzerTests testNoDenseVectorFailsForMagnitude elastic#133013 Mute org.elasticsearch.xpack.esql.optimizer.OptimizerVerificationTests testRemoteEnrichAfterCoordinatorOnlyPlans elastic#133015 Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/160_exists_query/Test exists query on _id field} elastic#133097 Rename initial to unreferenced in transport versions (elastic#133082) Rename exception type header (elastic#133045) ESQL: Pluggable tests for Operator status (elastic#132876) ESQL: Mark new signatures in MIN and MAX (elastic#132980) Don't try to serialize half-baked cluster info (elastic#132756) migrate ml_rollover_legacy_indices transport version (elastic#133008) Enable `exclude_source_vectors` by default for new indices (elastic#131907) Expose APIs needed by flush during translog replay (elastic#132960) Change reporting_user role to leverage reserved kibana privileges (elastic#132766) Update TasksIT for batched execution (elastic#132762) ...
szybia commented Aug 19, 2025
rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/simulate.ingest/10_basic.yml Show resolved Hide resolved
elasticsearchmachine added the Team:Data Management Collaborator
| Pinging @elastic/es-data-management (Team:Data Management) |
* upstream/main: (58 commits) Fixing flaky LoggedExec (tests) (elastic#133215) CPS search should not use `skip_unavailable` (elastic#132927) Don't fail search if bottom doc can't be formatted (elastic#133188) Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupByNothing elastic#133225 Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedSetDocValuesWithSkipperSmall elastic#133224 Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedNumberMergeAwayAllValuesWithSkipper elastic#133223 Adding support for index.number_of_replicas to data stream settings (elastic#132748) Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedDocValuesSingleUniqueValue elastic#133221 Fix VectorSimilarityFunctionsIT (elastic#133206) Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupBySubset elastic#133220 Increase the number of FORK branches in ForkGenerator (elastic#132019) Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=STORED]} elastic#133218 Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=DOC_VALUES]} elastic#133217 Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=NONE]} elastic#133216 Set default processor allocation for test clusters (elastic#133204) Add mapper for exponential histograms (elastic#132493) Fix offset handling in Murmur3Hasher (elastic#133193) unmute testDoesNotResolveClosedIndex (elastic#133115) Fix an AWS SDK v2 release note (elastic#133155) Limit the depth of a filter (elastic#133113) ...
PeteGillinElastic approved these changes Aug 21, 2025
Member
PeteGillinElastic left a comment
PeteGillinElastic left a comment There was a problem hiding this comment.
I think we're finally there! Just one comment, probably more for information than to be changed in this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Return a HTTP 400 status code rather than a 500 on missing processor or invalid grok pattern within simulate API
Closes #120731