Loading

Reindex a time series data stream

Serverless Stack

Reindexing allows you to copy documents from an existing time series data stream (TSDS) to a new one. All data streams support reindexing, but time series data streams require special handling due to their time-bound backing indices and strict timestamp acceptance windows.

Important

When you reindex, the result is a single backing index of a new data stream.

To reindex, follow the steps on this page.

Note

This process only applies to time series data streams without a downsampling configuration. To reindex a downsampled data stream, reindex the backing indices individually, then add them to a new, empty data stream.

These high-level steps summarize the process of reindexing a time series data stream. Each step is detailed in a later section.

  1. Create an index template for the destination data stream
  2. Update the template with temporary settings for reindexing
  3. Run the reindex operation
  4. Revert the temporary index settings
  5. Perform a manual rollover to create a new backing index for incoming data

The examples on this page use Dev Tools Console syntax.

Create an index template for the new TSDS, using your preferred mappings and settings:

 PUT _index_template/my-new-tsds-template { "index_patterns": ["my-new-tsds"], "priority": 100, "data_stream": {}, "template": { "settings": { "index.mode": "time_series" }, "mappings": { "properties": { "@timestamp": { "type": "date" }, "dimension_field": { "type": "keyword", "time_series_dimension": true }, "metric_field": { "type": "double", "time_series_metric": "gauge" } } } } }

To support the reindexing process, you need to temporarily modify the template:

  1. Set index.time_series.start_time and index.time_series.end_time index settings to match the lowest and highest @timestamp values in the old data stream.
  2. Set index.number_of_shards to the sum of all primary shards of all backing indices of the old data stream.
  3. Clear the index.lifecycle.name index setting (if any), to prevent ILM from modifying the destination data stream during reindexing.
  4. (Optional) Set index.number_of_replicas to zero, to speed up reindexing. Because the data gets copied in the reindexing process, you don't need replicas.
 PUT _index_template/new-tsds-template { "index_patterns": ["new-tsds*"], "priority": 100, "data_stream": {}, "template": { "settings": { "index.mode": "time_series", "index.routing_path": ["host", "service"], "index.time_series.start_time": "2023-01-01T00:00:00Z", "index.time_series.end_time": "2025-01-01T00:00:00Z", "index.number_of_shards": 6, "index.number_of_replicas": 0, "index.lifecycle.name": null }, "mappings": { ... } } }
  1. Lowest timestamp value in the old data stream
  2. Highest timestamp value in the old data stream
  3. Sum of the primary shards from all source backing indices
  4. Speed up reindexing
  5. Pause ILM

Run the reindex operation:

 POST /_reindex { "source": { "index": "old-tsds" }, "dest": { "index": "new-tsds", "op_type": "create" } }

After reindexing completes, update the index template again to remove the temporary settings:

  • Remove the overrides for index.time_series.start_time and index.time_series.end_time.
  • Restore the values of index.number_of_shards, index.number_of_replicas, and index.lifecycle.name (as applicable).
 PUT _index_template/new-tsds-template { "index_patterns": ["new-tsds*"], "priority": 100, "data_stream": {}, "template": { "settings": { "index.mode": "time_series", "index.routing_path": ["host", "service"], "index.number_of_replicas": 1, "index.lifecycle.name": "my-ilm-policy" }, "mappings": { ... } } }
  1. Restore replicas
  2. Re-enable ILM

Create a new backing index with a manual rollover request:

 POST new-tsds/_rollover/

The destination data stream is now ready to accept new documents.