Terraform Version

1.14.5

Use Cases

Organizations that already run Elasticsearch as core infrastructure need a way to store Terraform remote state without introducing additional services. Currently, teams in this situation are forced to stand up a separate storage system (e.g. S3, Consul, Postgres) solely for Terraform state, adding operational overhead, additional credentials to manage, and another point of failure.

Attempted Solutions

The closest existing option is the http backend, which could theoretically work by placing an HTTP proxy in front of Elasticsearch. However, this approach requires:

1. Writing and maintaining a separate intermediary service that translates between the HTTP backend protocol and Elasticsearch's API
2. Deploying and operating that service as additional infrastructure
3. Implementing state locking manually in the proxy layer, since the HTTP backend's locking support is limited

This defeats the purpose of using existing infrastructure -- it just trades one dependency (a state backend) for another (a custom proxy).

Example of what this workaround looks like:

# Requires a custom proxy service running at this address terraform { backend "http" { address = "http://es-state-proxy:8080/state/myproject" lock_address = "http://es-state-proxy:8080/lock/myproject" unlock_address = "http://es-state-proxy:8080/unlock/myproject" } }

The consul and pg backends serve as precedent for supporting infrastructure that teams already operate. Elasticsearch falls into the same category.

Proposal

A native elasticsearch backend that stores state as documents in an Elasticsearch index. Configuration would look like:

terraform { backend "elasticsearch" { endpoints = ["https://elasticsearch.internal:9200"] index = "terraform_remote_state" username = "terraform" password = "secret" } }

Key design points of the implementation:

• Authentication: Supports username/password, API key, and bearer token (mutually exclusive, validated at configure time)
• TLS: Supports custom CA certificates, client certificates for mutual TLS, and skip-verification -- all via file paths or inline PEM, plus environment variable overrides
• Locking: Uses Elasticsearch's op_type=create for atomic lock acquisition -- no external lock service needed
• Workspaces: Each workspace gets its own state and lock document within the same index, identified by state-{workspace} and lock-{workspace} document IDs
• Index management: The backend auto-creates the index with appropriate mappings on first use
• Environment variables: All configuration fields can be set via ELASTICSEARCH_* environment variables

I have a working implementation with full test coverage and am willing to maintain this backend long-term. I'd like to add myself to CODEOWNERS for this backend.

The CONTRIBUTING.md notes that the team is not currently merging new state storage backends. I'm opening this issue to discuss whether an exception is possible given the willingness to maintain it long-term.

Existing community-maintained backends (pg, cos) serve as precedent for third-party-maintained backends in the repository.

References

No response