This repository contains necessary resources to deploy Elastic Serverless Forwarder(ESF). You can read details about ESF at Elastic Serverless Forwarder for AWS.

Section below explains how to use this repository to deploy ESF. You should first clone this repository.

For example, using SSH:

git clone git@github.com:elastic/terraform-elastic-esf.git

Following tools should be installed in your system to utilize this repository:

• curl
• terraform (v1.12.0 or higher)

1. Define secrets and variables in *.auto.tfvars files.

   See variables.tf for the list of variables declared or read section Inputs. Example:

    # variables.auto.tfvars lambda-name = "my-esf-lambda" release-version = "lambda-v1.21.1" # See https://github.com/elastic/elastic-serverless-forwarder/tags # config-file-bucket = "my-esf-bucket" # Uncomment if s3 bucket pre-exists aws_region = "eu-central-1" # config-file-local-path = "./config.yaml" # Uncomment if local config path is used inputs = [ { type = "cloudwatch-logs" id = "<some_arn>" outputs = [ { type = "elasticsearch" args = { elasticsearch_url = "https://url.com" api_key = "<some_api_key>" es_datastream_name = "logs-esf.cloudwatch-default" } } ] } ]

   Please read Inputs configuration section for more details on how to configure the inputs.

2. Execute terraform init

3. Execute terraform apply

1. Update the release-version value in your *.auto.tfvars file to the desired version.

   Latest releases are available at ESF releases

2. Execute terraform apply

Note: Read Create and upload config.yaml to S3 bucket if you need more details on how the inputs should be configured.

Note: Read Fields to know which values are expected for each field input.

Warning: If you use s3-sqs input type, you also need to configure s3-buckets variable.

When applying these configuration files, a config.yaml file will always be uploaded to an S3 bucket. This S3 bucket will be the one specified in config-file-bucket, or, if the value is left empty, a new S3 bucket will be created.

Following this, we will create the content for the config.yaml file. This file will be built based on:

• Variable inputs. This variable is not required.
• Local configuration file found in config-file-local-path. This variable is also not required.

If both variables are provided, both will be considered. Otherwise, just the one that was given. If none are provided, the config.yaml file will be:

"inputs": []

It does not make sense to leave both empty.

You can see the following examples on the resulting config.yaml file.

Configure the inputs variable as:

inputs = [ { type = "cloudwatch-logs" id = "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*" outputs = [ { type = "elasticsearch" args = { elasticsearch_url = "<url>" api_key = "<api key>" es_datastream_name = "logs-esf.cloudwatch-default" } } ] } ]

Do not configure the config-file-bucket variable, which will be left as (empty) since that is the default.

The config.yaml placed inside the bucket will be:

"inputs": - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*" "outputs": - "args": "api_key": "<api key>" "elasticsearch_url": "<url>" "es_datastream_name": "logs-esf.cloudwatch-default" "type": "elasticsearch" "type": "cloudwatch-logs"

Do not configure the inputs variable, which will be left as [] since that is the default.

Configure config-file-local-path variable:

config-file-local-path = "./config.yaml"

And the local config.yaml file looks like:

"inputs": - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*" "outputs": - "args": "api_key": "<api key>" "elasticsearch_url": "<url>" "es_datastream_name": "logs-esf.cloudwatch-default" "type": "elasticsearch" "type": "cloudwatch-logs"

Configure both inputs and config-file-local-path like in the previous examples.

The config.yaml placed inside the bucket will be:

"inputs": - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*" "outputs": - "args": "api_key": "<api key>" "elasticsearch_url": "<url>" "es_datastream_name": "logs-esf.cloudwatch-default" "type": "elasticsearch" "type": "cloudwatch-logs" - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*" "outputs": - "args": "api_key": "<api key>" "elasticsearch_url": "<url>" "es_datastream_name": "logs-esf.cloudwatch-default" "type": "elasticsearch" "type": "cloudwatch-logs"

object({
 batch_size = optional(number, 10)
 batching_window_in_second = optional(number, 0)
 })

{
 "batch_size": 10,
 "batching_window_in_second": 0
}

list(object({
 type = string
 id = string
 outputs = list(object({
 type = string
 args = object({
 elasticsearch_url = optional(string)
 logstash_url = optional(string)
 cloud_id = optional(string)
 api_key = optional(string)
 username = optional(string)
 password = optional(string)
 es_datastream_name = string
 batch_max_actions = optional(number)
 batch_max_bytes = optional(number)
 ssl_assert_fingerprint = optional(string)
 compression_level = optional(string)
 })
 }))
 }))

object({
 security-groups = list(string)
 subnets = list(string)
 })

{
 "security-groups": [],
 "subnets": []
}