Pentagon is a small application designed to run as a Kubernetes CronJob to periodically copy secrets stored in Vault or Google Secret Manager into equivalent Kubernetes Secrets, keeping them synchronized. Naturally, this should be used with care as "standard" Kubernetes Secrets are simply obfuscated as base64-encoded strings. However, one can and should use more secure methods of securing secrets including Google's KMS and restricting roles and service accounts appropriately.

That's a good question. If you have a highly-available Vault setup that is stable and performant and you're able to modify your applications to query Vault, that's a completely reasonable approach to take. Similarly, if you are able to modify your application to query Google Secret Manager, that's an entirely valid solution. If you don't have such a setup, Pentagon provides a way to cache things securely in Kubernetes secrets which can then be provided to applications without directly introducing a dependency on Vault or Google Secret Manager.

Pentagon requires a YAML configuration file, the path to which should be passed as the first and only argument to the application. It is recommended that you store this configuration in a ConfigMap and reference it in the CronJob specification. A sample configuration follows:

vault: url: <url to vault> authType: # "token" or "gcp-default" token: <token value> # if authType == "token" is provided defaultEngineType: # "kv" or "kv-v2" (currently supported) role: "vault role" # if left empty, queries the GCP metadata service tls: # optional [tls options](https://godoc.org/github.com/hashicorp/vault/api#TLSConfig) namespace: <kubernetes namespace for created secrets> label: <label value to set for the 'pentagon'-created secrets> mappings: # mappings from vault paths to kubernetes secret names - vaultPath: secret/data/vault-path secretName: k8s-secretname vaultEngineType: # optionally "kv" or "kv-v2" to override the defaultEngineType specified above secretType: Opaque # optionally - default "Opaque" e.g.: "kubernetes.io/tls" additionalSecretLabels: # optionally add labels to the secret environment: dev team: core-services # mappings from google secrets manager paths to kubernetes secret names - sourceType: gsm path: projects/my-project/secrets/my-secret/versions/latest secretName: my-secret - sourceType: gsm path: projects/my-project/secrets/my-other-secret secretName: defaults-to-latest-version additionalSecretLabels: environment: dev team: core-services

By default, Pentagon will add a metadata label with the key pentagon and the value default. At the least, this helps identify Pentagon as the creator and maintainer of the secret.

You can also specify custom labels for each secret mapping using the additionalSecretLabels filed. These labels will be added to the Kubernetes secret alongside the required pentagon label.

If you set the label configuration parameter, you can control the value of the label, allowing multiple Pentagon instances to exist without stepping on each other. Setting a non-default label also enables reconciliation which will cleanup any secrets that were created by Pentagon with a matching label, but are no longer present in the mappings configuration. This provides a simple way to ensure that old secret data does not remain present in your system after its time has passed.

Apparently, different Vault secrets engines have slightly different APIs for returning data. For instance, here is the response for version 1 of the key/value store:

{ "request_id": "12a0c057-f475-4bbd-6305-e4c07e66805c", "lease_id": "", "renewable": false, "lease_duration": 2764800, "data": { "foo": "world" }, "wrap_info": null, "warnings": null, "auth": null }

Notice that the data object has the foo key embedded directly. Alternatively, here is the response for version 2 of the key/value store:

{ "request_id": "78b921ae-79a8-d7e3-da16-336b634fff22", "lease_id": "", "renewable": false, "lease_duration": 0, "data": { "data": { "foo": "world" }, "metadata": { "created_time": "2019-10-01T19:36:25.285387Z", "deletion_time": "", "destroyed": false, "version": 1 } }, "wrap_info": null, "warnings": null, "auth": null }

Notice the extra data element nested inside the outer data. Vault secrets engines can be mounted at arbitrary paths and it does not appear to be possible to reliably detect which engine was used in the API response directly. In order to properly unwrap the secret data,indicate either kv or kv-v2 as the vaultEngineType in the configuration. In the common case of using only one secrets engine, simply define the defaultEngineType in the vault configuration block and the mapping-level vaultEngineType will inherit the default. For compatibility, the unset default value defaults to kv. Note that this differs from the current default that Vault itself uses for the key/value secrets engine.

Google Secret Manager's API simply returns arbitrary bytes as the value of a secret, making no assumptions about its encoding. Kubernetes Secrets, on the other hand, can contain multiple key/value pairs. If you would like a single Google Secret Manager Secret to unwrap into multiple key/value pairs in the Kubernetes Secret, add gsmEncoding: "json" to the mapping value. Then store a JSON document in Google Secret Manager with JSON that will successfully unmarshal to a map[string]any. The key in that map will be used as the key of the Kubernetes Secret. If that value is a string or number, the value will be stored without any quoting. If the value is a JSON object or array it will be stored directly as the string serialization of that structure.

In cases where gsmEncoding is not set to json, the key's value will default to the name of the secret (secretName in the mapping). If you would like to override this, set gsmSecretKeyValue to your preferred key.

Also, Google Secret Manager Secrets have versions which can be specified in the configuration mapping's Path. If you do not specify a specific version (with the /versions/... suffix), /versions/latest will automatically be appended to the path.

The application will return 0 on success (when all keys were copied/updated successfully). A complete list of all possible return values follows:

Pentagon is intended to be run as a cron job to periodically sync keys. In order to create/update Kubernetes secrets extra permissions are required. It is recommended to grant those extra permissions to a separate service account which the application will also use. The following roles is a sample configuration:

apiVersion: batch/v1beta1 kind: CronJob metadata: name: pentagon spec: schedule: "0 15 * * *" concurrencyPolicy: Replace jobTemplate: metadata: labels: app: pentagon spec: parallelism: 1 completions: 1 template: spec: serviceAccountName: pentagon # run with a service account that has access to create/update secrets terminationGracePeriodSeconds: 10 restartPolicy: OnFailure containers: - name: pentagon image: vimeo/pentagon:v1.1.0 args: ["/etc/pentagon/pentagon.yaml"] imagePullPolicy: Always resources: limits: cpu: 250m memory: 128Mi requests: cpu: 250m memory: 128Mi volumeMounts: - name: pentagon-config mountPath: /etc/pentagon readOnly: true volumes: - name: pentagon-config configMap: name: pentagon-config --- apiVersion: v1 kind: ConfigMap metadata: name: pentagon-config data: pentagon.yaml: |  vault:  url: https://vault.address  authType: gcp-default  tls: # optional if you have custom requirements  capath: /etc/cas/custom-root-ca.crt  label: mapped  mappings:  - vaultPath: secret/config/main/foo.key  secretName: foo-key  - vaultPath: secret/ssl/tls/domain.com  secretName: domain.com  secretType: "kubernetes.io/tls" --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: pentagon rules: - apiGroups: ["*"] resources: - secrets verbs: ["get", "list", "create", "update", "delete"] --- apiVersion: v1 kind: ServiceAccount metadata: name: pentagon --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: pentagon roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pentagon subjects: - kind: ServiceAccount name: pentagon

Pentagon is a production of Vimeo's Core Services team with lots of support from Vimeo SRE.

• @sergiosalvatore
• @dfinkel
• @sachinagada