1

HashiCorp Vault Agent creates a sidecar that talks to the Vault server and injects secrets as files into containers, where the files are located under /vault/secrets/.

"render all defined templates to the ephemeral in-memory mount at /vault/secrets/ available to other containers in the pod"

We simulated a "root takeover" attack on the kubernetes host itself, and we were saddened to see this:

root@k8s-host-24:~# cat /srv/kubelet/pods/0f407b88-1512-4612-9aa9-62189b12d5ed/volumes/kubernetes.io~empty-dir/vault-secrets/secrets.properties auth.password=glc_eyJvI[...]

It seems in this case, the "weakest link" is the file stored under /vault/secrets/.

While I understand the argument "facing root takeover, all bets are off" and "you should control root access instead"

Even if we tried to remove direct access to the pod itself:

kubectl -n thenamespace exec -it the-pod-5786b55b6f-z77jp -- bash Defaulted container "the-pod" out of: the-pod, vault-agent-init (init) error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "e7224d07630c58882024052a5aa50d4c808507b79436294a04": OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown

Q1. Is there a way for Vault to inject the secrets as environment variables directly into the business pods, without relying on this /vault/secrets/file which is more like a sitting duck at this point

Q2. Can the combination of a) source the file as an environment variable + b) discard the file to mitigate this issue?

Q3. Is there a way to resolve this?

Improve this question
4
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Commented Nov 18, 2024 at 17:56
  • 1
    If someone controls the k8s worker node, they can surely run a process inside the kernel namespace of each container. I.e., they can surely run a shell inside each pod, don't they? So they can read every secret, envars or files, that Vault created in the pods (and can also read the Vault credentials). Vault didn't solve the problem of how to securely configure an application, just the problem of how to securely manage secrets. If your TM includes root on the worker, you need to move to enclaves to keep the pods safe. Commented Nov 19, 2024 at 13:40
  • Having said that, you can tell Spring to use vault directly: spring.io/guides/gs/vault-config. This should make Spring read the secrets directly from Vault and keep them in memory (double check that). Note that a root attacker can still dump the secret from the memory of each (containerized) process. Commented Nov 19, 2024 at 13:41
  • I took away BASH, i.e. there is no way to exec inside the container (updating the question to reflect that) What you said would still hold even if there is no way to exec inside the container? Commented Nov 19, 2024 at 16:07

1 Answer 1

Reset to default
2

Fundamentally, when using a standard Container runtime (e.g. Containerd, CRI-O), pods are just processes on the node and as such a user with full root access to the node will be able to access any information they have access to, no matter what steps you take to try and restrict that.

If you want a situation where a node root user can't necessarily access the contents of a container running on that node, I'd recommend you look at projects like confidential containers which specifically aim to address this kind of risk.

Improve this answer
1
  • Very interesting link, upvoting Commented Nov 20, 2024 at 11:57

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.