How to Secure Kubernetes Secrets: 12 Steps, 90 Min [2026]

Pods can then either use Vault Agent as a sidecar to auto-fetch and refresh secrets to a shared volume, or you can point ESO's SecretStore at Vault instead of AWS Secrets Manager, reusing the same ExternalSecret pattern from Step 4. The Vault route makes sense when you already run Vault for non-Kubernetes systems and want one control plane instead of three separate cloud-native tools. HashiCorp's own Vault documentation covers the full set of auth methods and secret engines beyond what this tutorial's Kubernetes-focused slice touches.

One tradeoff worth naming: adding Vault to a stack that already runs ESO against a cloud secrets manager means you now have two systems capable of injecting secrets into the same cluster. Pick one as the source of truth per application rather than letting both manage the same ExternalSecret — otherwise you end up debugging which system last wrote a given value when a credential mysteriously changes.

Step 7b: Weigh the operational cost of each option

Every layer added to a secrets pipeline is also a layer someone has to operate at 2 a.m. when it breaks. Running your own HashiCorp Vault cluster means owning unseal procedures, storage backend backups, and a second set of upgrade cycles independent of Kubernetes itself — a real cost that's easy to underestimate when a proof-of-concept Vault instance looks simple. Cloud-native secrets managers paired with ESO shift that operational burden to the cloud provider, at the cost of vendor lock-in and, for high-volume secret access patterns, per-API-call pricing that can add up in AWS Secrets Manager or Azure Key Vault if you're not caching or batching reads.

Sealed Secrets carries the lowest infrastructure footprint of the four approaches covered here — no external vault, no per-call billing, just a controller and a keypair — but it also offers the weakest audit trail, since there's no central log of who accessed which secret value and when the way a cloud secrets manager or Vault provides natively. Match the approach to what your compliance requirements actually demand rather than defaulting to the most feature-complete option for every workload; a small internal tool with no regulatory scope doesn't need the same pipeline as a payments service.

Step 8: Lock down RBAC around secrets

None of the tooling above matters if a broad ClusterRole still grants blanket read access to secrets across every namespace. Scope access down to exactly what each service account needs.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader-app-only
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials-synced"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-secret-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: app-sa
    namespace: production
roleRef:
  kind: Role
  name: secret-reader-app-only
  apiGroup: rbac.authorization.k8s.io

The resourceNames field is the part most teams skip. Without it, a Role granting get on secrets applies to every secret in the namespace, not just the one the service is supposed to read. Naming the specific resource closes that gap.

Step 9: Rotate secrets without downtime

A secrets pipeline that can't rotate values without a deployment is only half finished. With ESO's refreshInterval already polling the backing store, rotation becomes a two-part problem: getting the new value into the Kubernetes Secret (ESO handles this), and getting your running pods to actually use it.

# Trigger an immediate ESO refresh instead of waiting for the interval
kubectl annotate externalsecret db-credentials -n production \
  force-sync=$(date +%s) --overwrite

# For apps that only read env vars at startup, roll the deployment
# after confirming the secret updated
kubectl rollout restart deployment/app -n production

Apps that mount secrets as a volume rather than an environment variable pick up updates automatically within about a minute (the kubelet sync period), no restart required — one more reason to prefer volume mounts over env vars for anything that rotates on a schedule.

Step 10: Add CI scanning so secrets never reach Git in the first place

Every layer above assumes a developer doesn't accidentally paste a raw credential into a commit. That assumption fails often enough that secret-scanning in CI is now standard practice. Run a scanner like Gitleaks against every pull request before it merges.

# .github/workflows/secrets-scan.yml
name: secrets-scan
on: [pull_request]
jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

If your team hasn't set this up yet, the full walkthrough is in our Gitleaks setup guide, which covers pre-commit hooks in addition to the CI gate shown here.

Step 11: Scan container images for baked-in secrets

Secrets don't only leak through Git — they also end up baked into container image layers when a Dockerfile COPYs a .env file or a build arg gets cached into an intermediate layer. Run an image scanner as part of your build pipeline to catch this before the image ships.

trivy image --severity HIGH,CRITICAL --scanners secret my-app:latest

Our Trivy container scanning guide covers setting this up as a CI gate that blocks builds on HIGH or CRITICAL findings — a practice now baked into most 2026 Kubernetes project checklists alongside probes and etcd backups.

Step 12: Verify the full pipeline end to end

Run through this checklist once everything is deployed to confirm the pipeline actually behaves the way you designed it.

# 1. Confirm encryption at rest is active
kubectl get secrets --all-namespaces -o json | \
  kubectl -n kube-system exec -i etcd-pod -- etcdctl get /registry/secrets/production/db-credentials-synced

# 2. Confirm ESO is syncing (status should show SecretSynced=True)
kubectl get externalsecret -A -o wide

# 3. Confirm no plaintext secrets in Git history
gitleaks detect --source . --verbose

# 4. Confirm RBAC scoping
kubectl auth can-i get secrets --as=system:serviceaccount:production:other-app -n production

# 5. Confirm CSI-mounted secrets never synced to etcd (if syncSecret disabled)
kubectl get secrets -n production | grep prod-app-secrets   # should return nothing

If step 1 returns encrypted binary garbage instead of readable base64, encryption at rest is working. If step 4 returns "no," your RBAC scoping held. Both should be true before you call this done.

Testing the pipeline against a simulated compromise

A verification checklist confirms the pipeline is configured correctly. It doesn't confirm the pipeline actually reduces damage when something goes wrong. Run a tabletop exercise before you trust this in production: pick one service account, simulate it being compromised, and walk through exactly what an attacker with that identity could reach.

# Impersonate a specific service account and see what it can actually read
kubectl auth can-i --list --as=system:serviceaccount:production:app-sa -n production

# Check if that service account's token could read secrets in OTHER namespaces
kubectl auth can-i get secrets --as=system:serviceaccount:production:app-sa -n staging
kubectl auth can-i get secrets --as=system:serviceaccount:production:app-sa -n kube-system

If the answer to either cross-namespace check comes back "yes," the RBAC scoping from Step 8 has a gap somewhere — most often a ClusterRoleBinding left over from an earlier, less careful setup. Before this project, a compromised pod's service account token in a poorly scoped cluster can often read every Secret in every namespace. After the RBAC tightening in Step 8, that same token should be limited to the handful of specifically named secrets its own workload actually uses. That before-and-after comparison is the real measure of whether this tutorial's changes did anything, not just whether the YAML applied without errors.

Common pitfalls

These are the mistakes that show up most often when teams build this stack for the first time.

• Forgetting to back up the Sealed Secrets private key. Losing it means every committed SealedSecret is permanently unreadable — there is no recovery path.
• Running encryption-at-rest migration only on new writes. Old secrets stay in plaintext-equivalent form in etcd until you force a rewrite with kubectl replace.
• Mismatching ESO and Kubernetes versions. ESO's short release cycle means a chart pinned six months ago may no longer be supported on your current cluster minor version.
• Skipping resourceNames in RBAC Roles. A Role scoped to the "secrets" resource without naming specific objects grants access to every secret in the namespace.
• Treating refreshInterval as instant. ESO polls on an interval, not a push — a compromised credential you rotate upstream is still live in the cluster until the next poll or a forced sync.
• Storing the Vault unseal keys or root token in the same cluster they protect. That defeats the separation of duties Vault is meant to provide.
• Assuming CSI-mounted secrets are automatically excluded from etcd. They only stay out of etcd if syncSecret.enabled is left off — many tutorials enable it by default for convenience, quietly reintroducing the exposure you were trying to avoid.

Troubleshooting

Working through these eight issues covers most of what breaks in a fresh secrets pipeline setup.

• ExternalSecret stuck in "SecretSyncedError." Run kubectl describe externalsecret <name> -n <namespace> and check the events. This is almost always an IAM permissions issue — the service account role can't read the specific secret path in AWS Secrets Manager.
• kubeseal fails with "no key could decrypt secret." The Sealed Secrets controller was reinstalled or its key rotated after the SealedSecret was created. Re-seal the value against the current controller key.
• CSI driver pod CrashLoopBackOff. Check that the correct cloud provider plugin (aws/azure/gcp) is installed alongside the base driver — the base Helm chart doesn't include any cloud provider by default.
• Secret value doesn't update in a running pod after rotation. Env-var-based secrets never update without a pod restart. Switch to volume mounts if you need live rotation.
• "Error from server (Forbidden)" when ESO tries to write the target Secret. The ESO service account itself needs RBAC permission to create/update Secrets in the target namespace — check its ClusterRole bindings.
• Vault Kubernetes auth returns "permission denied." The bound service account name or namespace in the Vault role doesn't match the pod's actual service account exactly — this is case-sensitive and namespace-scoped.
• Gitleaks flags false positives on test fixtures. Add a .gitleaksignore file with the specific fingerprint hashes for known-safe test values rather than disabling the rule entirely.
• Encryption at rest configured but secrets still readable as plain base64 via etcdctl. The API server wasn't restarted after the EncryptionConfiguration change, or the flag path is wrong — check kube-apiserver pod logs for a startup error referencing the encryption config file.

Comparing the four approaches

No single tool covers every case. Here's how the four pieces of this tutorial's stack actually differ in practice.

Advanced tips for scaling this beyond one cluster

Once the base pipeline works on one cluster, a few practices make it hold up across a fleet. First, use ESO's ClusterSecretStore resource instead of a namespaced SecretStore when the same backing vault serves multiple namespaces or teams — it avoids duplicating the same store definition dozens of times. Second, if you're running multiple clusters against the same cloud account, scope IAM roles per-cluster rather than sharing one broad role, so a compromised cluster can't read secrets belonging to another. Third, combine the Secrets Store CSI Driver's rotationPollInterval setting with a readiness probe that fails if a mounted secret file is older than your rotation SLA — this turns "the secret didn't rotate" from a silent failure into a visible one in your monitoring dashboards. Fourth, if your organization is also standardizing internal developer workflows, a self-service portal like the one covered in our Backstage developer portal setup guide can expose "request a secret" as a templated action instead of a Slack message to the platform team, which cuts down on secrets getting passed around outside the pipeline entirely.

Finally, treat the RBAC hardening in Step 8 as an ongoing audit, not a one-time setup. New services get added, old ones get decommissioned, and stale RoleBindings granting broad secret access tend to survive long after the workload they were built for is gone. Our CIS Benchmarks hardening guide includes a section on periodic RBAC review that pairs well with the checklist in Step 12 above. Teams running Kubernetes autoscaling alongside this pipeline should also check our Karpenter for EKS Autoscaling guide, since new nodes spinning up need the same IRSA and CSI driver configuration as existing ones or secret mounts will silently fail on fresh capacity.

The complete working project

Here's every piece from this tutorial assembled into a single deployable set of manifests, structured the way you'd lay it out in a real GitOps repo.

k8s-secrets-project/
├── 00-encryption/
│   └── encryption-config.yaml
├── 01-eso/
│   ├── helm-values.yaml
│   ├── secretstore-aws.yaml
│   └── externalsecret-db.yaml
├── 02-sealed-secrets/
│   ├── api-key-sealedsecret.yaml
│   └── key-backup.yaml          # stored outside Git, gitignored
├── 03-csi-driver/
│   ├── secretproviderclass.yaml
│   └── app-pod.yaml
├── 04-vault/
│   ├── vault-policy.hcl
│   └── vault-role.yaml
├── 05-rbac/
│   ├── role-secret-reader.yaml
│   └── rolebinding-app.yaml
├── 06-ci/
│   ├── secrets-scan.yml          # Gitleaks GitHub Action
│   └── image-scan.yml            # Trivy secret scanner
└── README.md

Deploy the whole stack in order:

kubectl apply -f 01-eso/secretstore-aws.yaml
kubectl apply -f 01-eso/externalsecret-db.yaml
kubectl apply -f 02-sealed-secrets/api-key-sealedsecret.yaml
kubectl apply -f 03-csi-driver/secretproviderclass.yaml
kubectl apply -f 03-csi-driver/app-pod.yaml
kubectl apply -f 05-rbac/role-secret-reader.yaml
kubectl apply -f 05-rbac/rolebinding-app.yaml

# Confirm everything synced
kubectl get externalsecret,sealedsecret,secretproviderclass -A

This layout keeps each concern isolated — encryption config, ESO sync rules, GitOps-safe sealed values, CSI mounts, Vault policy, RBAC, and CI scanning all live in their own directory, so a change to one doesn't require touching the others.

Frequently asked questions

Are Kubernetes Secrets encrypted by default?
No. Native Secret objects are base64-encoded, which is trivially reversible, not encrypted. You must explicitly configure an EncryptionConfiguration with a real provider (KMS, aescbc, secretbox) to get actual encryption at rest in etcd.

Do I need External Secrets Operator, Sealed Secrets, and the CSI driver all at once?
No. Most teams pick one primary approach based on their setup — ESO if you already use a cloud secrets manager, Sealed Secrets if you need GitOps-safe manifests without a cloud vault, and the CSI driver if you need secrets to never touch etcd at all. They can coexist, but a single cluster rarely needs all three running for the same workloads.

What Kubernetes version does External Secrets Operator 2.7 support?
ESO 2.7, released June 26, 2026, supports Kubernetes 1.34 and 1.35. Earlier 2.x releases (2.0 through 2.6) also cover the 1.34–1.35 range, but each carries its own short end-of-life window, so pin your version and track ESO's stability matrix rather than assuming backward compatibility indefinitely.

Is HashiCorp Vault still necessary if I'm using a cloud-native secrets manager?
Only if you have workloads outside Kubernetes — VMs, CI runners, third-party services — that also need centralized secret access. If everything you run lives inside one cloud's Kubernetes clusters, the native cloud secrets manager (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) paired with ESO is usually simpler to operate than running Vault yourself.

What happens if I lose the Sealed Secrets controller's private key?
Every SealedSecret ever encrypted against that key becomes permanently undecryptable. There's no recovery mechanism — this is by design, since the whole point is that only the controller holding that key can decrypt the values. Back the key up to secure storage outside the cluster immediately after installation.

Can I mount a secret from AWS Secrets Manager without ever creating a Kubernetes Secret object?
Yes, using the Secrets Store CSI Driver with syncSecret.enabled=false. The value is mounted directly into the pod's filesystem as a volume and never written to the Kubernetes API or etcd, which satisfies stricter compliance requirements around secret exposure.

How often should production secrets be rotated?
There's no universal number, but most compliance frameworks referenced in 2026 platform security guides recommend 90 days for standard credentials and far shorter windows — hours to days — for anything tied to a security incident or a departing employee. The automation in Step 9 is what makes short rotation windows operationally realistic instead of a manual burden.

Why does my ExternalSecret show "SecretSynced" but the pod still uses an old value?
If your application reads the secret as an environment variable, Kubernetes does not automatically update environment variables in a running container after the underlying Secret changes — only a new pod (via rollout or restart) picks up the new value. Volume-mounted secrets update in place within about a minute due to the kubelet's sync period.

Related Coverage

• From Backyard Feeders to Biodiversity Data: The Rise of Smart Wildlife Cameras
• When Thousands of Players Press Spin: How Online Casinos Handle Traffic Spikes
• What Muse Spark 1.3 actually changes
• What Shipped in OpenClaw 2.0
• Why Anthropic's August Was Different From Every Prior Month

Elias Virtanen

Cybersecurity Analyst

Elias Virtanen is the Cybersecurity Analyst at Tech Insider, bringing hands-on expertise from his background in penetration testing and security consulting. He previously worked as a security researcher at F-Secure in Helsinki, where he focused on threat intelligence and vulnerability disclosure. Elias covers ransomware trends, zero-trust architecture, and the evolving regulatory landscape including NIS2 and the EU Cyber Resilience Act. He holds a CISSP certification and an MSc in Information Security from Aalto University.

View all articles