Secrets management

A secret is anything that grants access: passwords, API keys, database URLs with credentials, private keys, OAuth client secrets, signing keys, session secrets. Leaked secrets are one of the most common ways attackers get in, because using a stolen key needs no exploit at all. In this lesson you'll keep secrets out of your repository, catch them automatically before they're committed, learn the correct order of operations when one leaks anyway, and replace long-lived secrets with ones fetched at runtime or issued for minutes at a time.

#Why Git is the worst place for a secret

You already know that Git keeps every version of every file. That's exactly the problem. A key committed once and deleted in the next commit is still in history, in every clone, in every fork, in CI caches, and possibly in the hosting provider's cached views of the old commit. Public repositories are scanned by automated bots within minutes of a push; cloud keys pushed to public GitHub are routinely abused before the author notices.

Private repositories aren't safe either. Everyone with read access, every CI job, every laptop with a clone and every backup now holds the key. When an employee's laptop or a CI system is compromised, the attacker greps the repository for credentials first.

The rule is simple: source code contains references to secrets, never the secrets themselves. The application reads the actual values at runtime from its environment, a mounted file, or a secret manager.

#Keeping secrets out: .env and .gitignore

During local development, a common pattern is an .env file that holds your personal development values, and is never committed. Commit a template instead, so teammates know which variables exist:

# .env.example (committed): names and safe placeholder values only
DATABASE_URL=postgresql://app:change-me@localhost:5432/app_dev
PAYMENTS_API_KEY=
SESSION_SECRET=

# .gitignore
.env
.env.*
!.env.example
*.pem
*.key

The ! line re-includes the template that the .env.* pattern would otherwise exclude. The application reads configuration from environment variables and fails loudly if a required one is missing:

import os

def required(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(f"missing required environment variable {name}")
    return value

DATABASE_URL = required("DATABASE_URL")
PAYMENTS_API_KEY = required("PAYMENTS_API_KEY")

In development, a tool such as python-dotenv, your IDE's run configuration, or docker compose --env-file loads .env into the environment. In production, the platform injects the variables (or better, the app fetches them from a secret manager, covered below).

One trap catches everyone once: .gitignore only affects untracked files. If .env was committed before you added the rule, Git keeps tracking it:

$ git check-ignore -v .env          # prints nothing: tracked files are never ignored
$ git rm --cached .env              # stop tracking, keep the local file
rm '.env'
$ git check-ignore -v .env
.gitignore:1:.env	.env
$ git commit -m "Stop tracking .env"

And if that .env contained real values, they are still in history. That's a leak; see the response section below.

#Catching leaks before they're committed

Humans paste keys into files. Automate the check so it happens before the commit exists. gitleaks is a widely used open-source scanner with hundreds of rules for specific providers (AWS, GitHub, Stripe, Slack and more) plus a generic high-entropy rule.

The easiest way to run it on every commit is the pre-commit framework, which manages hooks declared in a committed config file, so the whole team gets the same checks:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1
    hooks:
      - id: gitleaks

$ pipx install pre-commit
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Now try to commit a hard-coded key:

$ git add settings.py
$ git commit -m "Add payments client"
Detect hardcoded secrets.................................................Failed
- hook id: gitleaks
- exit code: 1

Finding:     PAYMENTS_API_KEY = "REDACTED"
Secret:      REDACTED
RuleID:      generic-api-key
Entropy:     5.000000
File:        settings.py
Line:        3
Fingerprint: settings.py:generic-api-key:3

9:35AM INF 0 commits scanned.
9:35AM INF scanned ~130 bytes (130 bytes) in 138ms
9:35AM WRN leaks found: 1

The commit is refused, and the hook redacts the value so the secret doesn't end up in your terminal scrollback or CI logs either. Move the value into .env, read it from the environment, and commit again.

You can run the same scanner directly. gitleaks git scans the full history of a repository, which is what you want when adopting it on an existing project, and gitleaks dir scans files on disk:

$ gitleaks git --redact -v .
...
Commit:      42f45ca683da92afca53dc9bd1d35e24288baf21
File:        settings.py
RuleID:      generic-api-key
...
9:41AM INF 2 commits scanned.
9:41AM INF scanned ~286 bytes (286 bytes) in 121ms
9:41AM WRN leaks found: 1

#Handling false positives

Test fixtures and example values will trigger rules. Don't disable the scanner; allowlist precisely. A .gitleaks.toml at the repository root extends the default rules:

title = "acme gitleaks config"

[extend]
useDefault = true

[[allowlists]]
description = "Fake credentials used in unit tests"
paths = ['''^testdata/''']

For a single line, add a gitleaks:allow comment at the end of it. For findings you've reviewed in old history, record their fingerprints in a .gitleaksignore file. All three are committed and reviewed, so exceptions are visible.

#Defense in depth

Pre-commit hooks run on developers' machines and can be skipped with git commit --no-verify or simply never installed. So also run the scanner in CI on every pull request as a required check (the CI lessons show how), and enable your Git host's server-side protection where available: GitHub's secret scanning with push protection and GitLab's secret push protection reject pushes containing recognised provider tokens.

#When a secret leaks: rotate first, then purge

When a secret reaches a remote, assume it's compromised. Work in this order:

1. Revoke or rotate the secret immediately. Create a new credential, deploy it, then disable the old one at the provider (AWS IAM, Stripe dashboard, database ALTER ROLE ... PASSWORD). This is the only step that actually stops an attacker. Do it even if the repository is private and even if "nobody could have seen it".
2. Check for misuse. Look at the provider's audit logs for the leaked credential's activity since the commit time: CloudTrail for AWS keys, API logs for SaaS tokens, connection logs for databases.
3. Remove it from history. Only now rewrite history so the value stops spreading.
4. Find the root cause and prevent recurrence: install the pre-commit hook, add the CI check, and ask why the secret was in a file at all.

Why rotate before purging? Purging can't recall copies. Forks, clones, CI caches, chat messages with the diff and anyone who already fetched it keep the old value. Rewriting history first wastes the critical minutes, and gives false comfort.

To purge, use git filter-repo (the maintained replacement for git filter-branch). Its --replace-text option rewrites matching strings in every commit:

$ printf '%s\n' 'q8Zt3LmX9vR2wK7pB4nY6cD1hF5sJ0aG==>REMOVED-SECRET' > ../replacements.txt
$ git filter-repo --replace-text ../replacements.txt
...
Completely finished after 0.04 seconds.
$ git show HEAD~1:settings.py
import os

PAYMENTS_API_KEY = "REMOVED-SECRET"
$ gitleaks git .
9:44AM INF no leaks found

Or remove a whole file from every commit with git filter-repo --invert-paths --path config/prod.env. Run it on a fresh clone; filter-repo removes the origin remote as a safety measure, so re-add it, force-push all branches and tags, and have every collaborator re-clone (merging their old clones would reintroduce the commits). On GitHub, old commits can remain reachable through pull-request refs and cached views; their support team can purge them. All of this is why step 1 matters most.

#Secret managers

A secret manager is a service whose job is to store secrets encrypted, hand them only to authorised identities, log every access, and support rotation. Applications fetch secrets at startup or on demand, so no secret lives in the repository, the image, or the CI configuration.

#HashiCorp Vault

Vault (or OpenBao, its open-source fork with a compatible API and a bao CLI) runs as its own service. With the key/value engine (version 2) you store and read secrets by path:

$ vault kv put -mount=secret myapp/db username=app password='s3cr3t-from-generator'
== Secret Path ==
secret/data/myapp/db

======= Metadata =======
Key                Value
---                -----
created_time       2026-10-11T10:15:02.418836Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

$ vault kv get -mount=secret -field=password myapp/db
s3cr3t-from-generator

Access is controlled by policies attached to identities. This policy lets the app read its own secrets and nothing else (note the data/ segment that KV version 2 inserts into API paths):

path "secret/data/myapp/*" {
  capabilities = ["read"]
}

The app authenticates to Vault with an identity it already has, such as its Kubernetes service account or cloud IAM role, so there's no bootstrap password to hide.

#Cloud secret managers

If you run in one cloud, its managed service is usually the simplest choice: AWS Secrets Manager, Google Cloud Secret Manager or Azure Key Vault. Access is governed by the same IAM roles as everything else:

$ aws secretsmanager get-secret-value --secret-id prod/myapp/db \
    --query SecretString --output text
{"username":"app","password":"s3cr3t-from-generator"}

$ gcloud secrets versions access latest --secret=myapp-db-password
s3cr3t-from-generator

In application code, fetch at startup and keep the value in memory:

import json
import boto3

def load_db_credentials() -> dict[str, str]:
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId="prod/myapp/db")
    return json.loads(resp["SecretString"])

The service's role (for example an ECS task role or EC2 instance profile) is granted secretsmanager:GetSecretValue on that one secret only. All managed services support versioning and scheduled rotation, so a rotation is a routine, tested operation rather than an emergency.

#Short-lived credentials

The best secret is one that expires before an attacker can use it. Long-lived credentials (a static API key, a database password unchanged for years, a cloud access key in CI) stay valid until someone notices. Short-lived credentials are issued on demand, expire in minutes or hours, and are tied to a verified identity.

Vault's dynamic secrets create a real database user per request, with a lease:

$ vault read database/creds/myapp-readonly
Key                Value
---                -----
lease_id           database/creds/myapp-readonly/2f6a614c-4aa2-7b19-24b9-ad944a8d4de6
lease_duration     1h
lease_renewable    true
password           A1a-p8sWq2FhT3kXv9zN
username           v-token-myapp-re-x7KbPq2tYr4WlZ3NsQ8d-1760177702

When the lease expires, Vault drops that database user. In the cloud, workloads use roles instead of keys: an EC2 instance profile, an ECS task role, GKE Workload Identity or Azure Managed Identity give the process temporary credentials that the platform rotates automatically. CI pipelines do the same through OIDC federation: the CI provider issues a signed token describing the job, and the cloud exchanges it for temporary credentials. You'll set that up in the CI lesson; the result is a pipeline with no stored cloud keys at all.

Other places secrets leak that these tools don't cover by themselves:

• Logs and error reports. Redact known secret fields in your logger; never log full request headers (they contain Authorization and cookies).
• Container images. A COPY .env or ARG NPM_TOKEN ends up in image layers or history. Use BuildKit secret mounts (RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci), which are never written to a layer.
• Frontend bundles. Anything in a browser bundle is public. Variables prefixed VITE_ or NEXT_PUBLIC_ are compiled into the JavaScript that every visitor downloads.

#Secrets in Kubernetes

Kubernetes has its own Secret object, a set of key-value pairs that pods consume as environment variables or mounted files. It's convenient, but it is a delivery mechanism, not a secret manager. The values are only base64-encoded, which anyone can reverse:

apiVersion: v1
kind: Secret
metadata:
  name: orders-db
  namespace: orders
type: Opaque
data:
  password: czNjcjN0LWZyb20tZ2VuZXJhdG9y    # base64 -d gives s3cr3t-from-generator

Two cluster settings decide how well native Secrets are protected:

• Encryption at rest. Secrets live in etcd, and by default the API server stores them there unencrypted. An EncryptionConfiguration on the API server encrypts them before they're written, ideally with a KMS provider so the key encryption key stays in your cloud's KMS. On managed clusters this is a cluster setting (check your provider's defaults; some now enable it automatically). On a self-managed control plane it looks like this:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}     # still lets the API server read Secrets written before encryption was enabled

• RBAC. Grant get, list and watch on secrets only to the identities that need them, and remember that anyone who can create pods in a namespace can mount its Secrets. The Kubernetes security lesson covers both in depth.

Even with both in place, don't create Secrets from YAML files in your repository: that's the "secret in Git" problem again, just base64-encoded. Keep the source of truth in your secret manager and use one of three common patterns to get values into the cluster.

#External Secrets Operator: sync into a Secret

The External Secrets Operator (ESO) runs in the cluster and copies values from Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault and others into ordinary Kubernetes Secrets, then keeps them in sync. A SecretStore says how to reach the manager; an ExternalSecret says which values to fetch and what Secret to create. Both contain only references, so they're safe to commit:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: orders
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: orders-secrets-reader   # IRSA: annotated with an IAM role (see workload identity below)
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: orders-db              # the Kubernetes Secret ESO creates and owns
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/orders/db      # the secret in AWS Secrets Manager
        property: password       # one field of its JSON value

$ kubectl get externalsecret -n orders
NAME        STORETYPE     STORE                REFRESH INTERVAL   STATUS         READY   LAST SYNC
orders-db   SecretStore   aws-secretsmanager   1h0m0s             SecretSynced   True    4m

Pods use orders-db like any other Secret. When you rotate the value in Secrets Manager, ESO updates the Secret within refreshInterval; mounted files follow automatically, while environment variables need a rollout restart.

#Secrets Store CSI Driver: mount as files

The Secrets Store CSI Driver mounts secrets straight from the manager into the pod's filesystem when the pod starts, using the pod's own identity. By default nothing is written to etcd at all. A SecretProviderClass (here with the AWS provider) lists what to fetch:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: orders-db
  namespace: orders
spec:
  provider: aws
  parameters:
    usePodIdentity: "true"           # authenticate with EKS Pod Identity
    objects: |
      - objectName: "prod/orders/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: password
            objectAlias: password

The pod mounts it as a read-only CSI volume:

    spec:
      serviceAccountName: orders-api
      containers:
        - name: api
          image: ghcr.io/acme/orders-api:1.4.2
          volumeMounts:
            - name: secrets
              mountPath: /mnt/secrets
              readOnly: true
      volumes:
        - name: secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: orders-db

The app reads /mnt/secrets/password. If the manager is unreachable when the pod starts, the pod stays in ContainerCreating until the mount succeeds, so the secret manager's availability now affects pod start-up. Rotation needs the driver's optional rotation feature enabled, or a pod restart.

#Sealed Secrets: encrypted in Git

Sealed Secrets takes the opposite approach for teams without a secret manager. A controller in the cluster holds a private key; you encrypt a Secret with its public key using the kubeseal CLI, and commit the resulting SealedSecret. Only the controller can decrypt it, into a normal Secret:

kubectl create secret generic orders-db -n orders \
  --from-file=password=./db-password.txt --dry-run=client -o yaml \
  | kubeseal --format yaml > orders-db.sealed.yaml

# orders-db.sealed.yaml (trimmed): safe to commit
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA1rO43cGDEq...

By default the ciphertext is bound to that name and namespace, so nobody can decrypt it by applying it somewhere else. The trade-offs: rotating means re-sealing and committing, every past ciphertext stays in Git history forever (so leaked controller keys expose all of them), and you must back up the controller's private key or lose every secret with it. SOPS, which encrypts values in files with a KMS key, is a similar Git-based option.

#Choosing a pattern

Use ESO as the default when you have a secret manager: it fits GitOps, works with any app that reads env vars or files, and handles rotation. Choose the CSI driver when secrets must never be stored in etcd, or when you want each pod to fetch with its own identity for a tighter audit trail. Use Sealed Secrets (or SOPS) when there is no secret manager, typically for small clusters, and plan to move off it as you grow.

#Workload identity: no static cloud keys

ESO, the CSI driver and your apps all have to authenticate to the secret manager. Storing an AWS access key in a Kubernetes Secret for that would bring back the long-lived credential you've been removing. Instead, use workload identity: the cloud trusts the cluster's signed ServiceAccount tokens and exchanges them for short-lived cloud credentials, exactly like OIDC federation from CI.

On EKS, EKS Pod Identity associates a ServiceAccount with an IAM role (it needs the eks-pod-identity-agent add-on, and the role's trust policy allows the pods.eks.amazonaws.com service principal to call sts:AssumeRole and sts:TagSession):

aws eks create-pod-identity-association --cluster-name prod \
  --namespace orders --service-account orders-api \
  --role-arn arn:aws:iam::123456789012:role/orders-api

The older IRSA (IAM Roles for Service Accounts), still widely used, annotates the ServiceAccount instead, with a trust policy on the cluster's OIDC provider that names system:serviceaccount:orders:orders-api as the allowed subject:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders-api
  namespace: orders
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/orders-api

On GKE, Workload Identity Federation for GKE lets you grant IAM roles directly to a Kubernetes ServiceAccount as a principal:

gcloud secrets add-iam-policy-binding orders-db \
  --role=roles/secretmanager.secretAccessor \
  --member="principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/acme-prod.svc.id.goog/subject/ns/orders/sa/orders-api"

Azure's equivalent is Microsoft Entra Workload ID. In every case the AWS, Google and Azure SDKs pick up the credentials automatically, the credentials expire within hours, and the permission is scoped to one ServiceAccount in one namespace, so a pod running as a different ServiceAccount gets nothing.

#Practice

In a scratch directory:

1. Create a Git repository with a small Python script that reads PAYMENTS_API_KEY from the environment, plus .env, .env.example and a .gitignore.
2. Set up pre-commit with gitleaks. Prove it works by adding PAYMENTS_API_KEY = "q8Zt3LmX9vR2wK7pB4nY6cD1hF5sJ0aG" to a Python file and trying to commit.
3. Bypass the hook once with git commit --no-verify to simulate a leak, then make one more normal commit on top.
4. Write the incident steps you would take, in order, then purge the key from history and verify with gitleaks git.

mkdir leak-lab && cd leak-lab && git init
printf '.env\n.env.*\n!.env.example\n' > .gitignore
printf 'PAYMENTS_API_KEY=\n' > .env.example
printf 'PAYMENTS_API_KEY=dev-only-local-value\n' > .env
cat > app.py <<'EOF'
import os
key = os.environ["PAYMENTS_API_KEY"]
print("key loaded, length", len(key))
EOF
cat > .pre-commit-config.yaml <<'EOF'
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1
    hooks:
      - id: gitleaks
EOF
pre-commit install
git add . && git commit -m "Initial app"           # passes; .env is ignored

echo 'PAYMENTS_API_KEY = "q8Zt3LmX9vR2wK7pB4nY6cD1hF5sJ0aG"' > settings.py
git add settings.py && git commit -m "Add settings" # blocked by gitleaks
git commit --no-verify -m "Add settings"            # simulated leak
echo "# notes" > NOTES.md && git add NOTES.md && git commit -m "Notes"

printf '%s\n' 'q8Zt3LmX9vR2wK7pB4nY6cD1hF5sJ0aG==>REMOVED-SECRET' > ../replacements.txt
git filter-repo --replace-text ../replacements.txt --force   # --force: not a fresh clone
gitleaks git .                                               # INF no leaks found
git log --oneline                                            # same 3 commits; the last two have new hashes

#Key takeaways

• Code holds references to secrets, never their values; once a secret is in Git history, every clone has it.
• Commit .env.example, ignore .env, and remember .gitignore doesn't untrack files that are already committed.
• Run gitleaks in a pre-commit hook and again as a required CI check, with precise allowlists instead of disabled rules.
• After a leak, rotate or revoke first, check for misuse, then purge history with git filter-repo; purging alone fixes nothing.
• Store secrets in Vault or a cloud secret manager, authorised by workload identity and audited on every read; in Kubernetes, sync or mount them with External Secrets or the CSI driver instead of committing Secret manifests, which are only base64.
• Prefer short-lived credentials (dynamic secrets, cloud roles, OIDC from CI) so a leaked value expires before it's useful.