This secrets rotation runbook gives you a repeatable way to rotate API keys, database credentials, Kubernetes Secrets, and TLS certificates with controlled cutovers, observable rollback conditions, and evidence that the old credential was revoked.
Overview
Credential rotation is safest when treated as a small, observable deployment rather than a one-line configuration change. The central pattern is to create or issue a replacement credential, make the application capable of using it, switch traffic or configuration, verify the new path, and only then disable the old credential. This dual-credential approach is not available for every system, but it should be the default design for production services whenever the provider supports overlapping credentials.
Before beginning, identify the secret, its owner, every consumer, the change window, and the exact revocation action. Record the planned rotation in your change-management or incident system. Do not paste secret values into tickets, chat, command history, CI logs, screenshots, or monitoring labels. A useful inventory records references and metadata instead: service name, environment, secret type, provider, last rotation date, expiry date, owner, consumers, and verification method.
Use a staged sequence:
- Discover: locate every application, job, worker, integration, dashboard, and administrator workflow that uses the credential.
- Prepare: confirm access to the secret manager, deployment system, logs, health checks, and provider console or API.
- Issue: create the replacement credential with the narrowest required permissions and a clear identifier such as a version or key ID.
- Deploy: load the replacement through the normal configuration path, preferably with a canary or staged rollout.
- Verify: test real authentication and relevant business operations while watching error rates, latency, logs, and provider audit events.
- Revoke: disable the previous credential after the agreed observation period, then verify that it can no longer authenticate.
- Record: capture timestamps, approvers, credential identifiers, deployment references, verification results, and revocation evidence without recording secret values.
For workload behavior and rollout safety, pair this runbook with appropriate readiness, liveness, and startup checks. The Kubernetes health checks checklist is useful when a credential change causes containers to restart or temporarily fail dependency checks.
Choosing a delivery mechanism
| Mechanism | Best use | Rotation concern | Control to add |
|---|---|---|---|
| Environment variables | Simple application configuration and short-lived process settings | Running processes usually need a restart to receive the new value | Use staged restarts, safe rollout limits, and log redaction |
| Kubernetes Secrets | Cluster workloads that already use Kubernetes-native configuration | Updating the object does not guarantee that every process reloads it | Use a deliberate reload or rollout and verify mounted-file behavior |
| Cloud secret manager | Centralized access control, versioning, and audit trails | Applications may cache values or require an explicit version change | Define refresh intervals, identity permissions, and rollback behavior |
| External Secrets operator | Synchronizing an external manager into Kubernetes | Sync delay, permissions, and controller health can affect propagation | Monitor synchronization status and test the complete chain |
Checklist by scenario
API keys and service tokens
- Confirm whether the provider supports two active keys, key IDs, scoped tokens, or a grace period.
- Identify all consumers, including scheduled jobs, local agents, third-party integrations, and dormant environments.
- Create the replacement with only the permissions, IP restrictions, audience, and expiry required by the workload.
- Store it in the approved secret manager and update the application reference without exposing the value to the deployment log.
- Roll out to a canary or a small worker group first. Test authentication and a representative read and write operation where safe.
- Monitor authentication failures and provider audit activity. Keep the old key active only for the defined observation window.
- Revoke the old key, then test that requests using it fail. Remove stale copies from local configuration, CI variables, and deployment manifests.
Database credentials
- Determine whether the database supports multiple users, overlapping passwords, role-based credentials, or a connection proxy.
- Create a replacement identity with the same minimum required grants. Avoid expanding privileges during rotation.
- Update connection pools, migration jobs, background workers, read replicas, and administrative tools that use the credential.
- Deploy the change gradually and confirm new connections succeed. Existing pooled connections may continue using the old credential until recycled.
- Check application errors, connection-pool exhaustion, transaction failures, and migration status.
- Terminate or drain old connections according to the database and application procedure, then disable the old identity or password.
- Verify that the application can complete its normal database operations and that the old credential is rejected.
Kubernetes Secrets
- Identify whether the secret is consumed as an environment variable, a mounted file, an image pull secret, or through an external synchronization controller.
- Update the source of truth first. Do not edit a generated object if a controller will overwrite it.
- Confirm namespace, labels, ownership, access permissions, and the expected propagation path.
- For environment variables, plan a controlled pod replacement. For mounted files, verify whether the application watches and reloads the file or requires a restart.
- Use a canary deployment and check pod readiness, application logs, dependency calls, and secret synchronization status.
- Ensure the previous value is absent from manifests, rendered configuration, debugging output, and temporary files.
After a workload change, review resource behavior as well as authentication. A failed rotation can cause restart loops or increased memory use; the Kubernetes resource requests and limits guide can help distinguish a credential failure from capacity symptoms.
TLS certificates
- Confirm the certificate names, SANs, issuer chain, private-key location, termination point, and services that present the certificate.
- Check whether renewal is automatic and whether the proxy, ingress controller, load balancer, or application reloads certificates without a restart.
- Obtain and validate the replacement certificate and chain in a non-production or staging path when practical.
- Deploy the certificate to every termination point, including regional, standby, internal, and administrative endpoints where applicable.
- Verify certificate expiry, hostname coverage, chain delivery, protocol behavior, and client compatibility from representative network locations.
- Confirm that the private key is protected and that the previous certificate and key are removed from active configuration and temporary storage.
- Record renewal and monitoring evidence, including the next expiry alert and the owner responsible for responding to it.
Emergency compromise or offboarding
When a credential may already be exposed, containment takes priority over a long observation window. Preserve relevant audit records, identify the blast radius, disable or revoke the compromised credential, and issue a replacement. Then deploy the replacement through the safest available path and inspect access logs for unauthorized use. If immediate revocation could interrupt a critical service, use a short, explicitly approved overlap only when the risk assessment supports it; do not leave a known-compromised credential active by default.
What to double-check
- Ownership: someone is accountable for issuing, deploying, verifying, and revoking the credential.
- Scope: the replacement has no broader permissions than the old credential and no unnecessary lifetime.
- Propagation: the new value reaches every consumer, including replicas, workers, cron jobs, CI/CD tools, and disaster-recovery environments.
- Reload behavior: you know whether a process restart, rolling deployment, sidecar refresh, or certificate reload is required.
- Observability: dashboards and alerts distinguish authentication failures from unrelated application or infrastructure failures. Relevant deployment health and service objectives can be checked against the SLO and error budget template.
- Rollback: rollback means restoring a known-good configuration, not automatically re-enabling an exposed or revoked secret.
- Evidence: record key IDs, secret versions, deployment IDs, approval references, test results, and revocation timestamps, never secret contents.
- Recovery: confirm that backups, infrastructure code, sealed configuration, and runbooks will not restore the old credential accidentally.
A practical verification sequence is to test the new credential directly where safe, test the application through its normal interface, inspect logs for authentication errors, check provider audit events, and then confirm rejection of the old credential. Keep tests narrow and reversible; avoid probing production systems with destructive operations merely to prove access.
Common mistakes
- Rotating only the visible application: forgotten workers, scripts, and integrations then fail later. Search deployment configuration and access records before changing anything.
- Assuming an update is a reload: environment variables are commonly read only at process start, while mounted files and controllers have different refresh behavior.
- Revoking first: this creates an avoidable outage when the replacement was misconfigured. Use overlap when supported, with a short and documented expiry.
- Testing only health endpoints: a process can be healthy while every database or provider call is rejected. Include an authenticated, representative transaction.
- Leaving old values in automation: CI variables, temporary files, rendered manifests, shell history, and debug logs can preserve a secret after the main store is updated.
- Using rollback as a reflex: do not restore a credential that was exposed, expired, or intentionally revoked. Issue a new credential and correct the deployment instead.
- Changing permissions during rotation: separate access redesign from credential replacement so failures are easier to diagnose and audit.
For container supply-chain context, pair credential work with image and deployment hygiene. The Docker image security scanning workflow provides a related remediation pattern for finding, prioritizing, and verifying fixes without losing a clear audit trail.
When to revisit
Use this runbook before every planned rotation, certificate renewal, employee or vendor offboarding event, infrastructure migration, and suspected credential compromise. Revisit the inventory on a regular operational cycle and schedule a formal review at least every six months, or sooner when providers, deployment platforms, Kubernetes integrations, secret managers, certificate automation, or application reload behavior changes.
At each review, confirm that provider-specific commands and permissions still match the current platform, that external secret synchronization is observable, that certificate renewal behavior has been tested, and that the revocation procedure is still available to the on-call team. Run a tabletop exercise for a high-impact credential at a practical interval, then update the runbook with any missing owner, dependency, alert, or rollback step.
Before closing the next rotation, ask: Did every consumer use the replacement? Did the old credential fail after revocation? Did monitoring show normal service behavior? Is the evidence complete without exposing a secret? If any answer is uncertain, leave the change open, assign an owner, and resolve the gap before declaring the rotation complete.