Rotating a production secret safely requires more than generating a replacement value. This runbook provides a repeatable workflow for API keys, database passwords, access tokens, signing keys, and TLS certificates, including ownership tracking, dual-credential cutovers, validation, rollback limits, and emergency revocation. Use it as a scheduled reference during routine DevOps secrets management and whenever a credential may have been exposed.
Overview
Secret rotation is the controlled replacement of a credential while preserving the availability and security of the systems that use it. The objective is not simply to create a new value. It is to make every legitimate consumer use the replacement, confirm that the old value is no longer accepted, and leave an auditable record of what changed.
Begin by identifying the secret type and its failure modes. An API provider may support two active keys, making a staged cutover possible. A database may allow a second user or password during a transition. A certificate renewal may require updating a load balancer, ingress controller, trust store, or client bundle. A short-lived token may instead be replaced by updating the issuing configuration and waiting for existing tokens to expire.
Use an approved secret manager rather than placing values in source code, tickets, chat messages, shell history, or unencrypted configuration files. The exact product may vary, but the workflow should consistently provide access control, versioning, audit records, and a way for workloads to retrieve the current value. For Kubernetes workloads, review the secret mount or environment-variable behavior before scheduling a change. For CI/CD, check both pipeline variables and the external systems that the pipeline authenticates to.
Rotation should be treated as an operational change with an owner, a maintenance window when appropriate, a validation plan, and a communication path. A useful companion is the Secrets Rotation Checklist for Cloud, CI/CD, and Kubernetes Workloads, which can be adapted to service onboarding and recurring reviews.
What to track
Maintain a secret inventory that describes how each credential is used without storing the credential itself. At minimum, record:
- Secret identifier: Use a stable name and distinguish production, staging, and development values.
- Type and purpose: Identify whether it is an API key, database password, OAuth client secret, signing key, certificate, private key, or service-account credential.
- System and consumers: List applications, workers, jobs, clusters, pipelines, integrations, and administrators that depend on it.
- Owner and backup owner: Assign a team responsible for the rotation and a second contact for incidents or holidays.
- Storage location: Record the secret manager path, cloud account, namespace, certificate store, or CI/CD variable group. Do not copy the secret into the inventory.
- Last and next rotation dates: Track when the value was changed and when the next review is due.
- Expiration and overlap rules: Note provider limits, certificate expiry, token lifetime, and whether old and new credentials can coexist.
- Propagation method: Document whether consumers reload automatically, require a process restart, need a rollout, or retrieve the value only at startup.
- Validation and revocation steps: Link to safe health checks and document how the old value will be disabled.
Track evidence of completion as well: the change reference, operator, timestamp, validation result, and confirmation that the old credential was disabled or expired. Avoid recording sensitive values in logs. If a command prints a token, password, private key, or authorization header, stop and redesign the check.
Use a two-credential pattern when possible
For API keys and database credentials, the safest transition usually has four phases: create the replacement, deploy consumers that can use it, verify the replacement, and revoke the old value. Keep the overlap window as short as practical. Do not assume that an application has switched merely because its deployment completed; verify successful authentication and inspect service-specific metrics or audit events.
If the provider permits only one active credential, plan a brief cutover or use a supported intermediate identity. In that case, document the outage risk, test the procedure outside production, and ensure the team knows that rollback may require issuing yet another credential rather than restoring the old one.
Cadence and checkpoints
Use a monthly or quarterly review to keep the inventory accurate, then rotate according to the credential’s exposure, provider behavior, and internal risk requirements. A schedule is useful only if it includes a readiness check and a completion check.
Before the rotation
- Confirm the owner, backup owner, affected services, and approved change window.
- Check whether the credential is present in source repositories, container images, build logs, crash reports, tickets, or local configuration. If exposure is suspected, treat the event as an emergency rather than waiting for the planned window.
- Test the replacement procedure in a non-production environment with comparable deployment and reload behavior.
- Confirm backups are not being mistaken for a rollback plan. Restoring an old configuration may restore an invalid or compromised credential.
- Prepare service-level checks, authentication tests, dashboards, and an escalation path.
During the rotation
- Create the replacement using the approved identity or secret-management workflow.
- Store it in the designated location and restrict access to the required workload and operators.
- Deploy or reload consumers. For Kubernetes, determine whether a secret update triggers the required application behavior; if not, perform a controlled rollout. For CI/CD, update the variable or integration and run a safe pipeline validation.
- Run a functional check against the actual dependency, such as a narrowly scoped API request, a read-only database connection, or a certificate handshake.
- Revoke the old value after the replacement is confirmed, then monitor for authentication failures and unexpected retries.
After the rotation
Record the result, close temporary access, remove obsolete configuration, and review logs for use of the retired credential. Keep the change record, but do not retain the secret value. If a certificate was renewed, check the complete chain, hostname coverage, expiry date, and every termination point that serves it. A certificate renewal checklist should include load balancers, ingress, proxies, service meshes, client trust stores, and monitoring alerts where applicable.
How to interpret changes
Not every authentication error means the rotation failed, but unexplained failures should be treated seriously. Compare the timing of errors with deployment, reload, and revocation events. A sudden increase in authorization failures may indicate that one consumer still uses the old value, a cache has not refreshed, a worker was not restarted, or the replacement has incorrect scope.
Separate the signals by consumer. Check application logs, pipeline results, database connection metrics, provider audit events, and certificate monitoring rather than relying on one overall health indicator. Pay attention to delayed jobs, scheduled tasks, infrequently used administrative paths, and disaster-recovery environments; these often reveal dependencies that normal traffic does not exercise.
Rollback requires careful judgment. If the old secret was revoked, exposed, or potentially copied, do not reactivate it simply to restore service. Generate a new replacement, correct the consumer configuration, and revoke any additional compromised value. If the issue is a deployment defect and the old credential remains trusted, a temporary rollback may be possible, but set an explicit deadline for completing the rotation again.
For a suspected leak, move directly to containment: identify the credential, restrict or revoke it, preserve relevant audit evidence, assess affected systems, and notify the incident owner. Do not wait for a routine rotation date. The SRE incident response runbook can provide a broader structure for triage, communication, mitigation, and review.
When to revisit
Review this runbook and the inventory at least monthly or quarterly, depending on the number and criticality of your credentials. Revisit it immediately when a provider changes its key, IAM, token, or certificate behavior; when a workload moves between cloud accounts or clusters; when a new CI/CD integration is introduced; or after an incident involving unauthorized access.
At each review, sample several entries rather than checking only whether dates are populated. Confirm that the named owner still supports the service, the listed consumers are complete, the secret path is correct, expiration alerts reach the right team, and the documented reload method still works. Select one lower-risk credential for a controlled rehearsal if the team has not performed a rotation recently.
Use this practical closing checklist before marking a rotation complete:
- Inventory entry is current and has an accountable owner.
- Replacement was created and stored without exposing its value.
- Every known consumer was updated or intentionally excluded.
- Authentication, authorization, and application health were validated.
- Old credentials were revoked, expired, or assigned a documented retirement deadline.
- Logs, alerts, temporary permissions, and rollout artifacts were reviewed.
- Evidence and follow-up actions were recorded without sensitive data.
Make rotation part of service onboarding and deployment readiness rather than an isolated security task. When platform teams standardize the inventory fields, secret-manager integration, deployment reload behavior, and validation checks, scheduled changes become easier to repeat and emergency revocation becomes faster and less disruptive.