Secrets Rotation Checklist for Cloud, CI/CD, and Kubernetes Workloads

Overview

A secret is any sensitive value used to authenticate or authorize a system: a database password, API key, private certificate key, webhook token, signing secret, registry credential, or service account credential. Secrets rotation means replacing that value with a new one and confirming that all legitimate consumers use the replacement before the old value is disabled.

The safest process is not simply “generate a new password and update the application.” It is a controlled change with four distinct phases:

1. Discover: identify the secret, its consumers, owner, expiration requirements, and current storage locations.
2. Prepare: create the replacement, confirm permissions, and make the application capable of receiving it without an outage.
3. Switch: distribute the replacement, reload or redeploy consumers, and verify successful use.
4. Retire: revoke or delete the previous value only after the transition is proven complete.

Use a dual-key or overlapping-validity approach whenever the provider supports it. The old and new credentials remain valid during a short migration window, allowing consumers to change independently. If a system permits only one active credential, schedule the change as an explicit maintenance operation, test the rollback path, and make the dependency owner part of the approval.

Secret storage should be separate from application code and ordinary configuration wherever practical. A cloud secret manager, dedicated vault, CI/CD secret store, or Kubernetes-integrated solution can centralize access controls and auditing. The specific product matters less than having clear ownership, narrowly scoped access, controlled delivery, and a tested recovery process.

Checklist by scenario

1. Start with an inventory and dependency map

Before changing a value, create or update a record for each secret. At minimum, capture:

• Secret name, system, environment, and data classification.
• Business and technical owner, escalation contact, and approving team.
• Issuer or provider, creation date, last rotation date, and planned next review.
• Every known consumer: services, workers, scheduled jobs, deployment systems, integrations, and local operational tooling.
• Storage and delivery locations, including secret manager paths, CI/CD variables, manifests, mounted files, and environment variables.
• Rotation method, overlap window, reload behavior, and rollback method.

Search source repositories, deployment configuration, infrastructure code, pipeline definitions, runtime configuration, and provider access logs for references. A repository search is useful but not sufficient: dynamically constructed names, manually configured integrations, and third-party systems can be missed. Record uncertainty instead of marking an unverified dependency as complete.

2. Rotate cloud credentials and managed-service secrets

1. Confirm that the person or automation performing the change has permission to create, update, disable, and audit the credential.
2. Check whether the provider supports multiple active credentials, staged versions, aliases, or a grace period.
3. Create the replacement with the minimum required permissions. Do not use rotation as a reason to expand access.
4. Store the new value in the approved secret manager. Avoid copying it into tickets, chat, shell history, or temporary files.
5. Update the consuming service or integration and reload, restart, or redeploy it according to its documented behavior.
6. Run a low-risk authentication check and inspect application logs, metrics, and provider activity for failures.
7. Revoke the previous credential only after all consumers are confirmed to use the replacement.
8. Record timestamps, actors, credential identifiers, verification results, and the final disposition of the old value.

A provider comparison should focus on operational capabilities rather than brand preference:

3. Rotate CI/CD variables and deployment credentials

List secrets used by source control, build runners, artifact registries, deployment tools, infrastructure automation, and release notifications. Identify whether each value is available to pull requests, forked builds, logs, or only protected environments.

• Create the replacement in the destination system before changing the pipeline reference.
• Use masked or protected variables where supported, but do not treat masking as permission control.
• Review scripts for commands that echo environment variables, print request headers, or expose authentication errors.
• Run a test build or deployment using a safe target, then verify the intended production path.
• Disable the previous token after checking scheduled workflows, manual release jobs, and less frequently used environments.

For broader pipeline controls, pair this process with the DevSecOps Pipeline Checklist and the CI/CD Pipeline Design Guide.

4. Rotate Kubernetes Secrets

First determine how the workload consumes the secret: as an environment variable, mounted file, image-pull credential, service account token, or external secret reference. Updating a Kubernetes Secret object does not guarantee that every running process immediately uses the new value. Environment variables commonly require a pod restart, while mounted-file behavior depends on the delivery mechanism and application reload support.

1. Confirm the target namespace, workload, secret key, and owning deployment or operator.
2. Generate the replacement outside the manifest repository unless the repository is explicitly designed for encrypted secret data.
3. Update the approved secret store or Kubernetes Secret through the established deployment path.
4. Trigger a controlled rollout when the application reads the value only at startup.
5. Check pod readiness, rollout status, application authentication, and dependent operations.
6. Ensure old pods, jobs, replicas, and nodes cannot continue using the retired credential.

Do not assume that deleting and recreating a Secret is harmless. Confirm how controllers, mounted volumes, rollout strategies, and replicas behave in the target cluster. The Kubernetes Deployment Checklist provides a useful companion for validating rollout and readiness controls.

What to double-check

• Application reload behavior: Does the process reread the value, require a restart, or cache it until the next connection?
• Connection pools: Existing database or API connections may continue briefly while new connections use the replacement. Test both paths.
• Hidden consumers: Check scheduled tasks, disaster-recovery environments, staging systems, support scripts, and infrequent integrations.
• Format and encoding: Preserve required prefixes, escaping, line breaks, certificate chains, and time zones. A valid secret in the wrong format still causes failure.
• Permissions: Compare the replacement's access with the intended policy. Rotation should not silently grant broader privileges.
• Rollback: Define who can restore the previous version, for how long, and under what evidence. A rollback is not complete until the application is verified again.
• Observability: Establish the expected success signal before the change: authentication metrics, request health, deployment status, queue processing, or a synthetic transaction.
• Exposure cleanup: Search recent logs, build artifacts, crash reports, tickets, and shell history for accidental disclosure. If the old value may have leaked, revoke it rather than relying on rotation alone.

Useful audit evidence includes the change request, approval, secret identifier rather than the secret value, initiating identity, timestamps, affected environments, verification output, failed attempts, rollback decisions, and revocation confirmation. Evidence should demonstrate control without reproducing sensitive material.

Common mistakes

• Rotating without an inventory: An unlisted consumer can keep using the old credential and fail when it is revoked.
• Updating storage but not the workload: The secret manager or Kubernetes object may contain the new value while running processes still hold the old one.
• Revoking too early: Immediate revocation removes the overlap needed to diagnose propagation delays and partial rollouts.
• Testing only the primary path: A health endpoint may pass while a background worker, export job, or deployment runner is broken.
• Putting secrets in infrastructure code: Terraform, manifests, logs, and CI output can preserve values in state, history, or artifacts. Follow the Terraform Best Practices guide and use approved secret references.
• Using one shared credential everywhere: A shared value makes ownership, least privilege, and safe rollback harder. Prefer separate credentials by application, environment, or function when practical.
• Calling manual work a runbook: A procedure should name the owner, commands or interfaces, checks, abort conditions, and evidence required—not just say “update the secret.”

When to revisit

Review this checklist before every planned rotation, access review, seasonal planning cycle, or major deployment-platform change. Revisit it immediately after an incident, suspected credential exposure, provider migration, namespace redesign, CI/CD runner change, or application change that affects configuration loading.

Set a schedule based on credential type, exposure, provider capability, and risk rather than applying one interval to every secret. High-impact credentials deserve a clearly assigned owner, an explicit expiration or review date, and a tested emergency procedure. Lower-risk values still need inventory and ownership so that they do not become forgotten access paths.

Before the next rotation, take these practical actions:

1. Choose one representative cloud, pipeline, and Kubernetes secret.
2. Map every consumer and document how each process reloads the value.
3. Test a dual-key or staged rotation in a non-production environment.
4. Define verification signals, rollback authority, and the revocation deadline.
5. Store the completed record with the relevant change and audit evidence.
6. Update the runbook whenever a tool, workflow, provider capability, or workload behavior changes.

Effective secrets management is a repeatable operational workflow, not a one-time cleanup. Keeping the inventory current and practicing the transition makes future rotations safer, faster, and easier to audit.