Kubernetes makes mounting secrets easy; it does not make secret lifecycle easy. Teams copy base64 blobs between namespaces, fork Helm charts with embedded credentials, or lean on long-lived tokens because rotation scripts are brittle. The result is sprawl: nobody knows which secret backs which workload, and revocation becomes a archaeology project.
Anchor on a single external source of truth—cloud KMS with CSI drivers, HashiCorp Vault, or your cloud provider’s secret manager—and treat cluster Secrets as ephemeral projections. Namespace-scoped access via RBAC and IAM-bound identities beats shared cluster-admin kubeconfigs.
Rotation needs automation with backoff and health checks. Prefer short-lived credentials (OIDC workload identity, dynamic database passwords) over static files. When you must store symmetric keys, document owners, TTLs, and break-glass procedures.
GitOps complicates the story in a good way: desired state stays in Git while secret references point to external IDs. Never commit plaintext; use sealed secrets or external-secret operators with tight scopes.
Audit readiness: maintain an inventory keyed by workload, classify data handled, and test restore paths quarterly. Exercises reveal whether your backups of secret metadata are as good as the secrets themselves.
Platform teams should publish golden patterns—one chart for mounting AWS Secrets Manager, one for Vault Agent sidecars—and measure adoption instead of mandating bespoke YAML per service.
Related: DevOps consulting and how we staff platform work.
A worked example: tracing one leaked credential across four systems
A team we advised discovered an exposed database password in a Slack message from eighteen months earlier, pasted during a debugging session and never rotated. Tracing its blast radius took three engineers two full days, because the credential had propagated further than anyone expected: it was baked into a Helm chart's values file committed to a private repo, mounted as a plain Kubernetes Secret in two namespaces, cached in a CI runner's environment from a pipeline that had since been deleted, and referenced by name in a runbook that on-call engineers still consulted during incidents.
None of those four locations showed up in a single inventory. The Helm chart lived in a different repo than the application code, the CI cache was invisible without runner-level access, and the runbook was a wiki page nobody had thought to search when the incident started. The credential itself was still valid the entire eighteen months, attached to a production database, because nothing forced rotation and no alert fired on an unusually old secret sitting untouched in the cluster.
The fix was not exotic—move the password into a secrets manager, mount it via a CSI driver, force ninety-day rotation, and add a scheduled job that flags any Secret object older than its rotation policy allows. The expensive part was the discovery, and that expense scales with how long sprawl goes unaddressed. A credential inventory built after the fact, in the middle of an incident, costs days per credential traced. A credential inventory built as a standing practice, checked automatically in CI and reviewed quarterly, costs minutes and catches the next stale secret before it turns into a war room with three engineers and a compliance team on the call.
The postmortem for that incident produced one more finding worth naming: the runbook itself was part of the sprawl. Runbooks get written once, during a stressful moment, and then treated as permanent reference material even though they embed exactly the kind of credential-by-name shortcut that a mature secrets practice is supposed to eliminate. Any audit of secret sprawl needs to include documentation, not just code and cluster state, or the fix will be undone the next time someone follows the runbook exactly as written.
A decision framework: when a CSI driver is enough, and when you need a full secrets operator
Teams often over-engineer or under-engineer this choice, and the right answer depends less on team size than on how many distinct trust domains your secrets cross. If every workload in a namespace can legitimately access the same set of credentials, and rotation cadence is uniform across them, a CSI secrets store driver pulling from your cloud provider's secret manager is enough—it mounts secrets as files or environment variables at pod start, keeps nothing persisted in etcd, and requires no new operator to maintain or upgrade on its own release cycle.
Reach for a dedicated secrets operator—Vault Agent injector, External Secrets Operator, or similar—once you have workloads with genuinely different trust levels sharing a cluster: a billing service that needs a payment processor key rotated weekly, alongside an internal reporting job that only needs read-only database credentials rotated quarterly. The operator gives you per-workload policy, dynamic secret generation for things like database credentials that do not need to exist until the moment a pod requests them, and an audit trail of exactly which pod fetched which secret and when. That audit trail is usually the deciding factor for teams under compliance pressure—when an auditor asks who accessed a specific credential in the last quarter, "we don't know, it was a mounted file" is a finding, not an answer, and it is the kind of finding that turns a routine audit into a remediation plan with a deadline.
The trap to avoid is adopting the heavier operator because it looks more serious on an architecture diagram, then never actually using the policy engine or the audit trail it exists to provide. We have seen clusters running Vault Agent sidecars on every pod, injecting secrets that are functionally static and never rotated, which is strictly worse than a CSI driver: more moving parts, more startup latency from the sidecar injection webhook, and none of the benefit the tool was adopted for. Match the tool to the actual trust boundary in front of you, not to what looks impressive when you are describing the platform to a new hire or a prospective enterprise customer during due diligence.
A useful gut check before committing to either path: count how many distinct rotation cadences and access policies your secrets actually need this quarter, not hypothetically next year. Most clusters we have reviewed need two or three tiers at most, and a surprising number would be fully served by a CSI driver plus disciplined namespace boundaries, with the secrets operator conversation deferred until a specific compliance requirement or a specific multi-tenancy need actually materializes. Write that count down and revisit it every two quarters—trust domains tend to multiply quietly as a platform grows, and the decision that was right at ten services is rarely still right at fifty. Treat it as a standing agenda item in your platform team's quarterly review, not a one-time architecture decision you make once and forget.