Kubernetes Secrets Management Failures

Kubernetes Secrets Management Failures

The Secret was locked down. Its value was in a crash log.

What Is Kubernetes Secrets Management Failures?

A Kubernetes Secret is base64 rather than encryption, and a Secret injected as an environment variable travels with the process environment into logs and crash dumps. You'll be refused the Secret itself, find its value printed in a crash log you were allowed to read, and use it against a payout API. Then you'll mount the key as a file, encrypt Secrets at rest with a KMS provider and rotate the leaked key. Maps to OWASP Kubernetes Top Ten K03, Secrets Management Failures.

What You'll Learn in Kubernetes Secrets Management Failures

Kubernetes Secrets Management Failures — Training Steps

  1. Try the Secret directly

    Bob holds a Morrowkin developer's kubeconfig, taken from a compromised laptop. Developers can read pods and logs in the payments namespace. He tries the obvious first: reading the payout service's Secret. Morrowkin moves money from card payments into merchants' bank accounts.

  2. A worker that keeps crashing

    Developers may read pods. One of the payout workers has restarted many times.

  3. Read the crash log

    Developers may read logs too. Bob reads the output of the crashed worker's last run.

  4. Use the payout key

    Bob calls the payout API with the key, from his own machine.

  5. Check your understanding

  6. A payout destination changed

    Morrowkin's fraud team reports that a merchant's payout destination was changed through the API by an unknown caller using the production key.

  7. Open the payout worker

    Alice opens the payout worker's deployment and the API server's encryption configuration.

  8. A Secret turned into an environment variable

    The key is read from the Secret, but it is handed to the process as an environment variable. Environment variables are inherited by child processes, shown by debugging tools and, here, printed by a crash handler.

  9. Mount the key as a file

    Alice mounts the Secret as a read-only file and tells the worker where to read it. A file is not part of the environment, so a dump of the environment no longer contains the key.

  10. Encryption at rest is off

    Secrets are stored in etcd. With the identity provider first, they are stored there exactly as they arrive: base64, readable by anyone who can read etcd or its backups.

Security Framework Coverage

OWASP Kubernetes Top 10

  • K03:2025 Secrets Management Failures

CWE

  • CWE-312 Cleartext Storage of Sensitive Information
  • CWE-522 Insufficiently Protected Credentials

MITRE ATT&CK

  • T1552.007 Unsecured Credentials: Container API

CIS Controls

  • CIS 3 Data Protection
  • CIS 4 Secure Configuration of Enterprise Assets and Software

NIST CSF

  • PR.AT-02 Individuals in specialized roles are provided with awareness and training so that they possess the knowledge and skills to perform relevant tasks with cybersecurity risks in mind
  • PR.DS Data Security
  • PR.PS Platform Security

NICE Framework

  • IO-WRL-005 Systems Administration
  • DD-WRL-001 Cybersecurity Architecture