Kubernetes Secrets & ConfigMaps: Configuration Management, etcd Encryption, Kubelet Caching, and External Secret Stores¶
How dynamic configuration and sensitive payload state land inside pods, how the control plane stores and protects them, and how node-level runtime engines propagate or isolate them. This chapter is the configuration counterpart of chapter 11 (Pod Internals), chapter 04 (etcd Internals), chapter 07 (Authentication & Authorization), and chapter 19 (Storage CSI).
The goal: by the end, you will understand the exact mechanics of ConfigMap and Secret API objects—from etcd storage serialization and envelope encryption to kubelet cache propagation strategies, atomic symlink tree swaps, subPath bind-mount pitfalls, immutable: true watch reduction, Secret Store CSI Driver / External Secrets Operator integrations, and enterprise secret rotation patterns.
Table of Contents¶
- The Configuration & Secrets Object Model
- 1.1 First-Principles: What are ConfigMaps & Secrets?
- 1.2 Object Specification & Basic Schemas
- 1.3 YAML Syntax Deep Dive in
data: Multiline & Scalar Block Types - 1.4 YAML Block Scalar Rosetta Stone Matrix
- 1.5 Real-World Example: Embedding Multiple Config Files
- etcd Storage Mechanics & Encryption at Rest
- Consumption Mechanism 1: Environment Variables (
env&envFrom) - Consumption Mechanism 2: Volume Mounts & Atomic Symlink Swaps
- The Kubelet Volume Manager & Propagation Engine
- The
subPathStatic Bind-Mount Trap - Control-Plane Scalability:
immutable: true - External Secret Management & Enterprise Integrations
- Secret Rotation Strategies & Signal Handlers
- RBAC Scoping & Security Boundaries
- Staff-Level Pitfalls & Anti-Patterns
- TL;DR Reference Card
1. The Configuration & Secrets Object Model¶
Kubernetes separates application logic (container images) from application parameters (ConfigMap) and sensitive payload credentials (Secret). Both are first-class API objects stored in etcd, but they have distinct specs, encoding contracts, and intended lifecycle constraints.
1.1 First-Principles: What are ConfigMaps & Secrets?¶
If you are new to Kubernetes, think of container images (docker build) as immutable software binaries—like a compiled .exe or an installed application package. You should never hardcode database hostnames, API URLs, feature flags, or passwords inside a container image.
Why?
1. Environment Mobility: The exact same container image must run in dev, staging, and production. Rebuilding an image just to change a database port or log level violates 12-Factor App principles and breaks software verification.
2. Security: Hardcoding passwords or TLS keys into an image means anyone with access to your container registry can steal your credentials.
Kubernetes provides two core objects to solve this:
* ConfigMap: Holds non-sensitive configuration data (key-value pairs, configuration file templates like nginx.conf, redis.conf, app.json, or environment variables).
* Secret: Holds sensitive configuration data (database passwords, API tokens, TLS private keys, SSH keys). Secrets have extra protection mechanisms like Base64 API representation, RBAC scoping, tmpfs RAM disk node mounts, and optional KMS etcd encryption.
How Do They Reach Your Container?¶
Once created in Kubernetes, ConfigMaps and Secrets can be delivered to your running application container in two fundamentally different ways:
┌─────────────────────────────────────────────────────────────────────────────┐
│ KUBERNETES API SERVER │
│ ┌───────────────────┐ ┌───────────────────┐ │
│ │ ConfigMap Object │ │ Secret Object │ │
│ └─────────┬─────────┘ └─────────┬─────────┘ │
└──────────────────┼─────────────────────────────────┼────────────────────────┘
│ │
┌───────────┴─────────────────────────────────┴───────────┐
│ TWO CONSUMPTION PATHS │
└─────────────┬─────────────────────────────┬─────────────┘
│ │
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ Path A: Environment Vars │ │ Path B: Mounted Files │
│ (process memory env) │ │ (files in directory) │
│ │ │ │
│ CONTAINER OS PROCESS │ │ CONTAINER FILESYSTEM │
│ DB_HOST=db.prod.local │ │ /etc/config/nginx.conf │
│ DB_PASS=supersecret │ │ /etc/secrets/tls.key │
└───────────────────────────┘ └───────────────────────────┘
1.2 Object Specification & Basic Schemas¶
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: prod
data:
# Simple key-value pair
LOG_LEVEL: "debug"
# Embedded configuration file using literal block scalar (|)
game.properties: |
enemies=aliens
lives=3
allowed=true
binaryData:
favicon.ico: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9h... # Base64 encoded raw bytes
---
apiVersion: v1
kind: Secret
metadata:
name: app-db-credentials
namespace: prod
type: Opaque
stringData:
DB_PASSWORD: "super-secret-pass" # Write-only convenience field (converted to base64 in data by apiserver)
data:
DB_USER: cG9zdGdyZXM= # Base64 encoded "postgres"
Field Mechanics¶
data: Key-value map (map[string]string). ForConfigMap, values must be valid UTF-8 strings. ForSecret, values are Base64-encoded byte arrays.stringData(Secret only): Write-only field provided for human convenience. UponPOST/PUT/PATCH, thekube-apiserverencodesstringDatavalues into Base64, populates thedatafield, and clearsstringData. It is never returned onGET/LIST.binaryData(ConfigMap only): Used to store unencoded binary data (e.g., gzip tarballs, raw certificates, image icons). Stored internally as Base64-encoded strings but distinct fromdatafor OpenAPI schema validation.
1.3 YAML Syntax Deep Dive in data: Multiline & Scalar Block Types¶
One of the most confusing parts for developers writing ConfigMaps and Secrets is the YAML syntax inside data:. You will see single quotes, double quotes, unquoted text, pipe symbols (|), strip pipes (|-), keep pipes (|+), and folded brackets (>).
Each syntax controls how newlines, spaces, and escape sequences are processed when Kubernetes turns your YAML into an actual string or mounted file inside the container.
YAML STRING SYNTAX CHOICES IN DATA
│
┌──────────────────────────┴──────────────────────────┐
▼ ▼
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
│ Inline Single-Line Syntaxes │ │ Multiline Block Scalar Syntaxes │
│ │ │ │
│ • Plain unquoted: KEY: value │ │ • Literal Clip (|): keeps \n │
│ • Single-quoted: KEY: 'val\n' │ │ • Literal Strip (|-): no \n │
│ • Double-quoted: KEY: "val\n" │ │ • Literal Keep (|+): all \n │
│ │ │ • Folded Clip (>): \n -> space │
│ │ │ • Folded Strip (>-): \n -> space│
└─────────────────────────────────┘ └─────────────────────────────────┘
1. Inline Single-Line Syntaxes¶
- Plain Unquoted (
KEY: value): Raw string value. Trap: YAML automatically parses unquoted numbers (8080), booleans (true), or nulls (null) as integers/booleans. Since Kubernetesdatafields must be strings,PORT: 8080will trigger an API server error:cannot unmarshal number into Go struct field of type string. Numbers and booleans must be quoted (PORT: "8080"). - Single Quoted (
KEY: 'value'): Raw literal string. No escape sequences are evaluated.'\n'is treated as two literal characters (\andn). - Double Quoted (
KEY: "value"): Evaluates escape sequences."line1\nline2"will be parsed into a two-line string containing an actual newline byte (0x0A).
2. Multiline Block Scalar Syntaxes (Embedding Whole Files)¶
When embedding entire configuration files (e.g., nginx.conf, redis.conf, prometheus.yml, shell scripts) into a ConfigMap data key, YAML multiline block scalars are used.
A. Literal Block Scalar: | (Literal Clip - Default)¶
- Behavior: Preserves all internal newlines exactly as written and appends exactly one trailing newline at the end.
- Primary Use Case: The standard choice for embedding configuration files (
nginx.conf,application.yaml, shell scripts).
/etc/config/nginx.conf):
B. Literal Strip Block Scalar: |- (Literal Strip)¶
- Behavior: Preserves all internal newlines but strips ALL trailing newlines at the end of the block.
- Primary Use Case: Storing single-line tokens, RSA public keys, or certificates where a trailing
\ncauses cryptographic hash checks, HTTP header parsing, or string comparisons to fail.
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.sD5f... (No trailing \n).
C. Literal Keep Block Scalar: |+ (Literal Keep)¶
- Behavior: Preserves all internal newlines AND keeps all trailing blank lines typed at the end of the YAML block.
- Primary Use Case: Files where specific blank lines at the end of the file are required by legacy parsers.
D. Folded Block Scalar: > (Folded Clip)¶
- Behavior: Folds single line breaks within paragraphs into spaces, but preserves double line breaks (blank lines) as paragraph breaks. Appends one trailing newline.
- Primary Use Case: Long single-line text strings (such as legal notices, SQL queries, or multiline bash commands) that you want to split across multiple lines in YAML for readability without inserting actual newlines in the result.
"Welcome to our Kubernetes cluster. This application is running in production.\n"
E. Folded Strip Block Scalar: >- (Folded Strip)¶
- Behavior: Folds single line breaks into spaces and strips all trailing newlines.
data:
DATABASE_URL: >-
postgres://dbuser:secretpass@postgres-service.prod.svc.cluster.local:5432/production_db?sslmode=require
"postgres://dbuser:secretpass@postgres-service.prod.svc.cluster.local:5432/production_db?sslmode=require" (Single line, no trailing \n).
1.4 YAML Block Scalar Rosetta Stone Matrix¶
The following reference matrix shows exact input YAML syntaxes and their resulting string representations inside Kubernetes:
| YAML Syntax | Name | Interior Newlines | Trailing Newlines | Example Input | Resulting String Value | Primary Use Case |
|---|---|---|---|---|---|---|
key: "val" |
Double-Quoted | Escaped (\n) |
None | A: "foo\nbar" |
"foo\nbar" |
Inline string with explicit escape sequences. |
key: 'val' |
Single-Quoted | Literal string | None | A: 'foo\nbar' |
"foo\\nbar" |
Raw string where backslashes must not be escaped. |
key: \| |
Literal Clip | Preserved | Exactly 1 \n |
A: \|\n line1\n line2\n |
"line1\nline2\n" |
Standard config files (nginx.conf, app.yaml). |
key: \|- |
Literal Strip | Preserved | 0 (Stripped) | A: \|-\n line1\n line2\n |
"line1\nline2" |
Single-line secrets, API tokens, certs, hashes. |
key: \|+ |
Literal Keep | Preserved | All kept | A: \|+\n line1\n\n\n |
"line1\n\n\n" |
Preserving exact file trailing whitespace. |
key: > |
Folded Clip | Converted to space | Exactly 1 \n |
A: >\n line1\n line2\n |
"line1 line2\n" |
Wrapping long human text into readable YAML lines. |
key: >- |
Folded Strip | Converted to space | 0 (Stripped) | A: >-\n line1\n line2\n |
"line1 line2" |
Long URLs or connection strings split for readability. |
1.5 Real-World Example: Embedding Multiple Config Files in One ConfigMap¶
In production Kubernetes deployments, a single ConfigMap often holds multiple distinct configuration files that get mounted into a container directory.
apiVersion: v1
kind: ConfigMap
metadata:
name: webserver-config
namespace: prod
data:
# Key 1: Becomes /etc/nginx/nginx.conf
nginx.conf: |
user nginx;
worker_processes auto;
events { worker_connections 1024; }
http {
include /etc/nginx/conf.d/*.conf;
}
# Key 2: Becomes /etc/nginx/conf.d/default.conf
default.conf: |
server {
listen 80;
location / {
proxy_pass http://localhost:8080;
}
}
# Key 3: Single line environment variable
MAX_CONNECTIONS: "5000"
When mounted as a volume at /etc/nginx/, Kubernetes creates files named nginx.conf, default.conf, and MAX_CONNECTIONS inside /etc/nginx/ containing the exact scalar string content.
1.2 Secret Types Matrix¶
Kubernetes uses the type field to enforce structural validation and semantic expectations for built-in controllers.
| Secret Type | Required Data Keys | Purpose / Primary Consumer |
|---|---|---|
Opaque |
Arbitrary user keys | Default type for user application credentials. |
kubernetes.io/service-account-token |
token, ca.crt, namespace |
Auto-generated service account tokens (legacy long-lived; projected tokens preferred). |
kubernetes.io/dockercfg |
.dockercfg |
Legacy Docker v1 authentication format. |
kubernetes.io/dockerconfigjson |
.dockerconfigjson |
Docker v2 JSON auth config used by kubelet for imagePullSecrets. |
kubernetes.io/basic-auth |
username, password |
HTTP Basic Authentication credentials. |
kubernetes.io/ssh-auth |
ssh-privatekey |
SSH private keys (e.g., git repo access). |
kubernetes.io/tls |
tls.crt, tls.key |
X.509 TLS certificate and private key pairs (used by Ingress, Istio, cert-manager). |
bootstrap.kubernetes.io/token |
token-id, token-secret |
Temporary tokens used during kubeadm join node bootstrapping. |
2. etcd Storage Mechanics & Encryption at Rest¶
A common operational misconception is that standard Kubernetes Secrets are encrypted by default. Base64 is an encoding format, not an encryption algorithm. Without explicit configuration, Secrets are stored in etcd as unencrypted, plaintext Base64 strings. Anyone with read access to etcd (or etcd backups) can read every secret in the cluster.
2.1 The etcd Storage Pipeline¶
kubectl apply / REST Client
│
▼
┌─────────────────┐
│ kube-apiserver │
└────────┬────────┘
│ Authentication & Authorization (RBAC)
▼
┌─────────────────┐
│ Mutating Webhook│
└────────┬────────┘
│
▼
┌──────────────────────────────────┐
│ Storage Transformer Layer │
│ │
│ ┌──────────────────────────────┐ │
│ │ Encryption Provider Chain │ │
│ │ (Identity / AES / KMS v2) │ │
│ └──────────────┬───────────────┘ │
└────────────────┼─────────────────┘
│ Serialized protobuf / encrypted envelope
▼
┌─────────────────┐
│ etcd Storage │ Key: /registry/secrets/prod/app-db-credentials
└─────────────────┘ Value: k8s:enc:kms:v2:provider-aws:base64bytes...
2.2 Encryption at Rest Architecture¶
To encrypt secrets in etcd, kube-apiserver must be launched with the flag --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps # Optional: encrypt sensitive ConfigMaps if required by compliance
providers:
- kms:
apiVersion: v2
name: aws-kms-provider
endpoint: unix:///var/run/kms-plugin/socket.sock
timeout: 3s
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIGEgc2VjcmV0IGlzIGEgc2VjcmV0IQ==
- identity: {} # Fallback: allows reading unencrypted legacy secrets
Provider Hierarchy & Security Properties:¶
identity: Default provider. Plaintext storage (no encryption).secretbox: XSalsa20 + Poly1305 symmetric encryption. Strong cipher, manual key management.aescbc/aesgcm: AES-CBC / AES-GCM with PKCS#7 padding. Requires manual key rotation in configuration files.kms(v2): Production Staff Standard. Envelope encryption backed by an external Hardware Security Module (HSM) or Cloud Key Management Service (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault).
2.3 KMS v2 Envelope Encryption Architecture¶
KMS v2 eliminates control-plane performance bottlenecks by using local Data Encryption Keys (DEKs) encrypted by a remote Key Encryption Key (KEK).
┌───────────────────────────────────────────────┐
│ kube-apiserver │
│ │
│ 1. Generate local DEK (random AES-256 key) │
│ 2. Encrypt Secret payload locally with DEK │
└───────┬───────────────────────────────┬───────┘
│ │
3. Send raw DEK over │ │ 5. Store Encrypted DEK
Unix Domain Socket │ │ + Encrypted Payload
▼ ▼
┌───────────────┐ ┌───────────────┐
│ KMS v2 Plugin │ │ etcd │
└───────┬───────┘ └───────────────┘
│ 4. Encrypt DEK with KEK
▼
┌───────────────┐
│ External KMS │ (AWS KMS / Vault / GCP KMS)
│ (Holds KEK) │
└───────────────┘
- Write Path:
apiservergenerates a unique DEK locally, encrypts the Secret payload with the DEK, sends the raw DEK to the KMS plugin over gRPC (Unix socket), receives the encrypted DEK (encrypted by KEK), and writes[Encrypted DEK + Encrypted Payload]into etcd. - Read Path:
apiserverreads the blob from etcd, sends the Encrypted DEK to KMS plugin for decryption, receives the raw DEK, and decrypts the Secret payload in memory. - DEK Caching: KMS v2 caches decrypted DEKs in memory using key IDs, avoiding per-read network round-trips to external Cloud KMS systems.
3. Consumption Mechanism 1: Environment Variables (env & envFrom)¶
Environment variables are the simplest way to consume ConfigMap and Secret data, but they carry severe operational and security trade-offs.
3.1 Spec Declarations¶
spec:
containers:
- name: app
image: myapp:1.0
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: DB_HOST
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: app-db-credentials
key: DB_PASSWORD
optional: false # Pod fails to start if key/secret is missing
envFrom:
- configMapRef:
name: app-config
- secretRef:
name: app-db-credentials
3.2 Runtime Behavior & Frozen State¶
Critical Rule: Environment variables are resolved once during container creation by the container runtime (
containerd/CRI-O). They are permanently frozen for the lifetime of the process.
If a user modifies a ConfigMap or Secret in the API server:
1. Existing running pods do not receive the update.
2. The environment variables inside /proc/<pid>/environ remain unchanged.
3. Applications must be restarted (e.g., via a rolling Deployment restart) to see updated environment variable values.
3.3 Security & Operational Exposure Risks¶
Env vars are widely considered an anti-pattern for sensitive secrets due to OS-level leakage vectors:
- Process Listing Exposure (
ps aux): In many legacy Linux environments or debugging containers, child processes or monitoring tools can view environment variables of running processes via/proc/<pid>/environ. - Crash Dumps & Application Logs: Application frameworks (Django, Node.js, Spring Boot) often print the full environment map (
process.env/os.environ) to stderr during unhandled exceptions or boot logs. - Child Process Inheritance: Any sub-process spawned via
fork()/exec()inherits the full parent process environment, exposing secrets to unauthorized scripts or third-party binaries. - No Granular Scoping:
envFromimports every key in aConfigMap/Secretinto the environment, creating accidental collisions with existing environment variables (PATH,HOST,PORT).
4. Consumption Mechanism 2: Volume Mounts & Atomic Symlink Swaps¶
Volume mounts provide dynamic, live updates of ConfigMap and Secret data inside containers without restarting pods.
4.1 Volume Specification¶
spec:
containers:
- name: app
image: myapp:1.0
volumeMounts:
- name: config-volume
mountPath: /etc/config
readOnly: true
- name: secret-volume
mountPath: /etc/secrets
readOnly: true
volumes:
- name: config-volume
configMap:
name: app-config
defaultMode: 0640
- name: secret-volume
secret:
secretName: app-db-credentials
defaultMode: 0400
4.2 The Atomic Symlink Tree Engine¶
When the kubelet mounts a ConfigMap or Secret volume into a container, it does not write flat files directly into the directory. Instead, it constructs an atomic symlink tree to allow instant, non-disruptive rotation across all mounted keys.
Directory Layout Inside Container Mount Path (/etc/config):¶
/etc/config/
├── ..data -> ..2026_08_04_07_30_00.123456789 (symlink to active timestamp dir)
├── ..2026_08_04_07_30_00.123456789/ (real directory containing files)
│ ├── game.properties
│ └── LOG_LEVEL
├── game.properties -> ..data/game.properties (symlink pointing via ..data)
└── LOG_LEVEL -> ..data/LOG_LEVEL (symlink pointing via ..data)
Atomic Rotation Sequence when ConfigMap is Updated:¶
- Kubelet creates a new timestamped directory:
/etc/config/..2026_08_04_07_35_42.987654321. - Kubelet writes the updated key files into this new directory.
- Kubelet creates a new temporary symlink:
/etc/config/..data_tmp -> ..2026_08_04_07_35_42.987654321. - Kubelet calls
renameat(2)to atomically swap/etc/config/..data_tmpto/etc/config/..data. - Kubelet asynchronously deletes the old timestamped directory (
..2026_08_04_07_30_00.123456789).
BEFORE UPDATE:
/etc/config/game.properties ────────► ..data/game.properties ────────► ..2026_08_04_07_30_00/game.properties
AFTER ATOMIC RENAME (renameat):
/etc/config/game.properties ────────► ..data/game.properties ────────► ..2026_08_04_07_35_42/game.properties
Why Atomic Symlinks Matter:¶
Applications opening /etc/config/game.properties will never read a partially written file or an empty buffer during a config update. The read operation either lands on the old directory or the new directory instantly.
5. The Kubelet Volume Manager & Propagation Engine¶
How does the kubelet detect that a ConfigMap or Secret has changed in etcd, and how long does it take for changes to reflect on disk?
5.1 Kubelet Change Detection Strategies¶
The kubelet's sync behavior is governed by the --configMapAndSecretChangeDetectionStrategy flag on kubelet.
┌──────────────────────────────────────────────┐
│ kube-apiserver │
└───────▲──────────────────────▲───────────────┘
│ │
Watch │ │ Get / List
(Event stream) │ │ (Polling)
│ │
┌────────────┴─────────────┐ ┌─────┴────────────────────┐
│ Strategy 1: Watch (Def) │ │ Strategy 3: Get │
└────────────┬─────────────┘ └─────┬────────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────┐
│ Kubelet Volume Manager │
│ - Local Cache │
│ - Sync Loop (default: 1 minute) │
└──────────────────────┬───────────────────────┘
│
▼ Atomic Symlink Swap
┌──────────────────────────────────────────────┐
│ Container Filesystem Mount (/etc/config) │
└──────────────────────────────────────────────┘
Watch(Default): Kubelet establishes a continuous APIWATCHconnection for allConfigMapandSecretobjects referenced by active pods on its node. When etcd changes, apiserver pushes an update event immediately to kubelet.Cache: Kubelet caches objects in an internal TTL cache (--sync-frequency, default 1m). Updates are picked up on cache expiration.Get: Kubelet issues a direct HTTPGETrequest tokube-apiserverevery time the pod sync loop executes. High API server load; anti-pattern for large clusters.
5.2 End-to-End Propagation Latency Math¶
The total delay between running kubectl apply -f configmap.yaml and seeing the new content inside the container file is:
- Watch Push: \(\approx 10-100 \text{ ms}\)
- Kubelet Sync Period: Controlled by
syncFrequency(default 1 minute). - TTL Cache Buffer: Controlled by
configMapAndSecretChangeDetectionStrategycache parameters. - Total Expected Delay: \(0 \text{ to } 60 \text{ seconds}\).
6. The subPath Static Bind-Mount Trap¶
A widespread configuration mistake occurs when mounting a single key from a ConfigMap or Secret into a directory using subPath.
6.1 The Broken subPath Spec¶
spec:
containers:
- name: nginx
image: nginx:1.27
volumeMounts:
- name: config-volume
mountPath: /etc/nginx/nginx.conf # Intended to override ONLY nginx.conf
subPath: nginx.conf # <--- THE TRAP!
volumes:
- name: config-volume
configMap:
name: nginx-config
6.2 Why subPath Breaks Live Propagation¶
When subPath is specified, the Linux kernel performs a direct file-to-file bind mount (mount --bind /tmp/proc/..data/nginx.conf /etc/nginx/nginx.conf).
Standard Mount (Dynamic Symlink):
/etc/nginx/ -> symlink -> ..data -> ..2026_08_04_07_35_42/ -> [Target File] (SWAPPABLE)
subPath Mount (Static Inode Lock):
/etc/nginx/nginx.conf -> Direct Bind-Mount to Inode #1049281 (FROZEN TO SPECIFIC INODE)
- When the
ConfigMapis updated, the kubelet creates a new timestamp directory (..2026_08_04_08_00_00) with a new inode fornginx.conf. - The kubelet updates the
..datasymlink to point to the new directory. - However, the container's
subPathmount is directly pinned to the OLD inode (#1049281). - The container will NEVER see the updated content. It remains stuck on the version of the file that existed when the container was created.
Staff Engineer Rule: Never use
subPathif you require dynamic configuration updates. If you must mount a single file into an existing directory containing other files, use a dedicated sub-directory mount or a sidecar reloader pattern.
7. Control-Plane Scalability: immutable: true¶
In large-scale Kubernetes clusters (e.g., 5,000+ nodes, 100,000+ pods), holding active WATCH connections for tens of thousands of static ConfigMap and Secret objects imposes severe memory and CPU overhead on kube-apiserver and kubelet.
7.1 Immutable Resources (immutable: true)¶
Kubernetes 1.19+ supports marking ConfigMaps and Secrets as immutable.
apiVersion: v1
kind: ConfigMap
metadata:
name: static-app-config
namespace: prod
immutable: true # <--- Disables API server and Kubelet watches
data:
database.json: |
{"max_connections": 500}
7.2 Scalability Benefits & Internal Behavior¶
STANDARD CONFIGMAP (Watch Active):
etcd ◄──► apiserver ◄────── WATCH Stream ──────► Kubelet (Holds open TCP socket)
IMMUTABLE CONFIGMAP (Watch Dropped):
etcd ◄──► apiserver (No Watch Established) Kubelet (Zero Watch Overhead)
- Watch Termination: Once the kubelet mounts an
immutableConfigMaporSecret, it immediately closes itsWATCHconnection tokube-apiserverfor that object. - etcd & API Server Relief: Reduces memory footprint (
watchCache), context switches, and network TCP socket allocations on control-plane nodes by up to 60% in large clusters. - Modification Protection: Any attempt to issue a
PATCHorPUTto an immutable object is rejected bykube-apiserverwith422 Unprocessable Entity. - Rotation Workflow: To change an immutable config, you must create a new object with a new name (e.g.,
app-config-v2) and perform a rolling deployment update.
8. External Secret Management & Enterprise Integrations¶
Native Kubernetes Secrets have two structural shortcomings in enterprise environments: 1. Storing Secrets in Git repository manifests (GitOps) leads to plaintext credential leaks. 2. Native Secrets lack automatic rotation, fine-grained auditing, and central governance provided by enterprise vaults (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault).
8.1 Architecture: Secret Store CSI Driver vs External Secrets Operator (ESO)¶
ENTERPRISE SECRET STORES
┌─────────────────────────────────────────────┐
│ HashiCorp Vault / AWS Secrets Manager / │
│ GCP Secret Manager / Azure Key Vault │
└──────────────▲───────────────▲──────────────┘
│ │
gRPC Protocol │ │ REST API (HTTPS)
│ │
┌──────────────────────────┴────┐ ┌───────┴───────────────────────────┐
│ Secret Store CSI Driver │ │ External Secrets Operator (ESO) │
│ (In-flight Ephemeral Mount) │ │ (Syncs Vault -> Native K8s Secret)│
│ │ │ │
│ - Mounts directly as volume │ │ - Creates real K8s Secret object │
│ - No native K8s Secret created│ │ - Works with env vars & standard │
│ - Stored in tmpfs (RAM disk) │ │ volume mounts │
└──────────────┬────────────────┘ └───────────────┬───────────────────┘
│ │
▼ ▼
┌───────────────────────────────┐ ┌───────────────────────────────────┐
│ Pod Mount: /var/run/secrets/ │ │ Native Secret Object: │
│ (ephemeral tmpfs) │ │ apiVersion: v1 / Kind: Secret │
└───────────────────────────────┘ └───────────────────────────────────┘
8.2 Architectural Trade-Off Analysis¶
| Feature / Dimension | Secret Store CSI Driver | External Secrets Operator (ESO) | Sealed Secrets (Bitnami) |
|---|---|---|---|
| Storage in etcd? | No (Bypasses etcd completely; mounts directly from Vault to container tmpfs). |
Yes (Fetches from Vault and creates a native K8s Secret). |
Yes (Decrypts SealedSecret CRD into a native K8s Secret). |
| GitOps Safety | Excellent (Manifest references external Vault path). | Excellent (Manifest contains ExternalSecret CRD referencing Vault key). |
Excellent (Asymmetrically encrypted ciphertext checked into Git). |
| Env Var Support | Requires SecretProviderClass secret synchronization opt-in. |
Native (Produces normal K8s Secret consumed via env). |
Native (Produces normal K8s Secret consumed via env). |
| Rotation Support | Automatic autorotation on volume mount (enableAutoRotation: true). |
Polling sync interval (refreshInterval: 1h). |
Manual re-encryption required upon master key rotation. |
| Blast Radius | Smallest (Secret exists only in pod memory/tmpfs). | Medium (Secret present in etcd & namespace). | Medium (Secret present in etcd & namespace). |
9. Secret Rotation Strategies & Signal Handlers¶
Updating a ConfigMap or Secret volume on disk is only half the battle. The application running inside the container must be informed that the file has changed.
9.1 The Four Production Rotation Patterns¶
CONFIGMAP/SECRET UPDATED
│
┌──────────────────────────────────┼──────────────────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Pattern 1: │ │ Pattern 2: │ │ Pattern 3: │
│ In-Process │ │ Sidecar / Signal│ │ Reloader │
│ File Watcher │ │ (SIGHUP Engine) │ │ Controller │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
▼ ▼ ▼
Application reloads Sidecar detects file Controller updates Pod
config dynamically in change & sends SIGHUP spec hash, triggering
memory (inotify / fsnotify) to main process Rolling Deployment Update
Pattern 1: In-Process Dynamic Reload (fsnotify / inotify)¶
Applications (e.g., NGINX, Envoy, Prometheus, Go services) use OS filesystem notifications (inotify on Linux) to watch /etc/config/..data. When the symlink rename occurs, the watcher triggers an in-memory config reload.
Pattern 2: Sidecar Signal Generator (SIGHUP)¶
For legacy applications that cannot watch files but support reload signals (e.g., kill -HUP <pid>):
A sidecar container shares the process namespace (shareProcessNamespace: true) or volume mount, watches the config file, and sends SIGHUP to the main application process.
Pattern 3: Deployment Immutable Hash / Reloader Controller¶
If applications read config only at startup, use an automated controller like Reloader (or Kustomize configMapGenerator).
Reloader watches ConfigMap/Secret changes and automatically injects an annotation containing the content hash into the Deployment template:
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
spec:
template:
metadata:
annotations:
config.k8s.io/hash: "a8f9c3d2e1b4..." # Triggers rolling update of Pods
Pattern 4: Immutable ConfigMap Name Versioning (Staff Best Practice)¶
Append a version suffix or content hash to the ConfigMap name (app-config-v1 \(\rightarrow\) app-config-v2). Update the Deployment spec to point to app-config-v2. This triggers a standard Kubernetes rolling update with full canary, rollback, and readiness gate protection.
10. RBAC Scoping & Security Boundaries¶
Secrets are primary targets for privilege escalation in Kubernetes clusters. Improper RBAC permissions can allow an unprivileged user or compromised pod to steal cluster-admin credentials.
10.1 The LIST/WATCH Privilege Escalation Vector¶
Security Rule: Never grant
listorwatchpermissions onsecretsglobally or across namespaces unless strictly required by security operators.
# DANGEROUS ROLE: Grants access to ALL secrets in namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: prod
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"] # <--- "list" allows dumping every secret at once!
- Why
listis Dangerous: A service account withgeton a specific secret name can only fetch that single secret. A service account withlistcan retrieve all secrets in the namespace in a single HTTP request (including TLS private keys, service account tokens, and database passwords). - Resource Names Restricting: Always scope RBAC
getaccess to specific secret instances usingresourceNames:
# SECURE ROLE: Scoped to specific Secret instance
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: app-secret-reader
namespace: prod
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-db-credentials"] # Restricts access to THIS secret only
verbs: ["get"]
11. Staff-Level Pitfalls & Anti-Patterns¶
11.1 The 1MB etcd Object Limit Crash¶
ConfigMaps and Secrets are stored as single keys in etcd. etcd enforces a hard maximum payload limit of 1MB per object (--max-request-bytes).
* Symptom: kubectl apply fails with Error from server (RequestEntityTooLarge): limit is 1048576 bytes.
* Fix: Do not embed large binary assets, fat Java JARs, or multi-megabyte dataset files into ConfigMaps. Use Persistent Volume Claims (PVCs), S3/GCS object storage, or OCI artifact registries instead.
11.2 The stringData Overwrite Footgun¶
When updating a Secret via kubectl apply, developers often forget that stringData is write-only. If a manifest specifies stringData alongside an existing data field, stringData will overwrite the corresponding keys in data silently upon apply.
11.3 Symlink Traversal Breakage in Custom Scripts¶
Custom shell scripts running inside containers that read mounted config files using fixed symlink dereferencing (e.g., cp /etc/config/file.txt /tmp/) will copy the static content at the time of copy, losing all future dynamic updates.
11.4 tmpfs Memory Overhead for Huge Secrets¶
Secrets mounted as volumes are backed by node memory (tmpfs). Mounting 500MB of secret files into a container consumes 500MB of node RAM and counts against the container's memory limits, potentially triggering an Out-Of-Memory (OOM) Kill.
12. TL;DR Reference Card¶
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ KUBERNETES SECRETS & CONFIGMAPS CHEAT SHEET │
├──────────────────────┬──────────────────────────────────┬──────────────────────────────┤
│ Concern │ ConfigMap │ Secret │
├──────────────────────┼──────────────────────────────────┼──────────────────────────────┤
│ Primary Purpose │ Non-sensitive application config │ Sensitive credentials/keys │
│ etcd Encoding │ Plaintext UTF-8 / Base64 binary │ Base64 data (Requires KMS) │
│ Default Node Mount │ Standard filesystem │ RAM-backed tmpfs (No swap) │
│ Max Size Limit │ 1 MB (etcd hard limit) │ 1 MB (etcd hard limit) │
├──────────────────────┴──────────────────────────────────┴──────────────────────────────┤
│ CONSUMPTION PATTERNS │
│ 1. Environment Vars │ STATIC / FROZEN at boot. No live updates. Exposed in /proc. │
│ 2. Volume Mounts │ DYNAMIC. Updated via atomic symlink swap (..data -> ..time). │
│ 3. subPath Mounts │ FROZEN. Static inode bind-mount; BREAKS dynamic updates. │
│ 4. immutable: true │ DROPS Kubelet watches. Reduces apiserver & etcd load by ~60%.│
├────────────────────────────────────────────────────────────────────────────────────────┤
│ ENTERPRISE BEST PRACTICES │
│ • Production Security : Enable KMS v2 envelope encryption in kube-apiserver. │
│ • GitOps & Rotation : Use Secret Store CSI Driver or External Secrets Operator (ESO).│
│ • RBAC Safety : Never grant "list/watch" on secrets globally; scope to names. │
│ • App Propagation : Use immutable CM names + rolling deploys, or fsnotify watchers.│
└────────────────────────────────────────────────────────────────────────────────────────┘