SmallStepCa
Great request! Let’s use the Feynman Technique to deeply understand how to run Smallstep CA in Docker Compose to issue external certificates. I’ll break this down step by step, in simple terms, as if teaching someone else.
🧠 Step 1: Explain it simply (as if teaching a beginner)
Smallstep CA (step-ca) is a certificate authority tool that you can run yourself.
It can issue TLS certificates to your servers, devices, or users.
We want to:
- Run step-ca in Docker with Docker Compose.
- Make it reachable externally (over the internet).
- Issue certificates for external domains (not just internal ones).
To do this, we’ll:
- Run a Smallstep CA container with persistent storage.
- Expose it via a reverse proxy (like Caddy or Nginx) so it’s internet-accessible.
- Bootstrap the CA with a provisioner (like ACME) so clients can request certificates automatically.
- Use DNS validation to prove we own the external domain.
🛠 Step 2: Identify key components
- step-ca: The Smallstep certificate authority service.
- step CLI: The command-line tool for interacting with the CA.
- ACME provisioner: Lets tools like Certbot or Caddy automatically get certs.
- Docker Compose: Runs everything in containers.
- Reverse Proxy: Exposes step-ca securely to the internet.
🧩 Step 3: Build it step by step
Here’s a minimal working example:
1️⃣ Create directories for persistent storage
mkdir -p ./step-ca/config ./step-ca/data
2️⃣ Initialize step-ca locally (generates keys, certs, config)
docker run --rm -it \
-v $(pwd)/step-ca/config:/home/step \
smallstep/step-ca step ca init \
--name "My Public CA" \
--dns "ca.example.com" \
--address ":9000" \
--provisioner admin@example.com \
--provisioner-password-file <(echo "changeit")
This generates:
- ca.json: config file for the CA.
- Root and intermediate keys/certs.
3️⃣ Docker Compose file
version: "3.8"
services:
step-ca:
image: smallstep/step-ca:latest
container_name: step-ca
restart: unless-stopped
ports:
- "9000:9000"
volumes:
- ./step-ca/config:/home/step
- ./step-ca/data:/home/step/data
command: ["step-ca", "/home/step/config/ca.json"]
4️⃣ Add ACME provisioner (in ca.json)
Inside ca.json, add:
{
"provisioners": [
{
"type": "ACME",
"name": "acme"
}
]
}
This allows Caddy, Certbot, or any ACME client to request certs.
5️⃣ Expose via reverse proxy (Caddy example)
Caddyfile:
ca.example.com {
reverse_proxy step-ca:9000
}
Caddy can get its own certs automatically.
6️⃣ Request a cert externally
Once your CA is live and DNS points to ca.example.com, you can:
step ca certificate "app.example.com" app.crt app.key
Or let a client (like a web server) use ACME to get certs automatically.
🔍 Step 4: Review & Simplify
So the idea is:
- Run step-ca in Docker.
- Initialize it with your domain (e.g., ca.example.com).
- Add an ACME provisioner so clients can use standard protocols.
- Use a reverse proxy (Caddy/Nginx) to expose it.
- Point DNS to it so clients can prove domain ownership and get certs.
That’s all you need to issue public-facing TLS certs from your own CA.
📦 Full Folder Structure
.
├── docker-compose.yml
├── step-ca/
│ ├── config/
│ │ ├── ca.json
│ │ └── ...
│ └── data/
└── Caddyfile
Would you also like me to automate adding the ACME provisioner in Docker Compose startup (no manual editing of ca.json)?
That way, everything is ready in one docker-compose up.
==================================
Perfect! Let’s go deeper with the Feynman Technique and make this fully automated — so you can spin up Smallstep CA with ACME ready to go just by running docker-compose up.
We’ll make it idiot-proof: no manual ca.json editing.
🧠 Step 1: Explain It Simply
Right now, you:
- Start step-ca.
- Manually edit ca.json to add "type": "ACME".
- Restart the container.
We want automation:
- The container runs a script on startup that:
- Checks if CA is initialized.
- If not, runs step ca init.
- Adds ACME provisioner automatically via step ca provisioner add.
- Starts the CA.
This way, you deploy anywhere with no manual intervention.
🛠 Step 2: Key Ingredients
- Dockerfile or init script: Automates initialization.
- Environment variables: Pass CA name, DNS, and ACME provisioner name.
- Entrypoint script: Boots CA if already initialized, or initializes if not.
🧩 Step 3: Build It
🔹 1. Create an init-step-ca.sh script
init-step-ca.sh:
#!/bin/sh
set -e
STEP_HOME=/home/step
CONFIG=$STEP_HOME/config/ca.json
DATA_DIR=$STEP_HOME/data
# Ensure directories exist
mkdir -p $STEP_HOME/config $STEP_HOME/data
if [ ! -f "$CONFIG" ]; then
echo "Initializing Smallstep CA..."
# Initialize CA non-interactively
step ca init \
--name "${CA_NAME:-My Public CA}" \
--dns "${CA_DNS:-ca.example.com}" \
--address ":9000" \
--provisioner "${CA_PROVISIONER:-admin@example.com}" \
--provisioner-password-file <(echo "${CA_PASSWORD:-changeit}") \
--root $STEP_HOME/config/root_ca.crt \
--key $STEP_HOME/config/root_ca_key \
--pki $STEP_HOME/config/pki \
--no-password --provisioner-password-file <(echo "${CA_PASSWORD:-changeit}")
echo "Adding ACME provisioner..."
step ca provisioner add acme --type ACME
else
echo "CA already initialized. Skipping setup."
fi
# Start step-ca
exec step-ca $CONFIG
This script:
- Initializes CA if no ca.json.
- Adds ACME provisioner automatically.
- Starts CA.
🔹 2. Dockerfile to add script
Dockerfile:
FROM smallstep/step-ca:latest
COPY init-step-ca.sh /usr/local/bin/init-step-ca.sh
RUN chmod +x /usr/local/bin/init-step-ca.sh
ENTRYPOINT ["init-step-ca.sh"]
🔹 3. Docker Compose File
docker-compose.yml:
version: "3.8"
services:
step-ca:
build: .
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
ports:
- "9000:9000"
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
🔹 4. One-Command Startup
Now just:
docker-compose up --build -d
✅ If CA isn’t initialized, it will:
- Initialize it.
- Add ACME provisioner.
- Start CA.
✅ If already initialized, it just starts CA.
🔍 Step 4: Simplify & Review
We now have fully automated Smallstep CA deployment:
- No manual edits.
- ACME ready out of the box.
- Environment variables make it easy to change domain, password, etc.
- Works great in CI/CD or infra-as-code setups.
🗂 Folder Structure
.
├── Dockerfile
├── docker-compose.yml
├── init-step-ca.sh
└── step-ca/
├── config/
└── data/
Would you like me to extend this further so Docker Compose also brings up Caddy (reverse proxy) automatically to expose it with HTTPS?
That way, your CA is instantly public-facing with a real cert.
=====================================
Perfect! Let’s extend our mental model using the Feynman Technique so your Docker Compose setup not only runs Smallstep CA but also brings up Caddy to serve it securely with HTTPS — all automated.
We’ll design it so you just run docker-compose up -d, and you have:
✅ A Smallstep CA container (with ACME provisioner auto-added).
✅ A Caddy reverse proxy container (auto-gets its own cert from Let’s Encrypt).
✅ Secure, public-facing CA at https://ca.example.com.
🧠 Step 1: Explain Simply
Think of Smallstep CA as your private Certificate Authority that speaks ACME.
Caddy is like a friendly receptionist that:
- Stands in front of your CA container.
- Automatically gets a public HTTPS cert for itself from Let’s Encrypt.
- Forwards all traffic securely to your CA container.
This means your CA is safely on the internet, and clients can trust its endpoint.
🛠 Step 2: Identify Pieces
We already have:
- Dockerfile → builds Smallstep CA container with init-step-ca.sh.
- docker-compose.yml → runs the CA with environment variables.
We’ll add:
- Caddy service → Runs official caddy image.
- Caddyfile config → Defines reverse proxy rules.
- Volumes → For persistent certs and configs.
🧩 Step 3: Build It
🔹 1. Caddyfile
Create a file called Caddyfile:
# Serve the CA at https://ca.example.com
ca.example.com {
reverse_proxy step-ca:9000
}
Caddy will:
- Listen on port 443.
- Request a trusted cert for ca.example.com from Let’s Encrypt.
- Proxy all traffic to the Smallstep CA container.
🔹 2. Update docker-compose.yml
Here’s the updated docker-compose.yml:
version: "3.8"
services:
step-ca:
build: .
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔹 3. How It Works Together
- step-ca starts → Initializes CA if needed, adds ACME provisioner, runs on port 9000.
- caddy starts → Reads Caddyfile, requests a cert for ca.example.com from Let’s Encrypt, reverse proxies all requests.
- Internet clients → Hit https://ca.example.com, see a publicly trusted HTTPS endpoint.
🔹 4. Folder Structure
.
├── Dockerfile
├── docker-compose.yml
├── init-step-ca.sh
├── Caddyfile
└── step-ca/
├── config/
└── data/
🔍 Step 4: Review & Simplify
Now you have Infrastructure as Code:
- docker-compose up -d spins up a private CA service that is publicly accessible.
- No manual steps: ACME provisioner is ready, Caddy gets its own TLS cert automatically.
- Easy to scale: Just change env vars to spin up a CA for another domain.
🏗 Next Possible Automation
- 🔑 Add DNS challenge support in Caddy for wildcard domains.
- 🔒 Use mutual TLS (mTLS) to secure access to your CA admin endpoints.
- 📜 Auto-generate a bootstrap script for clients (to trust your CA root).
Would you like me to extend this so Caddy also terminates HTTPS for step-ca’s admin interface (not just proxy it)?
That would let you keep step-ca itself bound to HTTP internally, while Caddy handles all encryption.
====================================
Perfect! Let’s go one level deeper with the Feynman Technique — now we’ll make Caddy fully terminate HTTPS for all of Smallstep CA, including its admin interface.
Currently:
- Caddy just proxies traffic to step-ca:9000.
- step-ca itself is handling HTTP/HTTPS internally.
- Admin endpoints are also exposed via Caddy, but encryption happens inside step-ca.
We want:
✅ Caddy to be the only TLS terminator (full HTTPS termination).
✅ step-ca to only serve plain HTTP internally on the Docker network (port 9000).
✅ Admin access secured by Caddy authentication (optional) or firewall rules.
🧠 Step 1: Explain it Simply
Think of Caddy as your security guard:
- Right now, both the security guard and the CA are doing bag checks (TLS termination).
- We only need one guard — Caddy.
- So we’ll let Caddy terminate HTTPS and talk HTTP to step-ca privately.
This simplifies cert management:
- Caddy uses Let’s Encrypt to protect your public CA endpoint.
- Inside Docker, it’s HTTP-only traffic (simpler, faster).
🛠 Step 2: Key Ingredients
- Configure step-ca to listen only on HTTP (--address ":9000").
- Configure Caddy to handle all TLS for ca.example.com.
- Optionally, lock down admin routes with basic auth or IP restrictions in Caddy.
🧩 Step 3: Build It
🔹 1. Adjust init-step-ca.sh
Make sure the --address flag is HTTP-only:
#!/bin/sh
set -e
STEP_HOME=/home/step
CONFIG=$STEP_HOME/config/ca.json
mkdir -p $STEP_HOME/config $STEP_HOME/data
if [ ! -f "$CONFIG" ]; then
echo "Initializing Smallstep CA..."
step ca init \
--name "${CA_NAME:-My Public CA}" \
--dns "${CA_DNS:-ca.example.com}" \
--address ":9000" \
--provisioner "${CA_PROVISIONER:-admin@example.com}" \
--provisioner-password-file <(echo "${CA_PASSWORD:-changeit}") \
--root $STEP_HOME/config/root_ca.crt \
--key $STEP_HOME/config/root_ca_key \
--pki $STEP_HOME/config/pki \
--no-password --provisioner-password-file <(echo "${CA_PASSWORD:-changeit}")
echo "Adding ACME provisioner..."
step ca provisioner add acme --type ACME
else
echo "CA already initialized. Skipping setup."
fi
exec step-ca $CONFIG --address ":9000"
This means step-ca will never listen on HTTPS — only HTTP internally.
🔹 2. Update Caddyfile for Full TLS Termination
We’ll make Caddy:
- Terminate TLS (via Let’s Encrypt).
- Proxy HTTP-only traffic to step-ca.
- Add optional admin security.
Caddyfile:
ca.example.com {
reverse_proxy http://step-ca:9000
# Optional: Secure admin endpoints (basic auth)
@admin {
path /admin/*
}
basic_auth @admin {
admin JDJhJDEwJGtYSEZ... # hashed password
}
}
🔑 Notes:
- reverse_proxy http://step-ca:9000 → Internal comms are HTTP.
- basic_auth block → Protects /admin/* routes.
Generate password hash with: - caddy hash-password --plaintext 'strongpassword'
🔹 3. docker-compose.yml
No change from the previous step except that now step-ca is HTTP-only internally:
version: "3.8"
services:
step-ca:
build: .
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔹 4. Security Flow
- Internet traffic: https://ca.example.com → Caddy (TLS terminator).
- Caddy proxies to: step-ca:9000 (internal HTTP).
- Admin interface: /admin/* secured by Basic Auth (Caddy).
- No TLS duplication → Simpler, fewer certs, better performance.
🔍 Step 4: Simplify & Review
We’ve simplified security:
- All HTTPS handled by Caddy (trusted Let’s Encrypt certs).
- step-ca is never exposed directly, reducing attack surface.
- Admin routes optionally locked down with basic_auth.
- Still fully automated: docker-compose up -d spins up a production-ready CA with a secure admin panel.
🗂 Folder Recap
.
├── Dockerfile
├── docker-compose.yml
├── init-step-ca.sh
├── Caddyfile
└── step-ca/
├── config/
└── data/
Would you like me to add mutual TLS (mTLS) between Caddy and step-ca for extra-hard security (so even internal HTTP traffic is verified)?
That would make it bulletproof even inside Docker.
==============================
Perfect! Let’s turn the security dial to 11 with the Feynman Technique: we’ll add mutual TLS (mTLS) between Caddy and step-ca so even the internal HTTP traffic inside Docker is authenticated and encrypted.
Now even if someone somehow got into your Docker network, they still couldn’t talk to your CA without a trusted client cert.
🧠 Step 1: Explain Simply
Right now:
- Caddy terminates TLS for clients on the internet.
- Inside Docker, Caddy talks to step-ca over plain HTTP.
We want:
- Caddy and step-ca to mutually verify each other using TLS certificates.
- The Docker network becomes a zero-trust zone — nobody can impersonate either service.
So:
- step-ca serves HTTPS internally, requiring a trusted client cert.
- Caddy presents a client certificate to step-ca.
- step-ca verifies Caddy, and Caddy verifies step-ca.
🛠 Step 2: Ingredients
We’ll need:
- A root CA certificate (we already have one from step-ca).
- A client certificate for Caddy, signed by step-ca.
- mTLS configuration in both step-ca and Caddy.
🧩 Step 3: Build It
🔹 1. Generate a Client Certificate for Caddy
Run these once after step-ca is initialized:
# Create keypair + CSR
step ca certificate "caddy" caddy.crt caddy.key
# Copy these to a safe directory:
mkdir -p caddy_certs
mv caddy.crt caddy.key caddy_certs/
We’ll mount these certs into the Caddy container.
🔹 2. Configure step-ca to Require Client Auth
Edit ca.json (step-ca/config/ca.json) and add a tls section:
{
"address": ":9000",
"root": "config/root_ca.crt",
"crt": "config/intermediate_ca.crt",
"key": "config/intermediate_ca_key",
"tls": {
"cipherSuites": ["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"],
"clientAuthType": "RequireAndVerifyClientCert",
"clientCAs": "config/root_ca.crt"
}
}
🔑 This means:
- step-ca will require and verify client certs.
- The client cert (Caddy’s) must chain up to root_ca.crt.
🔹 3. Adjust Caddyfile for mTLS
Update Caddyfile:
ca.example.com {
reverse_proxy https://step-ca:9000 {
transport http {
tls
tls_insecure_skip_verify # optional, or trust explicitly
tls_trusted_ca_certs /certs/root_ca.crt
client_certificate /certs/caddy.crt /certs/caddy.key
}
}
@admin {
path /admin/*
}
basic_auth @admin {
admin JDJhJDEwJGtYSEZ... # hashed password
}
}
🔑 What’s happening:
- transport http { tls ... } → tells Caddy to use HTTPS to step-ca.
- client_certificate → sends Caddy’s signed cert.
- tls_trusted_ca_certs → tells Caddy to trust step-ca’s root CA.
🔹 4. Update Docker Compose
Mount certificates into the Caddy container:
version: "3.8"
services:
step-ca:
build: .
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
- ./caddy_certs:/certs # contains caddy.crt, caddy.key, root_ca.crt
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔹 5. Boot the System
docker-compose up --build -d
✅ step-ca now requires client certs for all incoming connections.
✅ Caddy presents its signed cert to prove identity.
✅ All traffic (even inside Docker) is encrypted + authenticated.
🔍 Step 4: Simplify & Review
We now have end-to-end mutual TLS:
- Caddy → step-ca: mTLS inside Docker.
- Clients → Caddy: HTTPS via Let’s Encrypt.
- No plain HTTP anywhere — every hop is cryptographically verified.
- Attackers inside Docker cannot impersonate Caddy because they’d need a signed client cert.
🗂 Final Folder Layout
.
├── Dockerfile
├── docker-compose.yml
├── init-step-ca.sh
├── Caddyfile
├── caddy_certs/
│ ├── caddy.crt
│ ├── caddy.key
│ └── root_ca.crt
└── step-ca/
├── config/
│ └── ca.json
└── data/
🔥 Key takeaway:
We’ve gone from a simple Smallstep CA in Docker → Automated setup → HTTPS termination with Caddy → Zero-trust, mTLS-protected internal network.
Would you also like me to add automated certificate rotation for the Caddy client certificate (so it renews itself via step-ca)? That way, you’ll never need to manually replace caddy.crt.
========================
Awesome — let’s go for full automation with the Feynman Technique:
We’ll make Caddy’s mTLS client certificate automatically rotate so you never touch caddy.crt manually.
🧠 Step 1: Explain Simply
Right now:
- We manually used step ca certificate to issue a caddy.crt and caddy.key.
- That cert will expire (default 24h–90d depending on policy).
- If we don’t rotate it, Caddy will stop talking to step-ca → CA breaks.
We want:
✅ A fully automated agent inside the Caddy container that:
- Uses the ACME or JWK provisioner to get a short-lived cert.
- Renews before expiry.
- Reloads Caddy automatically.
This makes the system hands-off: certificates rotate daily, and Caddy always has a fresh client cert.
🛠 Step 2: Ingredients
We’ll use:
- step-ca ACME provisioner → already running.
- step CLI → to fetch certs via ACME.
- A cron/loop script → runs in the container to renew before expiry.
- caddy reload → reloads TLS credentials dynamically.
🧩 Step 3: Build It
🔹 1. Create a Cert Renewal Script
Make renew-caddy-cert.sh:
#!/bin/sh
set -e
CERT_PATH="/certs/caddy.crt"
KEY_PATH="/certs/caddy.key"
ROOT_CA="/certs/root_ca.crt"
DOMAIN="${CADDY_CLIENT_IDENTITY:-caddy}"
# Loop forever
while true; do
echo "[INFO] Checking Caddy client cert expiration..."
# If cert doesn't exist OR expires in <24h, renew
if [ ! -f "$CERT_PATH" ] || ! step certificate verify --root "$ROOT_CA" "$CERT_PATH" >/dev/null 2>&1 || \
[ $(step certificate inspect --format json "$CERT_PATH" | jq -r '.validity.end' | xargs -I{} date -d {} +%s) -le $(date -d "+24 hours" +%s) ]; then
echo "[INFO] Renewing Caddy client cert..."
step ca certificate "$DOMAIN" "$CERT_PATH" "$KEY_PATH" \
--provisioner "${CA_PROVISIONER}" \
--root "$ROOT_CA" \
--provisioner-password-file /secrets/provisioner_password
echo "[INFO] Reloading Caddy..."
caddy reload
fi
echo "[INFO] Sleeping for 12h..."
sleep 43200
done
What this does:
- Every 12h, checks if the cert is expiring in <24h.
- If yes, renews it using the provisioner password.
- Reloads Caddy to use the new cert.
🔹 2. Mount Secrets and Root Certs
Store your provisioner password in a file:
echo "changeit" > secrets/provisioner_password
🔹 3. Update Dockerfile to Install Step CLI in Caddy Container
We’ll make a custom Caddy image that has the step CLI and renewal script:
FROM caddy:latest
# Install step CLI
RUN apk add --no-cache bash curl jq && \
curl -fL https://dl.step.sm/gh-release/cli/docs-cli-install/v0.26.0/step-cli_0.26.0_amd64.apk -o step-cli.apk && \
apk add --allow-untrusted step-cli.apk && \
rm -f step-cli.apk
COPY renew-caddy-cert.sh /usr/local/bin/renew-caddy-cert.sh
RUN chmod +x /usr/local/bin/renew-caddy-cert.sh
ENTRYPOINT ["sh", "-c", "renew-caddy-cert.sh & caddy run --config /etc/caddy/Caddyfile --adapter caddyfile"]
This makes the Caddy container self-managing its own mTLS certs.
🔹 4. Update docker-compose.yml
version: "3.8"
services:
step-ca:
build: ./step-ca
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
build: ./caddy
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
environment:
- CADDY_CLIENT_IDENTITY=caddy
- CA_PROVISIONER=admin@example.com
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
- ./caddy_certs:/certs
- ./secrets:/secrets
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔍 Step 4: Simplify & Review
✅ Fully automated mTLS client cert rotation:
- The Caddy container self-manages its client certificate with step ca certificate.
- No manual renewal ever — even short-lived certs (1d) are fine.
- Caddy hot-reloads itself after renewal, no downtime.
- Security is zero-trust and hands-free.
Now your CA deployment is:
- Publicly accessible with Let’s Encrypt certs.
- Internally locked with mTLS, rotated daily.
- No manual intervention required.
🗂 Final Structure
.
├── Caddyfile
├── caddy/
│ ├── Dockerfile
│ └── renew-caddy-cert.sh
├── caddy_certs/
├── secrets/
│ └── provisioner_password
├── docker-compose.yml
├── step-ca/
│ ├── Dockerfile
│ ├── config/
│ │ └── ca.json
│ └── data/
└── init-step-ca.sh
🔥 Key Concept:
We’ve built a self-healing PKI service:
- The CA rotates its own intermediate certs.
- Caddy rotates its client certs.
- Everything runs from docker-compose up.
Would you like me to extend this further so even the CA’s intermediate certificate auto-rotates and updates Caddy dynamically? That would make ALL certs in the chain self-updating with zero manual steps.
========================
awesome — let’s finish the loop: auto-rotate your Intermediate CA and make sure Caddy keeps working without any manual tweaks. i’ll keep it Feynman-style and practical.
🧠 the simple idea
- step-ca signs leaf certs with an intermediate.
- when you rotate (replace) that intermediate, old leaf certs keep working because each leaf PEM bundle shipped to clients already includes the intermediate that signed it. new certs get the new intermediate. you just need to reload step-ca. (smallstep.com)
- because Caddy trusts your root CA (not a specific intermediate), it doesn’t need any change when the intermediate rotates. your mTLS from Caddy→step-ca keeps working as long as:
- Caddy’s client cert keeps renewing (we already automated that), and
- Caddy verifies step-ca using the root trust file.
(Both are already in our setup.)
🔒 production note (be careful with the root key)
Rotating the intermediate requires the root private key to sign the new intermediate. Best practice is to keep that key offline or in an HSM, attach it only during rotation, then detach again. The official guide shows the rotate flow and a HUP reload. (smallstep.com)
🧩 what we’ll add
- a rotation script inside the step-ca container that:
- checks the current intermediate’s expiry,
- if it’s near expiry, creates a fresh intermediate signed by the root,
- atomically swaps files, then sends SIGHUP to step-ca (hot reload). (smallstep.com)
- (optional but safer) mount the root key only during rotation windows, or fetch it from a KMS/HSM.
- ensure Caddy stays happy (it already does, but we’ll also reload if you ever change trust files).
1) rotation script for step-ca
Create rotate-intermediate.sh and bake it into the step-ca image (next section shows Dockerfile):
#!/bin/sh
set -euo pipefail
STEP_HOME=${STEP_HOME:-/home/step}
CFG="$STEP_HOME/config/ca.json"
CERTS="$STEP_HOME/config"
SECRETS="$STEP_HOME/secrets"
ROOT_CRT="$CERTS/root_ca.crt"
ROOT_KEY="$SECRETS/root_ca_key"
INT_CRT="$CERTS/intermediate_ca.crt"
INT_KEY="$SECRETS/intermediate_ca_key"
# How soon before expiry to rotate (e.g., 14 days)
ROTATE_BEFORE="${ROTATE_BEFORE:-336h}" # 14*24h
check_rotate_needed() {
end_epoch=$(step certificate inspect --format json "$INT_CRT" | jq -r '.validity.end' | xargs -I{} date -d {} +%s)
rotate_before_epoch=$(date -d "+$ROTATE_BEFORE" +%s)
[ "$end_epoch" -le "$rotate_before_epoch" ]
}
rotate_intermediate() {
echo "[rotate-int] Backing up current intermediate..."
ts=$(date +%Y%m%d%H%M%S)
cp "$INT_CRT" "$INT_CRT.$ts.bak"
cp "$INT_KEY" "$INT_KEY.$ts.bak"
echo "[rotate-int] Creating NEW intermediate signed by ROOT..."
# Profile shorthand from docs; equivalent template also works
step certificate create \
--profile intermediate-ca \
--ca "$ROOT_CRT" \
--ca-key "$ROOT_KEY" \
"Intermediate CA (rotated $ts)" \
"$INT_CRT.tmp" \
"$INT_KEY.tmp"
echo "[rotate-int] Swapping into place atomically..."
mv -f "$INT_CRT.tmp" "$INT_CRT"
mv -f "$INT_KEY.tmp" "$INT_KEY"
echo "[rotate-int] Signaling step-ca (HUP) to reload..."
# PID 1 is step-ca in our container
kill -HUP 1
echo "[rotate-int] Done."
}
# Loop forever (check every 12h)
while true; do
if check_rotate_needed; then
rotate_intermediate
else
echo "[rotate-int] No rotation needed."
fi
sleep 43200 # 12h
done
Why this works: the official guidance says to create a new intermediate with step certificate create --profile intermediate-ca and send HUP; new leaves will be signed by the new intermediate and clients are fine because they trust the root. (smallstep.com)
2) step-ca Dockerfile (add the rotator)
FROM smallstep/step-ca:latest
# jq for JSON parsing, tini for sane signals (optional)
RUN apk add --no-cache jq tini
COPY init-step-ca.sh /usr/local/bin/init-step-ca.sh
COPY rotate-intermediate.sh /usr/local/bin/rotate-intermediate.sh
RUN chmod +x /usr/local/bin/*.sh
# Use tini to forward signals so HUP reaches step-ca
ENTRYPOINT ["/sbin/tini","--","sh","-c","/usr/local/bin/rotate-intermediate.sh & exec /usr/local/bin/init-step-ca.sh"]
- init-step-ca.sh is your existing initializer (adds ACME, etc.).
- rotate-intermediate.sh runs alongside and only needs the root key available.
Security tip: If you can’t permanently mount the root key, mount it only during planned windows (e.g., docker compose run --rm step-ca rotate-now variant) or fetch from a KMS/HSM for the few seconds needed to sign, then wipe.
The same Smallstep production doc covers intermediate rotation steps and HUP reload. (smallstep.com)
3) docker-compose changes (root key handling)
Add your root cert & key as a mount (or secret) only if you accept the risk:
services:
step-ca:
build: ./step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
- ROTATE_BEFORE=336h # 14 days
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
# ⚠️ optionally mount root key for auto-rotation
- ./root-secrets/root_ca.crt:/home/step/config/root_ca.crt:ro
- ./root-secrets/root_ca_key:/home/step/secrets/root_ca_key:ro
If you won’t keep the root key mounted, remove those two lines and run a one-shot job (or temporarily mount) when you want rotation.
4) do we need to “update Caddy dynamically”?
- Usually no. Caddy’s upstream TLS verification is pinned to /certs/root_ca.crt, not the intermediate. When step-ca rotates the intermediate, Caddy still trusts the chain because the root stays the same. Nothing to change or fetch. (Smallstep’s guide also notes client trust is anchored at the root, and new certs bundle the new intermediate.) (smallstep.com)
- If you did pin Caddy to a specific intermediate (not recommended), switch it to trust the root or teach Caddy to trust a bundle file you rotate. Caddy’s transport can point at a CA bundle path (tls_trusted_ca_certs). You could periodically re-write that file and caddy reload. (Global Caddyfile options and reload behavior are documented.) (Caddy Web Server)
- FYI, step-ca exposes /intermediates.pem and /roots.pem endpoints if you ever want a job to sync trust bundles; but for our mTLS hop we’re already good trusting the root. (smallstep.com)
5) what changes for our earlier pieces?
- Caddy client cert auto-renewal: unchanged (keeps renewing via provisioner). When the intermediate rotates, new client certs will simply chain to the new intermediate; step-ca validates them against the root CA it already trusts. (Short-lived leaves + root-anchored trust is the intended flow in Smallstep.) (smallstep.com)
- Zero-downtime: kill -HUP 1 instructs step-ca to reread config/keys and continue. (smallstep.com)
✅ recap
- We added an intermediate rotation sidecar that:
- checks expiry,
- creates a fresh intermediate with the root key,
- swaps files and HUPs step-ca.
- Caddy requires no changes because it trusts the root, not the intermediate; your mTLS channel keeps flowing.
- Keep the root key safe. If you don’t want it mounted, trigger rotation manually or via a secure KMS/HSM flow.
if you want, i can tweak the script to pull the root key from a KMS/HSM (e.g., HashiCorp Vault transit, AWS KMS w/ signer) so the key never touches disk — and still follow the same rotate+HUP pattern.
===========================
Nice — let’s make this crystal clear and actionable with the Feynman Technique: you want the root private key to stay inside a KMS/HSM and have your rotation process request signatures from that KMS/HSM instead of reading a root key file on disk. I’ll explain the idea simply, then give two concrete, production-ready patterns (HashiCorp Vault and AWS KMS) with command/snippet-level examples and a drop-in change for your rotation sidecar. I’ll also cover PKCS#11/HSM options and operational security notes.
1) The simple idea (as if I’m teaching a beginner)
- You need to create a new intermediate certificate that is signed by your root key whenever the intermediate is about to expire.
- Instead of mounting the root private key on disk inside the container (risky), you keep the root key inside a KMS or HSM.
- Your rotation script creates a CSR (certificate signing request) for the new intermediate and sends that CSR to the KMS/HSM to be signed (KMS returns a DER/PEM signature or a signed certificate).
- The rotation script then installs the signed intermediate into step-ca and sends SIGHUP to reload — no root key ever left the KMS/HSM.
This gives you the security of an offline/managed root key with the automation of atomic intermediate rotation.
(Background: Smallstep supports PKCS#11, Google/AWS KMS and other integrations for signing. We’ll use those integration points rather than trying to export raw secret material.) (Smallstep)
2) Two practical options (pick one that fits your infra)
Option A — HashiCorp Vault (recommended when you already use Vault)
Vault can host/manage keys or act as a signing service (transit or PKI engines). Two patterns:
A1. Vault PKI managed-key — Vault’s PKI engine can act as the signer for an intermediate. Use Vault’s managed-keys feature so Vault holds the signing key and produces the signed intermediate cert. (Good when Vault is your CA manager). (HashiCorp Developer)
A2. Vault Transit — Use Vault transit to sign a CSR (you create CSR locally, send to transit/sign, get back signature or signed cert). Useful when you want step-ca to run but root key stays in Vault.
Example flow (Vault Transit signing)
- Create an intermediate keypair and CSR locally inside the step-ca container (no root key present).
- Send CSR to Vault transit or Vault PKI API to sign (Vault returns signed cert).
- Atomically swap intermediate files and HUP step-ca.
Commands (illustrative):
# 1) create key & CSR in step-ca container (temporary files)
step certificate create --csr --key intermediate_key.pem intermediate.csr \
--profile intermediate-ca "Intermediate CA (rotating)" /dev/null
# OR use openssl:
openssl genpkey -algorithm RSA -out intermediate_key.pem -pkeyopt rsa_keygen_bits:4096
openssl req -new -key intermediate_key.pem -subj "/CN=Intermediate CA (rotating)" -out intermediate.csr
# 2) ask Vault to sign the CSR (example using Vault PKI role or transit)
# Vault PKI:
curl --header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{"csr":"'"$(cat intermediate.csr | base64 -w0)"'","format":"pem_bundle","ttl":"8760h"}' \
https://vault.example:8200/v1/pki/root/sign-intermediate
# Vault transit (custom tool may be needed to assemble cert from signature)
After you receive the signed intermediate PEM, write it to config/intermediate_ca.crt and mv the corresponding key into secrets/intermediate_ca_key, then kill -HUP 1 to reload step-ca. Use Vault ACLs and audit logging for security. (HashiCorp Developer)
Option B — AWS KMS (or other cloud KMS) signing tool (good for AWS shops)
AWS KMS lets you sign data with an asymmetric key. There are community tools (and examples) that turn a CSR into a signed certificate using KMS (you often need a small helper to create the certificate structure because KMS returns only raw signature). Examples and patterns exist (see aws-kms-sign-csr or similar scripts). (moonbase.sh, Amazon Web Services, Inc.)
Example flow (AWS KMS)
- Generate an intermediate keypair & CSR inside the step-ca runtime (private key stays in step-ca's config directory).
- Use a helper (e.g., aws-kms-sign-csr or a small Python script) to have AWS KMS sign the CSR with a KMS asymmetric key.
- Combine the CSR + KMS signature into an X.509 certificate (helper tools do this), produce intermediate_ca.crt.
- Atomically swap and HUP step-ca.
Sketch (pseudo):
# create CSR locally
openssl genpkey -algorithm RSA -out intermediate_key.pem -pkeyopt rsa_keygen_bits:4096
openssl req -new -key intermediate_key.pem -subj "/CN=Intermediate CA (rotated)" -out intermediate.csr
# sign csr with KMS using helper (this is project-specific; see aws-kms-sign-csr)
aws-kms-sign-csr --key-id arn:aws:kms:... --csr intermediate.csr --out signed-intermediate.pem
# swap files
mv signed-intermediate.pem /home/step/config/intermediate_ca.crt
mv intermediate_key.pem /home/step/secrets/intermediate_ca_key
kill -HUP 1
Note: AWS KMS does not directly return a PEM certificate; helper scripts wrap the raw signature into a cert structure. Use KMS asymmetric signing (RSA or ECDSA) and careful policy/Audit controls. (moonbase.sh, Amazon Web Services, Inc.)
Option C — PKCS#11 / Hardware HSM (on-prem or CloudHSM)
If you have an HSM (YubiHSM, SoftHSM for dev, AWS CloudHSM), Smallstep supports PKCS#11 URIs and step-kms-plugin so the signing operation is performed inside the device. You can direct step or step-ca to use --kms 'pkcs11:...' when creating certificates or performing signing. This is ideal for strong, FIPS-backed key protection. (Smallstep)
Example when signing with PKCS#11 via step CLI:
step certificate create --profile intermediate-ca --kms 'pkcs11:module-path=/usr/local/lib/softhsm/libsofthsm2.so;token=smallstep?pin-value=1234' \
"Intermediate CA (rotated)" intermediate_ca.crt
3) Concrete change to your rotate-intermediate.sh (KMS-backed flow)
Below is a drop-in edit of your earlier rotation script that creates CSR locally, asks the KMS (or Vault) to sign it, installs the returned certificate, and HUPs step-ca. This is a scaffold — replace the sign_with_kms() function with your KMS/Vault helper command.
#!/bin/sh
set -euo pipefail
STEP_HOME=${STEP_HOME:-/home/step}
CERTS="$STEP_HOME/config"
SECRETS="$STEP_HOME/secrets"
INT_CRT="$CERTS/intermediate_ca.crt"
INT_KEY="$SECRETS/intermediate_ca_key"
ROTATE_BEFORE="${ROTATE_BEFORE:-336h}" # 14d
check_rotate_needed() {
[ ! -f "$INT_CRT" ] && return 0
end_epoch=$(step certificate inspect --format json "$INT_CRT" | jq -r '.validity.end' | xargs -I{} date -d {} +%s)
rotate_before_epoch=$(date -d "+$ROTATE_BEFORE" +%s)
[ "$end_epoch" -le "$rotate_before_epoch" ]
}
sign_with_kms() {
# Replace this body with Vault/AWS-KMS/PKCS11 signing steps.
# Expected output: write the signed cert to stdout or to $1 (signed cert path)
SIGNED_OUT="$1"
# Example: call a helper that uses Vault transit or aws-kms-sign-csr
/usr/local/bin/sign-csr-with-kms --csr "$2" --out "$SIGNED_OUT"
}
rotate_intermediate() {
ts=$(date +%Y%m%d%H%M%S)
echo "[rotate-int] Generating new keypair and CSR..."
openssl genpkey -algorithm RSA -out "$INT_KEY.tmp" -pkeyopt rsa_keygen_bits:4096
openssl req -new -key "$INT_KEY.tmp" -subj "/CN=Intermediate CA (rotated $ts)" -out "$CERTS/intermediate.csr.tmp"
echo "[rotate-int] Sending CSR to KMS/HSM for signing..."
sign_with_kms "$CERTS/intermediate_ca.crt.tmp" "$CERTS/intermediate.csr.tmp"
echo "[rotate-int] Swapping files atomically..."
mv -f "$CERTS/intermediate_ca.crt.tmp" "$INT_CRT"
mv -f "$INT_KEY.tmp" "$INT_KEY"
rm -f "$CERTS/intermediate.csr.tmp"
echo "[rotate-int] HUP step-ca to reload new intermediate..."
kill -HUP 1
echo "[rotate-int] rotation complete."
}
while true; do
if check_rotate_needed; then
rotate_intermediate
else
echo "[rotate-int] no rotation needed"
fi
sleep 43200
done
Key point: sign_with_kms is where you plug in a Vault API call, an aws-kms-sign-csr invocation, or a PKCS#11 plugin call. The rotation script never requires direct read access to a root private key — only the KMS/HSM does the signing.
4) Practical helper implementations & tips
- Vault PKI: call /v1/pki/root/sign-intermediate or use vault write pki/root/sign-intermediate csr=@intermediate.csr format=pem_bundle ttl=.... Vault returns cert/pem bundle. (HashiCorp Developer)
- Vault Transit / Managed Keys: Use transit to sign the CSR or use the PKI engine with managed-key feature to keep key material in KMS. See Vault docs for managed keys. (HashiCorp Developer)
- AWS KMS: use community helper aws-kms-sign-csr or write a small script calling Sign API and build an X.509 cert structure (libraries exist to assist). Audit KMS usage and restrict Sign permission tightly. (moonbase.sh, Amazon Web Services, Inc.)
- PKCS#11 / HSM: If you own an HSM, use PKCS#11 URIs with Smallstep’s KMS plugin or SoftHSM during dev. Smallstep has examples for CloudHSM and PKCS#11. (Smallstep)
5) Operational & security considerations (must-read)
- Never export root private key unless during an audited, ephemeral rotation window. Prefer KMS/HSM-hosted signing APIs or HSM-backed managed keys.
- Use least privilege: limit who/what can call the KMS Sign or Vault signing endpoint. Use IAM/Vault ACLs and rotate KMS credentials.
- Audit logs: ensure KMS/Vault audit logs are centralised — you want a trace for every signing operation.
- Backups & disaster recovery: if root key is in an HSM cluster, plan HSM DR. If using Vault managed-key, document recovery and unseal processes.
- Test rotation in staging before applying to production — HUP reloads are fine, but verify clients handle chain changes.
- Short-term access: If you ever must mount root key temporarily (e.g., for an emergency rotation), automate mount/unmount and log it.
6) References / further reading
- Smallstep docs — step-ca & cryptographic protections (PKCS#11, Cloud KMS support). (Smallstep)
- Smallstep CLI examples for --kms / PKCS#11. (Smallstep)
- Vault PKI & transit guides (managed-key approach). (HashiCorp Developer)
- AWS KMS signing patterns & community helper tools. (moonbase.sh, Amazon Web Services, Inc.)
TL;DR — what to do next (practical checklist)
- Pick KMS/HSM (Vault, AWS KMS, CloudHSM, PKCS#11 HSM).
- Implement sign_with_kms() in the rotation script to send CSR to KMS/Vault and return PEM.
- Test rotation in staging: CSR → KMS sign → swap intermediate → kill -HUP 1.
- Lock down KMS/Vault permissions and enable audit logging.
- Put monitoring/alerts on rotation failures.
If you want, I’ll:
- produce a ready-to-run sign-with-vault.sh that calls Vault’s PKI endpoint and returns a PEM bundle, or
- produce a sign-with-aws-kms.sh helper using aws-kms-sign-csr and show how to wire it into your rotate-intermediate.sh.
Which helper should I build for you now — Vault or AWS KMS (or PKCS#11/HSM)?
==========================
Perfect, now we’ll take this to the next security tier: your root CA key never even lands in a container’s filesystem. Instead, the root signing operation is done by a secure key service: Vault, AWS KMS, or an HSM via PKCS#11. Let’s break this down step by step using the Feynman Technique:
🧠 Start Simple: What Problem Are We Solving?
- Previously, we mounted the root key into the step-ca container to rotate intermediates.
- That’s a security risk: root key exposure (even read-only) is dangerous.
- Solution: keep the root private key offline or in a secure signing service and let step-ca call a signer API or PKCS#11 module.
- Rotation script stays the same conceptually; only the signing mechanism changes.
🔑 Three Options
Option | Security Level | How it works |
|---|---|---|
Vault PKI Secrets Engine | 🔒🔒🔒 | Vault stores the root CA key. step uses Vault’s API to sign new intermediates. |
AWS KMS (or Azure Key Vault, GCP KMS) | 🔒🔒 | KMS manages the key material; step-ca uses AWS SDK/CLI to sign CSRs. |
PKCS#11 HSM (CloudHSM, YubiHSM, SoftHSM) | 🔒🔒🔒🔒 | step-ca loads a PKCS#11 module; signing happens on hardware. |
🧩 How to Make This Work with Smallstep
Smallstep step-ca supports external key signers (Vault, PKCS#11, or KMS). The high-level flow:
- Root CA private key lives in HSM/KMS/Vault.
- step-ca holds only the public certificate.
- When rotate-intermediate.sh needs a new intermediate:
- It creates a CSR for the new intermediate.
- Calls the HSM/Vault/KMS to sign it with the root.
- Swaps files and sends SIGHUP.
🔐 1. Vault as Root Signer
Vault PKI Engine manages your root CA. You generate the root in Vault, then use Vault’s sign-intermediate endpoint.
Vault Setup:
vault secrets enable pki
vault write pki/root/generate/internal \
common_name="My Root CA" ttl=87600h
vault write pki/config/urls \
issuing_certificates="https://vault.example.com/v1/pki/ca" \
crl_distribution_points="https://vault.example.com/v1/pki/crl"
Rotate Script Snippet:
# Generate CSR
step certificate create "Intermediate CA" \
--profile intermediate-ca \
--csr intermediate_ca.csr intermediate_ca_key
# Ask Vault to sign it
vault write -format=json pki/root/sign-intermediate \
csr=@intermediate_ca.csr \
format=pem_bundle \
ttl=43800h | jq -r .data.certificate > intermediate_ca.crt
Vault returns the signed intermediate. Replace the old cert/key pair and reload step-ca.
This keeps the root key entirely inside Vault.
☁️ 2. AWS KMS Root Key
AWS KMS doesn’t store X.509 cert metadata by default; instead, you create a keypair in KMS and use it to sign the CSR:
- Create asymmetric CMK in AWS KMS:
- aws kms create-key --key-usage SIGN_VERIFY --key-spec RSA_4096
- Export the public key and build a self-signed root cert once:
- aws kms get-public-key --key-id <KEY_ID> > root_pub.pem
- openssl req -new -x509 -keyform engine -engine kms \
- -key <KEY_ID> -out root_ca.crt
- Modify rotate-intermediate.sh to:
- Generate CSR for the new intermediate.
- Call AWS KMS Sign API:
- aws kms sign \
- --key-id <KEY_ID> \
- --signing-algorithm RSASSA_PKCS1_V1_5_SHA256 \
- --message-type RAW \
- --message fileb://intermediate_ca.csr \
- --output text --query Signature > signature.bin
- Rebuild the signed intermediate bundle using the returned signature.
This is trickier because AWS KMS signs raw data, so you’d use step crypto sign or OpenSSL to integrate.
🔒 3. PKCS#11 (HSM)
Smallstep step-ca has first-class PKCS#11 integration.
Instead of a file key, ca.json points to a PKCS#11 key:
{
"authority": {
"provisioners": [...],
"key": {
"type": "pkcs11",
"module": "/usr/local/lib/softhsm/libsofthsm2.so",
"token": "RootCA",
"label": "root-key",
"pin": "1234"
}
}
}
Now all step certificate create calls automatically use the HSM for signing. The rotate-intermediate.sh script is unchanged except you don’t copy the root key at all — signing is delegated to the HSM.
🔗 Docker Compose Pattern
Here’s how this looks in docker-compose.yml for Vault:
services:
step-ca:
image: smallstep/step-ca:latest
environment:
VAULT_ADDR: "https://vault.example.com"
VAULT_TOKEN: "${VAULT_TOKEN}"
volumes:
- ./step-ca/config:/home/step/config
command: >
sh -c "/usr/local/bin/rotate-intermediate.sh & exec step-ca /home/step/config/ca.json"
- The Vault token is injected at runtime.
- No root key files are mounted.
For AWS KMS or HSM:
- Mount ~/.aws or PKCS#11 libraries instead of key files.
📝 Summary
✅ We removed root key exposure.
✅ Rotation script now requests signing services from Vault, KMS, or HSM.
✅ Caddy remains unaffected — still trusts the root.
✅ Security posture jumps to enterprise-grade.
Would you like me to fully rewrite the rotate-intermediate.sh script for Vault (simplest) so you can drop it straight into your Docker Compose?
=================================
Nice — here’s a complete, production-ready rotate-intermediate.sh that talks to HashiCorp Vault PKI to sign a new intermediate certificate (so your root key never leaves Vault). I wrote it Feynman-style: simple to understand, annotated, and ready to drop into the step-ca container as the rotation sidecar. It:
- Creates a fresh intermediate key & CSR inside the container (private key never leaves the container).
- Calls Vault’s PKI sign-intermediate API (authenticated via token or AppRole).
- Receives the signed intermediate PEM bundle and atomically swaps it into place.
- Sends SIGHUP to PID 1 (assumed to be step-ca) so step-ca hot-reloads the new intermediate.
- Runs continuously and rotates ROTATE_BEFORE (default 14 days) before expiry.
- Keeps backups of old keys/certs and logs actions.
Drop this in your step-ca image (/usr/local/bin/rotate-intermediate.sh) and make the image include openssl, jq, and curl.
rotate-intermediate.sh
#!/bin/sh
# rotate-intermediate.sh
# Rotate the intermediate CA by requesting Vault to sign a new intermediate cert.
# Designed to run inside the step-ca container alongside step-ca (PID 1).
#
# Environment variables (with defaults):
# VAULT_ADDR - URL for Vault (required)
# VAULT_AUTH_METHOD - "token" or "approle" (default: token)
# VAULT_TOKEN - Vault token (if VAULT_AUTH_METHOD=token)
# VAULT_ROLE_ID - role_id (if VAULT_AUTH_METHOD=approle)
# VAULT_SECRET_ID - secret_id (if VAULT_AUTH_METHOD=approle)
# VAULT_PKI_PATH - Vault PKI mount path for sign-intermediate (default: pki/root/sign-intermediate)
# STEP_HOME - step home dir (default: /home/step)
# ROTATE_BEFORE_DAYS - rotate this many days before intermediate expiry (default: 14)
# CHECK_INTERVAL_HOURS- check interval in hours (default: 12)
# INTERMEDIATE_TTL - TTL to request for new intermediate from Vault (e.g. 43800h, default: 43800h = 5y)
# DEBUG - if "1", prints more debug info
#
set -euo pipefail
# --- configuration defaults ---
: "${VAULT_ADDR:?VAULT_ADDR must be set (e.g. https://vault.example:8200)}"
: "${VAULT_AUTH_METHOD:=token}"
: "${VAULT_PKI_PATH:=pki/root/sign-intermediate}"
: "${STEP_HOME:=/home/step}"
: "${ROTATE_BEFORE_DAYS:=14}"
: "${CHECK_INTERVAL_HOURS:=12}"
: "${INTERMEDIATE_TTL:=43800h}"
: "${DEBUG:=0}"
CERT_DIR="$STEP_HOME/config"
SECRETS_DIR="$STEP_HOME/secrets" # keep private keys here
INT_CRT="$CERT_DIR/intermediate_ca.crt"
INT_KEY="$SECRETS_DIR/intermediate_ca_key"
CSR_TMP="$CERT_DIR/intermediate.csr.tmp"
CRT_TMP="$CERT_DIR/intermediate_ca.crt.tmp"
KEY_TMP="$SECRETS_DIR/intermediate_ca_key.tmp"
BACKUP_DIR="$CERT_DIR/backups"
VAULT_TOKEN_FILE="${VAULT_TOKEN_FILE:-}" # optional path to token file if used
mkdir -p "$CERT_DIR" "$SECRETS_DIR" "$BACKUP_DIR"
log() { echo "$(date --iso-8601=seconds) [rotate-int] $*"; }
dbg() { [ "$DEBUG" = "1" ] && echo "$(date --iso-8601=seconds) [rotate-int DEBUG] $*"; }
# --- helpers for Vault authentication ---
get_vault_token_via_approle() {
# requires VAULT_ADDR, VAULT_ROLE_ID, VAULT_SECRET_ID
if [ -z "${VAULT_ROLE_ID:-}" ] || [ -z "${VAULT_SECRET_ID:-}" ]; then
log "ERROR: VAULT_ROLE_ID and VAULT_SECRET_ID must be set for approle auth"
return 1
fi
dbg "Requesting Vault token via AppRole..."
resp=$(curl -sSf -X POST "${VAULT_ADDR}/v1/auth/approle/login" \
-d "{\"role_id\":\"${VAULT_ROLE_ID}\",\"secret_id\":\"${VAULT_SECRET_ID}\"}")
token=$(echo "$resp" | jq -r '.auth.client_token')
if [ -z "$token" ] || [ "$token" = "null" ]; then
log "ERROR: failed to fetch Vault token with AppRole"
return 1
fi
echo "$token"
}
get_vault_token() {
# precedence: env VAULT_TOKEN, file VAULT_TOKEN_FILE, approle
if [ -n "${VAULT_TOKEN:-}" ]; then
dbg "Using VAULT_TOKEN from env"
echo "$VAULT_TOKEN"
return 0
fi
if [ -n "$VAULT_TOKEN_FILE" ] && [ -f "$VAULT_TOKEN_FILE" ]; then
dbg "Reading Vault token from file"
cat "$VAULT_TOKEN_FILE"
return 0
fi
if [ "$VAULT_AUTH_METHOD" = "approle" ]; then
get_vault_token_via_approle
return $?
fi
log "ERROR: No Vault token available. Provide VAULT_TOKEN or set VAULT_AUTH_METHOD=approle with VAULT_ROLE_ID/VAULT_SECRET_ID."
return 1
}
# --- check if rotation needed ---
rotate_needed() {
# If no intermediate exists, rotate.
if [ ! -f "$INT_CRT" ]; then
dbg "No intermediate certificate present -> rotate needed"
return 0
fi
# extract end date in seconds since epoch
end_date=$(openssl x509 -in "$INT_CRT" -noout -enddate | sed 's/notAfter=//')
end_epoch=$(date -d "$end_date" +%s)
rotate_before_epoch=$(date -d "+${ROTATE_BEFORE_DAYS} days" +%s)
dbg "Intermediate expires: $end_date (epoch $end_epoch)"
dbg "Rotate threshold epoch: $rotate_before_epoch"
if [ "$end_epoch" -le "$rotate_before_epoch" ]; then
dbg "Rotation is needed (expiry within threshold)"
return 0
fi
dbg "Rotation not needed"
return 1
}
# --- perform rotation via Vault ---
perform_rotation() {
ts=$(date +%Y%m%d%H%M%S)
log "Starting intermediate rotation (ts=$ts)"
# 1) generate new keypair (private key stays in container)
log "Generating new intermediate key..."
openssl genpkey -algorithm RSA -out "$KEY_TMP" -pkeyopt rsa_keygen_bits:4096
chmod 600 "$KEY_TMP"
# 2) create CSR for intermediate
log "Creating CSR..."
# Subject can be customized; keep CN with timestamp so it's identifiable
openssl req -new -key "$KEY_TMP" -subj "/CN=Intermediate CA (rotated $ts)" -out "$CSR_TMP"
# 3) get Vault token
token=$(get_vault_token)
if [ $? -ne 0 ] || [ -z "$token" ]; then
log "ERROR: could not obtain a Vault token"
rm -f "$KEY_TMP" "$CSR_TMP"
return 1
fi
# 4) call Vault PKI sign-intermediate endpoint
log "Requesting Vault sign-intermediate..."
# Vault expects the CSR PEM as 'csr' param; request PEM bundle output
resp=$(curl -sSf -X POST \
--header "X-Vault-Token: ${token}" \
--data-urlencode "csr@${CSR_TMP}" \
--data-urlencode "format=pem_bundle" \
--data-urlencode "ttl=${INTERMEDIATE_TTL}" \
"${VAULT_ADDR}/v1/${VAULT_PKI_PATH}" 2>&1) || {
log "ERROR: Vault sign-intermediate failed: $resp"
rm -f "$KEY_TMP" "$CSR_TMP"
return 1
}
# 5) extract certificate bundle (PEM)
signed_cert=$(echo "$resp" | jq -r '.data.certificate // .data.certificate_bundle // .data.certificate_chain // empty')
# Some Vault setups return .data.certificate (the signed cert), .data.issuing_ca, .data.certificate_bundle etc.
if [ -z "$signed_cert" ] || [ "$signed_cert" = "null" ]; then
# try alternative fields
signed_cert=$(echo "$resp" | jq -r '.data.certificate // empty')
fi
if [ -z "$signed_cert" ]; then
log "ERROR: Could not extract signed certificate from Vault response: $resp"
rm -f "$KEY_TMP" "$CSR_TMP"
return 1
fi
# Write signed cert to temp file
echo "$signed_cert" > "$CRT_TMP"
chmod 644 "$CRT_TMP"
# 6) backup current cert/key and rotate atomically
if [ -f "$INT_CRT" ]; then
cp -a "$INT_CRT" "$BACKUP_DIR/intermediate_ca.crt.$ts.bak"
fi
if [ -f "$INT_KEY" ]; then
cp -a "$INT_KEY" "$BACKUP_DIR/intermediate_ca_key.$ts.bak"
fi
log "Swapping new intermediate into place..."
mv -f "$CRT_TMP" "$INT_CRT"
mv -f "$KEY_TMP" "$INT_KEY"
rm -f "$CSR_TMP"
# 7) signal step-ca to reload (PID 1)
if kill -HUP 1 2>/dev/null; then
log "Sent SIGHUP to PID 1 (step-ca) to reload new intermediate."
else
log "WARNING: failed to signal PID 1. You may need to restart step-ca manually."
fi
log "Rotation complete. New intermediate installed."
return 0
}
# --- main loop ---
log "rotate-intermediate.sh starting. Checking every ${CHECK_INTERVAL_HOURS} hours. Rotate ${ROTATE_BEFORE_DAYS} days before expiry."
while true; do
if rotate_needed; then
if perform_rotation; then
log "Rotation successfully performed."
else
log "Rotation failed. Will retry after interval."
fi
fi
sleep "$(expr ${CHECK_INTERVAL_HOURS} \* 3600)"
done
Dockerfile snippet (step-ca image)
Make sure the step-ca image contains openssl, jq, and curl. Example Dockerfile additions:
FROM smallstep/step-ca:latest
# install required packages
RUN apk add --no-cache openssl jq curl
COPY init-step-ca.sh /usr/local/bin/init-step-ca.sh
COPY rotate-intermediate.sh /usr/local/bin/rotate-intermediate.sh
RUN chmod +x /usr/local/bin/*.sh
# run rotator in background then start init script (which starts step-ca)
ENTRYPOINT ["/sbin/tini", "--", "sh", "-c", "/usr/local/bin/rotate-intermediate.sh & exec /usr/local/bin/init-step-ca.sh"]
(You may already have a similar Dockerfile from previous steps; just ensure these tools are present and the rotate script is copied.)
docker-compose snippet (how to pass Vault auth info safely)
Prefer Docker secrets or an environment injection mechanism that doesn’t leak tokens in logs. Example (simple):
services:
step-ca:
build: ./step-ca
environment:
- VAULT_ADDR=https://vault.example.com:8200
- VAULT_AUTH_METHOD=approle
- VAULT_ROLE_ID=${VAULT_ROLE_ID}
- VAULT_SECRET_ID=${VAULT_SECRET_ID}
- ROTATE_BEFORE_DAYS=14
- CHECK_INTERVAL_HOURS=12
- INTERMEDIATE_TTL=43800h
secrets:
- vault_role_id
- vault_secret_id
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/secrets:/home/step/secrets
secrets:
vault_role_id:
file: ./secrets/vault_role_id
vault_secret_id:
file: ./secrets/vault_secret_id
If you use VAULT_TOKEN instead, prefer mounting it as a Docker secret or reading from a file path (VAULT_TOKEN_FILE) rather than plainly in environment.
Vault PKI config reminder
Make sure your Vault PKI is set up and the path in VAULT_PKI_PATH points to the signing endpoint you expect (commonly pki/root/sign-intermediate for an offline root). Example setup commands in Vault:
vault secrets enable -path=pki pki
vault write pki/root/generate/internal common_name="My Root CA" ttl=87600h
vault write pki/config/urls issuing_certificates="https://vault.example/v1/pki/ca" crl_distribution_points="https://vault.example/v1/pki/crl"
For the sign-intermediate call you may have the API at v1/pki/root/sign-intermediate (depends on how you mounted the PKI engine and which path holds your root).
Testing & validation checklist (Feynman-style steps)
- In staging, set VAULT_ADDR to your Vault dev server and VAULT_AUTH_METHOD=token with a short-lived token that can call sign-intermediate.
- Start the step-ca container; the rotate script will run. If no intermediate exists, it will perform a rotation and install one.
- Check /home/step/config/intermediate_ca.crt and /home/step/secrets/intermediate_ca_key exist and have been swapped.
- Verify step-ca has reloaded (logs should show HUP or CA reloaded).
- Monitor for rotation logs every CHECK_INTERVAL_HOURS. Force an expiry test by setting ROTATE_BEFORE_DAYS large or use a short TTL in Vault for testing.
Security notes (must-read)
- The root private key remains inside Vault. The container only sends a CSR; Vault signs and returns the certificate. This is strong security.
- Use Vault ACLs so the signing endpoint can only be used by the rotation job (AppRole with constrained policies).
- Ensure Vault audit logging is enabled for all sign operations.
- Consider mutual TLS between step-ca and Vault if Vault is on the network (or use a private Vault network).
- Monitor and alert on rotation failures.
If you want, I can:
- produce a ready-to-run sign-with-vault helper (a small binary/script) that handles edge cases and prints better diagnostics, or
- convert this script to use Vault CLI instead of curl (if you prefer the vault binary),
- or adapt this to AppRole bootstrap steps (create role/policies example).
Which of those would you like next?
===========================
Great, let’s fully rewrite rotate-intermediate.sh so it’s OpenBao-native (no Vault references, just OPENBAO_*). I’ll keep it production-ready with safe defaults, explicit error handling, and comments for clarity.
🔑 rotate-intermediate.sh (OpenBao version)
#!/usr/bin/env bash
#
# rotate-intermediate.sh
# Rotate the step-ca intermediate CA certificate using OpenBao PKI.
# Requires: curl, jq, openssl
#
# Environment variables to configure:
# OPENBAO_ADDR - URL of OpenBao (e.g. https://openbao.example:8200)
# OPENBAO_PKI_PATH - Path of PKI backend (default: pki/root/sign-intermediate)
# OPENBAO_TOKEN - Token for auth (or leave blank if using AppRole)
# OPENBAO_ROLE_ID - AppRole Role ID (optional, for AppRole auth)
# OPENBAO_SECRET_ID - AppRole Secret ID (optional, for AppRole auth)
# INTERMEDIATE_TTL - Lifetime of intermediate cert (default: 43800h ~5y)
# INT_KEY - Path to intermediate private key
# INT_CRT - Path to intermediate certificate
#
set -euo pipefail
# Defaults
: "${OPENBAO_PKI_PATH:=pki/root/sign-intermediate}"
: "${INTERMEDIATE_TTL:=43800h}"
: "${INT_KEY:=/home/step/secrets/intermediate_ca_key}"
: "${INT_CRT:=/home/step/secrets/intermediate_ca.crt}"
KEY_TMP=$(mktemp)
CSR_TMP=$(mktemp)
CRT_TMP=$(mktemp)
log() { echo "[$(date --iso-8601=seconds)] $*"; }
# Get a token either from env or via AppRole
get_openbao_token() {
if [[ -n "${OPENBAO_TOKEN:-}" ]]; then
echo "$OPENBAO_TOKEN"
return 0
fi
if [[ -z "${OPENBAO_ROLE_ID:-}" || -z "${OPENBAO_SECRET_ID:-}" ]]; then
log "ERROR: No OPENBAO_TOKEN or AppRole credentials set"
exit 1
fi
log "Fetching token from OpenBao via AppRole..."
local resp
resp=$(curl -sS -X POST "${OPENBAO_ADDR}/v1/auth/approle/login" \
-d "{\"role_id\":\"${OPENBAO_ROLE_ID}\",\"secret_id\":\"${OPENBAO_SECRET_ID}\"}")
echo "$resp" | jq -r '.auth.client_token'
}
# Request a signed intermediate cert from OpenBao
sign_intermediate() {
local csr_file="$1"
local token="$2"
curl -sS -X POST \
--header "X-Vault-Token: ${token}" \
--data-urlencode "csr@${csr_file}" \
--data-urlencode "format=pem_bundle" \
--data-urlencode "ttl=${INTERMEDIATE_TTL}" \
"${OPENBAO_ADDR}/v1/${OPENBAO_PKI_PATH}"
}
# Perform the rotation
rotate_intermediate() {
log "Generating new intermediate CA key and CSR..."
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out "$KEY_TMP"
openssl req -new -key "$KEY_TMP" \
-subj "/CN=Intermediate CA $(date +%Y-%m-%d)" \
-out "$CSR_TMP"
log "Authenticating to OpenBao..."
local token
token=$(get_openbao_token)
if [[ -z "$token" ]]; then
log "ERROR: Failed to get token"
exit 1
fi
log "Requesting signed intermediate certificate..."
local resp
resp=$(sign_intermediate "$CSR_TMP" "$token") || {
log "ERROR: Failed to sign intermediate"
exit 1
}
local cert
cert=$(echo "$resp" | jq -r '.data.certificate // .data.certificate_bundle // empty')
if [[ -z "$cert" ]]; then
log "ERROR: No certificate returned: $resp"
exit 1
fi
echo "$cert" > "$CRT_TMP"
log "Swapping in new key + certificate..."
mv -f "$KEY_TMP" "$INT_KEY"
mv -f "$CRT_TMP" "$INT_CRT"
rm -f "$CSR_TMP"
log "Reloading step-ca (PID 1)..."
kill -HUP 1
log "Rotation completed successfully."
}
rotate_intermediate
🔍 How this works
- Generates a fresh RSA key + CSR.
- Fetches token:
- Uses $OPENBAO_TOKEN if supplied.
- Falls back to AppRole if $OPENBAO_ROLE_ID + $OPENBAO_SECRET_ID are set.
- Posts CSR to OpenBao PKI’s sign-intermediate endpoint.
- Swaps the intermediate key + cert atomically.
- Sends SIGHUP to step-ca (assuming PID 1 in container) to reload dynamically.
🔐 Best practice reminders
- Store secrets in Docker/K8s secrets, not env vars in Compose.
- Rotate AppRole Secret IDs often.
- Audit log all sign operations in OpenBao.
- Use cap_add: SYS_PTRACE or kill permissions in container if needed for reload.
Would you like me to add this into your docker-compose.yaml (with correct volume mounts, secrets, and OpenBao environment) so it’s completely automated?
==============================
Perfect — let’s make this completely automated. We’ll bake the new rotate-intermediate.sh script into your Docker setup, wire in OpenBao secrets, and have Docker Compose handle everything (CA, Caddy, cert rotation).
We’ll follow Feynman’s steps: explain in simple terms first, then go into detail.
🧠 Step 1: Explain It Like You’re Five
- You have a CA (Smallstep) inside Docker.
- You now have a script (rotate-intermediate.sh) that talks to OpenBao to get a fresh intermediate cert.
- Instead of running this script manually, Docker Compose will:
- Mount the script and certificates.
- Have a lightweight rotator container that runs it on a schedule.
- Store secrets like OPENBAO_ROLE_ID and OPENBAO_SECRET_ID securely in Docker secrets.
- Automatically reload step-ca when it gets a new cert.
This way, your PKI is hands-free: certs always rotate safely via OpenBao, and Caddy sees updates automatically.
🛠 Step 2: Directory Layout
Here’s the updated folder structure:
.
├── docker-compose.yml
├── rotate-intermediate.sh
├── step-ca/
│ ├── config/ca.json
│ └── data/
├── caddy_certs/
│ ├── root_ca.crt
│ ├── caddy.crt
│ └── caddy.key
└── secrets/
├── openbao-role-id
└── openbao-secret-id
🧩 Step 3: Updated docker-compose.yml
version: "3.9"
secrets:
openbao-role-id:
file: ./secrets/openbao-role-id
openbao-secret-id:
file: ./secrets/openbao-secret-id
services:
step-ca:
image: smallstep/step-ca:latest
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
- ./caddy_certs:/certs
networks:
- stepnet
ca-rotator:
image: alpine:latest
container_name: ca-rotator
restart: unless-stopped
depends_on:
- step-ca
volumes:
- ./rotate-intermediate.sh:/usr/local/bin/rotate-intermediate.sh
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
environment:
OPENBAO_ADDR: https://openbao.example.com:8200
OPENBAO_PKI_PATH: pki/root/sign-intermediate
INTERMEDIATE_TTL: 43800h
secrets:
- source: openbao-role-id
target: OPENBAO_ROLE_ID
- source: openbao-secret-id
target: OPENBAO_SECRET_ID
entrypoint: ["/bin/sh", "-c"]
command: |
apk add --no-cache bash curl jq openssl &&
chmod +x /usr/local/bin/rotate-intermediate.sh &&
while true; do
/usr/local/bin/rotate-intermediate.sh || echo "Rotation failed"
sleep 86400 # rotate daily (can tune to 30 days, etc.)
done
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔍 Step 4: How It Works
- Secrets Management:
- openbao-role-id and openbao-secret-id are Docker secrets mounted inside ca-rotator for authentication.
- This is safer than hardcoding tokens.
- ca-rotator Service:
- Lightweight Alpine container that installs bash, curl, jq, openssl.
- Runs rotate-intermediate.sh in a loop (once a day or custom interval).
- Requests a fresh intermediate cert from OpenBao, swaps it in, reloads step-ca.
- Caddy Reload:
- Caddy doesn’t need changes — it watches cert files and reloads automatically.
- Security:
- OpenBao remains your single source of trust.
- Intermediate rotation is now automated.
- Role/Secret IDs are isolated from the CA container (defense-in-depth).
🔐 Bonus Hardening Ideas
- Use PKCS#11 with OpenBao to store root keys in an HSM/YubiHSM.
- Run ca-rotator as a Kubernetes CronJob if moving to k8s.
- Reduce rotation interval to every 30–60 days for maximum crypto hygiene.
- Make step-ca read-only (no private root key) so compromise has limited blast radius.
TL;DR
This Compose stack now:
✅ Runs step-ca with no root key on disk (root in OpenBao).
✅ Rotates intermediates automatically, securely.
✅ Keeps secrets safe using Docker secrets.
✅ Gives you a hands-free PKI with OpenBao as the trust anchor.
Would you also like me to add a healthcheck + Slack/Email alert if rotation fails (for production monitoring)?
=========================
Perfect, let’s extend the automation to add healthchecks + failure alerts using Slack/Email, while keeping the “explain like I’m five” clarity.
We’ll use Feynman’s Technique again: start simple, then go deep.
🧠 Step 1: ELI5 (Explain Like You’re Five)
- We already have a robot (the ca-rotator container) that rotates certificates daily.
- We now give that robot a health check: “Did you rotate okay?”
- If the robot says no, we’ll send a message to Slack or an email saying “Cert rotation failed!”
- This makes your PKI setup self-monitoring.
🛠 Step 2: Implementation Plan
We’ll add:
✅ Docker healthcheck: marks container as unhealthy if rotation fails.
✅ Slack webhook support: send a message when unhealthy.
✅ Optional Email alert: using msmtp (lightweight SMTP client).
🧩 Step 3: Updated docker-compose.yml Snippet for ca-rotator
ca-rotator:
image: alpine:latest
container_name: ca-rotator
restart: unless-stopped
depends_on:
- step-ca
volumes:
- ./rotate-intermediate.sh:/usr/local/bin/rotate-intermediate.sh
- ./alert.sh:/usr/local/bin/alert.sh
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
environment:
OPENBAO_ADDR: https://openbao.example.com:8200
OPENBAO_PKI_PATH: pki/root/sign-intermediate
INTERMEDIATE_TTL: 43800h
SLACK_WEBHOOK_URL: https://hooks.slack.com/services/XXX/YYY/ZZZ
ALERT_EMAIL: ops@example.com
secrets:
- source: openbao-role-id
target: OPENBAO_ROLE_ID
- source: openbao-secret-id
target: OPENBAO_SECRET_ID
entrypoint: ["/bin/sh", "-c"]
command: |
apk add --no-cache bash curl jq openssl msmtp &&
chmod +x /usr/local/bin/rotate-intermediate.sh /usr/local/bin/alert.sh &&
while true; do
if /usr/local/bin/rotate-intermediate.sh; then
echo "Rotation successful"
else
echo "Rotation failed"
/usr/local/bin/alert.sh "Rotation failed on $(date)"
fi
sleep 86400
done
healthcheck:
test: ["CMD-SHELL", "test -f /tmp/last_rotation_success && find /tmp/last_rotation_success -mmin -1440 | grep ."]
interval: 1h
timeout: 30s
retries: 3
networks:
- stepnet
🔔 Step 4: alert.sh Script
#!/usr/bin/env bash
# alert.sh
# Sends alert to Slack and optional email
MESSAGE=$1
send_slack() {
if [[ -n "${SLACK_WEBHOOK_URL:-}" ]]; then
curl -s -X POST -H 'Content-type: application/json' \
--data "{\"text\":\":rotating_light: *CA Rotation Alert*: ${MESSAGE}\"}" \
"${SLACK_WEBHOOK_URL}"
fi
}
send_email() {
if [[ -n "${ALERT_EMAIL:-}" ]]; then
echo -e "Subject: CA Rotation Alert\n\n${MESSAGE}" | msmtp "${ALERT_EMAIL}"
fi
}
send_slack
send_email
🔍 How It Works
- rotate-intermediate.sh writes a marker file /tmp/last_rotation_success if it succeeds.
- The healthcheck checks if that file is updated within the last 24 hours.
- If healthcheck fails, Docker marks ca-rotator as unhealthy.
- alert.sh sends a Slack message (and/or email) on rotation failure.
🔐 Why This Is Production-Grade
- Healthcheck integrates with Docker monitoring (Prometheus, Grafana, Portainer).
- Alerts give early warning if rotation fails due to OpenBao outage, network issues, or misconfiguration.
- Slack webhook keeps it lightweight (no extra service).
- Email is optional for legacy ops teams.
🏗 Suggested Workflow
- Rotation script succeeds → updates timestamp file.
- Healthcheck passes → container stays healthy.
- Rotation fails → alert fires, container goes unhealthy → ops investigates.
🚀 TL;DR
We now have:
✅ Automated cert rotation (from OpenBao).
✅ A healthcheck to detect failures.
✅ Slack/Email alerts if rotation breaks.
✅ Full Docker-native automation.
Would you like me to replace Slack with PagerDuty/Prometheus integration for enterprise-grade alerting?
=========================
Perfect — let’s make this enterprise-grade: replacing Slack with PagerDuty + Prometheus alerting while keeping the Feynman clarity.
We’ll follow the same 4-step structure:
🧠 Step 1: ELI5 (Explain Like You’re Five)
- Before, your robot (the CA rotator) said “help!” in Slack.
- Now, instead of Slack, we connect it to PagerDuty (to wake up on-call engineers) and Prometheus (to graph and alert on health over time).
- Prometheus scrapes health metrics from Docker.
- Alertmanager watches these metrics and triggers PagerDuty incidents if something fails.
This means your cert rotation pipeline is now enterprise-ready with 24/7 alerting.
🛠 Step 2: How It Fits Together
- Prometheus scrapes cadvisor or Docker Engine API to track container health.
- ca-rotator healthcheck status is exposed as container_last_seen and container_state.
- Alertmanager is configured with a PagerDuty integration key.
- If the rotation job is failing or container is unhealthy, Alertmanager opens a PagerDuty incident.
🧩 Step 3: Updated docker-compose.yml (Adding Prometheus + Alertmanager)
version: "3.9"
secrets:
openbao-role-id:
file: ./secrets/openbao-role-id
openbao-secret-id:
file: ./secrets/openbao-secret-id
pagerduty-key:
file: ./secrets/pagerduty-key
services:
step-ca:
image: smallstep/step-ca:latest
container_name: step-ca
restart: unless-stopped
networks: [stepnet]
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
ports:
- "80:80"
- "443:443"
networks: [stepnet]
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
- ./caddy_certs:/certs
ca-rotator:
image: alpine:latest
container_name: ca-rotator
restart: unless-stopped
depends_on: [step-ca]
networks: [stepnet]
secrets:
- source: openbao-role-id
target: OPENBAO_ROLE_ID
- source: openbao-secret-id
target: OPENBAO_SECRET_ID
environment:
OPENBAO_ADDR: https://openbao.example.com:8200
OPENBAO_PKI_PATH: pki/root/sign-intermediate
INTERMEDIATE_TTL: 43800h
volumes:
- ./rotate-intermediate.sh:/usr/local/bin/rotate-intermediate.sh
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
entrypoint: ["/bin/sh", "-c"]
command: |
apk add --no-cache bash curl jq openssl &&
chmod +x /usr/local/bin/rotate-intermediate.sh &&
while true; do
if /usr/local/bin/rotate-intermediate.sh; then
date > /tmp/last_rotation_success
else
echo "Rotation failed"
fi
sleep 86400
done
healthcheck:
test: ["CMD-SHELL", "test -f /tmp/last_rotation_success && find /tmp/last_rotation_success -mmin -1440 | grep ."]
interval: 1h
timeout: 30s
retries: 3
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
networks: [stepnet]
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
networks: [stepnet]
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
restart: unless-stopped
ports:
- "9093:9093"
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
secrets:
- pagerduty-key
networks: [stepnet]
networks:
stepnet:
driver: bridge
📜 prometheus.yml (Scrape cAdvisor + Alertmanager)
global:
scrape_interval: 30s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
📜 alertmanager.yml (PagerDuty Integration)
global:
resolve_timeout: 5m
route:
receiver: pagerduty
receivers:
- name: pagerduty
pagerduty_configs:
- routing_key_file: /run/secrets/pagerduty-key
severity: critical
🔍 How It Works
- cAdvisor exposes container metrics (health, restart count, etc.).
- Prometheus scrapes cAdvisor.
- Alertmanager fires if:
- ca-rotator is unhealthy for >1h
- Rotation success timestamp is older than 24h.
- PagerDuty triggers an incident using the integration key stored in a Docker secret.
🏗 Example Prometheus Alert Rule
groups:
- name: rotation_alerts
rules:
- alert: CARotationStale
expr: time() - container_last_seen{name="ca-rotator"} > 86400
for: 1h
labels:
severity: critical
annotations:
summary: "CA Rotation job hasn't succeeded in 24h"
description: "Check the ca-rotator container logs."
🔐 Why This Is Better Than Slack
Feature | Slack Webhook | PagerDuty + Prometheus |
|---|---|---|
Alerts Ops Team | Manual setup | 24/7 on-call rotation |
Escalation Policies | No | Yes |
History & Analytics | Minimal | Full |
Infra Metrics | No | Yes |
🚀 TL;DR
Now you get:
✅ Automated CA rotation with OpenBao.
✅ Healthcheck monitoring.
✅ Prometheus metrics for dashboards.
✅ PagerDuty integration for serious incident escalation.
Would you like me to bundle Prometheus + Alertmanager config into a single Helm chart for easier deployment (Docker or K8s)?
=========================
Perfect, let’s level up your monitoring stack: now the ca-rotator itself will export Prometheus metrics rather than relying on timestamp hacks.
We’ll rewrite it Feynman-style: start simple, go deep, then show working configs.
🧠 Step 1: ELI5
- Before, Prometheus guessed rotation health by “checking timestamps.”
- Now, the ca-rotator container will tell Prometheus directly:
- ca_rotation_success 1 (last run OK)
- ca_rotation_timestamp 1693472000 (UNIX time of success)
- Prometheus just scrapes metrics from http://ca-rotator:9100/metrics.
- This is cleaner, more reliable, and production-grade.
🛠 Step 2: Design
We’ll:
✅ Add a ca-rotator container to your docker-compose.yml.
✅ Use a simple Python exporter (Flask + Prometheus client) to expose metrics.
✅ Run rotation daily, updating metrics dynamically.
✅ Alert via Prometheus + Alertmanager if rotation fails.
📦 Step 3: Folder Layout
prometheus-stack/
├─ docker-compose.yml
├─ prometheus.yml
├─ alertmanager.yml
├─ rules/
│ └─ rotation_alerts.yml
└─ ca-rotator/
├─ Dockerfile
├─ rotate_intermediate.sh
└─ exporter.py
🐍 Step 4: exporter.py (Prometheus Metrics Exporter)
from flask import Flask, Response
from prometheus_client import CollectorRegistry, Gauge, generate_latest
import time, os, threading, subprocess
app = Flask(__name__)
registry = CollectorRegistry()
rotation_success = Gauge('ca_rotation_success', 'Whether last rotation was successful', registry=registry)
rotation_timestamp = Gauge('ca_rotation_timestamp', 'Timestamp of last successful rotation', registry=registry)
ROTATE_SCRIPT = "/usr/local/bin/rotate_intermediate.sh"
ROTATE_INTERVAL = int(os.getenv("ROTATE_INTERVAL", 86400)) # default: 24h
def rotate_certificates():
while True:
try:
subprocess.check_call([ROTATE_SCRIPT])
rotation_success.set(1)
rotation_timestamp.set(time.time())
print("[OK] Rotation successful")
except subprocess.CalledProcessError:
rotation_success.set(0)
print("[FAIL] Rotation failed")
time.sleep(ROTATE_INTERVAL)
@app.route("/metrics")
def metrics():
return Response(generate_latest(registry), mimetype="text/plain")
if __name__ == "__main__":
threading.Thread(target=rotate_certificates, daemon=True).start()
app.run(host="0.0.0.0", port=9100)
🐳 Step 5: Dockerfile for ca-rotator
FROM python:3.11-slim
RUN apt-get update && apt-get install -y bash curl jq openssl && \
pip install flask prometheus_client && \
rm -rf /var/lib/apt/lists/*
COPY rotate_intermediate.sh /usr/local/bin/rotate_intermediate.sh
COPY exporter.py /exporter.py
RUN chmod +x /usr/local/bin/rotate_intermediate.sh
CMD ["python", "/exporter.py"]
🔧 Step 6: Add ca-rotator to docker-compose.yml
version: "3.9"
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./rules:/etc/prometheus/rules
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.enable-lifecycle"
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
restart: unless-stopped
ports:
- "9093:9093"
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ca-rotator:
build: ./ca-rotator
container_name: ca-rotator
restart: unless-stopped
environment:
OPENBAO_ADDR: https://openbao.example.com:8200
OPENBAO_PKI_PATH: pki/root/sign-intermediate
INTERMEDIATE_TTL: 43800h
volumes:
- ./ca-rotator/rotate_intermediate.sh:/usr/local/bin/rotate_intermediate.sh
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
ports:
- "9100:9100"
volumes:
prometheus_data:
📜 Step 7: Update prometheus.yml to Scrape ca-rotator
global:
scrape_interval: 30s
rule_files:
- "/etc/prometheus/rules/*.yml"
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
- job_name: 'alertmanager'
static_configs:
- targets: ['alertmanager:9093']
- job_name: 'ca-rotator'
static_configs:
- targets: ['ca-rotator:9100']
📜 Step 8: Improve Alert Rule (rotation_alerts.yml)
groups:
- name: rotation_alerts
rules:
- alert: CARotationFailed
expr: ca_rotation_success == 0
for: 30m
labels:
severity: critical
annotations:
summary: "CA Rotation job failed"
description: "The CA rotation script has failed in the last run. Investigate immediately."
- alert: CARotationStale
expr: time() - ca_rotation_timestamp > 86400
for: 1h
labels:
severity: warning
annotations:
summary: "CA Rotation stale"
description: "No successful rotation in over 24 hours."
🚀 How It Works Now
- ca-rotator container runs rotation automatically every 24 hours.
- It exports Prometheus metrics about success/failure and timestamp.
- Prometheus scrapes those metrics directly.
- Alertmanager triggers PagerDuty if failure or stale metrics.
- Fully Docker-native, no timestamp hacks.
🔥 Why This Is Better
✅ Direct metrics = no brittle log scraping.
✅ Metrics easily integrated with Grafana dashboards.
✅ Alerts are fine-grained: failure vs stale rotation.
✅ The rotation logic and monitoring logic are fully encapsulated in one container.
Would you like me to add Grafana with a prebuilt dashboard for CA rotation health next? That way you’d see green/red rotation status visually.
======================
Perfect! Let’s extend your Docker stack with Grafana and ship a ready-made dashboard for CA rotation health. I’ll keep using the Feynman Technique: start simple, then dive deep with a working example.
🧠 Step 1: Explain Like I’m 5
- We already have Prometheus collecting metrics like ca_rotation_success and ca_rotation_timestamp.
- Grafana is a “TV screen for numbers” – it makes those metrics visual.
- We’ll add Grafana in Docker, pre-configure a dashboard JSON that shows:
- A green/red status panel for rotation success.
- A gauge or graph showing time since last rotation.
- Alerts synced with Prometheus/Alertmanager.
🛠 Step 2: Design
- Add Grafana service to docker-compose.yml.
- Store dashboards and datasources in ./grafana/provisioning/.
- Preconfigure Prometheus as a data source (no clicking needed).
- Ship a dashboard JSON for CA rotation health.
📂 Step 3: Directory Structure
prometheus-stack/
├─ docker-compose.yml
├─ prometheus.yml
├─ alertmanager.yml
├─ rules/
│ └─ rotation_alerts.yml
├─ grafana/
│ ├─ provisioning/
│ │ ├─ dashboards/
│ │ │ └─ ca-rotation.json
│ │ └─ datasources/
│ │ └─ prometheus.yml
└─ ca-rotator/
├─ Dockerfile
├─ rotate_intermediate.sh
└─ exporter.py
🐳 Step 4: Extend docker-compose.yml
version: "3.9"
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./rules:/etc/prometheus/rules
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.enable-lifecycle"
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
restart: unless-stopped
ports:
- "9093:9093"
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ca-rotator:
build: ./ca-rotator
container_name: ca-rotator
restart: unless-stopped
environment:
OPENBAO_ADDR: https://openbao.example.com:8200
OPENBAO_PKI_PATH: pki/root/sign-intermediate
INTERMEDIATE_TTL: 43800h
volumes:
- ./ca-rotator/rotate_intermediate.sh:/usr/local/bin/rotate_intermediate.sh
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
ports:
- "9100:9100"
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- grafana_data:/var/lib/grafana
volumes:
prometheus_data:
grafana_data:
📜 Step 5: Grafana Datasource (grafana/provisioning/datasources/prometheus.yml)
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
📊 Step 6: Grafana Dashboard JSON (grafana/provisioning/dashboards/ca-rotation.json)
{
"id": null,
"title": "CA Rotation Health",
"tags": ["ca", "rotation", "security"],
"timezone": "browser",
"schemaVersion": 27,
"version": 1,
"refresh": "10s",
"panels": [
{
"type": "stat",
"title": "Rotation Success",
"datasource": "Prometheus",
"targets": [
{
"expr": "ca_rotation_success",
"legendFormat": "Success"
}
],
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": 0 },
{ "color": "green", "value": 1 }
]
}
}
}
},
{
"type": "gauge",
"title": "Time Since Last Rotation (seconds)",
"datasource": "Prometheus",
"targets": [
{
"expr": "time() - ca_rotation_timestamp"
}
],
"fieldConfig": {
"defaults": {
"min": 0,
"max": 86400
}
}
}
]
}
📜 Step 7: Grafana Dashboard Provisioning File
Create grafana/provisioning/dashboards/dashboards.yml:
apiVersion: 1
providers:
- name: "Default"
orgId: 1
folder: ""
type: file
disableDeletion: false
updateIntervalSeconds: 30
options:
path: /etc/grafana/provisioning/dashboards
🚀 Step 8: Run It
docker-compose up -d
- Grafana: http://localhost:3000 (admin/admin)
- Prometheus: http://localhost:9090
- Alertmanager: http://localhost:9093
🔥 What You Get
✅ Fully automated Grafana setup, no manual clicking.
✅ Dashboard shows rotation health at a glance.
✅ Prometheus and Grafana run side by side in Docker.
✅ Easy to add Grafana alerts (or keep them in Prometheus).
Would you like me to pre-configure Grafana alerts too (so Grafana itself sends PagerDuty/Slack alerts alongside Alertmanager)? Or keep alerting centralized in Prometheus/Alertmanager?
=====================
Perfect — let’s teach Grafana to alert you out-of-the-box. We’ll keep using the Feynman Technique:
🧠 Step 1: Explain Like I’m 5
Grafana already shows your CA rotation health on dashboards.
Now we’ll make Grafana yell (via PagerDuty, Slack, or email) when something breaks:
- Grafana has Alerting rules that watch metrics.
- We define them in YAML/JSON so they’re ready at container startup.
- Alerts trigger contact points (PagerDuty, Slack, Email).
- This runs alongside Prometheus/Alertmanager — some teams like Grafana’s alerting because it’s visual and flexible.
🛠 Step 2: Design
- Add grafana/provisioning/alerting/ directory.
- Add alert rules to fire if:
- ca_rotation_success == 0
- time() - ca_rotation_timestamp > X hours
- Add contact points for PagerDuty (replace with your API key).
📂 Step 3: File Layout
grafana/
├─ provisioning/
│ ├─ datasources/prometheus.yml
│ ├─ dashboards/
│ │ ├─ dashboards.yml
│ │ └─ ca-rotation.json
│ └─ alerting/
│ ├─ contactpoints.yaml
│ └─ alert-rules.yaml
📜 Step 4: Contact Points (grafana/provisioning/alerting/contactpoints.yaml)
apiVersion: 1
contactPoints:
- orgId: 1
name: pagerduty
receivers:
- uid: pagerduty
type: pagerduty
settings:
integrationKey: "${PAGERDUTY_ROUTING_KEY}" # Set as env var
You can also add Slack:
- orgId: 1
name: slack
receivers:
- uid: slack
type: slack
settings:
url: "${SLACK_WEBHOOK_URL}"
📜 Step 5: Alert Rules (grafana/provisioning/alerting/alert-rules.yaml)
apiVersion: 1
groups:
- orgId: 1
name: CA Rotation Alerts
folder: Security
interval: 1m
rules:
- uid: rotation_failed
title: "CA Rotation Failed"
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
expr: ca_rotation_success
format: time_series
interval: ""
intervalFactor: 2
legendFormat: ""
refId: A
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
expr: ca_rotation_success == 0
format: time_series
interval: ""
intervalFactor: 2
refId: B
noDataState: Alerting
execErrState: Alerting
for: 2m
annotations:
description: "CA rotation failed. Investigate immediately."
labels:
severity: critical
notificationSettings:
contactPoint: pagerduty
groupBy: ["alertname"]
- uid: rotation_stale
title: "CA Rotation Stale (>24h)"
condition: B
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
expr: time() - ca_rotation_timestamp
format: time_series
refId: A
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
expr: (time() - ca_rotation_timestamp) > 86400
format: time_series
refId: B
noDataState: Alerting
execErrState: Alerting
for: 5m
annotations:
description: "No CA rotation in the last 24h. Verify rotation service."
labels:
severity: warning
notificationSettings:
contactPoint: pagerduty
groupBy: ["alertname"]
🐳 Step 6: Update docker-compose.yml
Add Grafana environment vars:
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
- PAGERDUTY_ROUTING_KEY=your_pagerduty_key
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxx/yyyy/zzzz
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- grafana_data:/var/lib/grafana
🚀 Step 7: Run
docker-compose up -d
- Grafana loads alerts, dashboards, and datasources automatically.
- If rotation fails, PagerDuty or Slack is notified instantly.
- Prometheus/Alertmanager can still handle alerts in parallel.
🔥 Benefits
✅ No manual Grafana clicking — fully reproducible in GitOps style.
✅ PagerDuty & Slack contact points set at startup.
✅ Alerts tied directly to CA rotation metrics.
✅ Easy to extend with email, Teams, or OpsGenie.
Would you like me to show a screenshot-style layout of this Grafana dashboard with panels and alert states (so your team sees exactly what’s deployed)?
========================
=========================
Nice — let’s Feynman this: I’ll explain why you’d use PKCS#11 + HSM for your root keys, then give practical, copy-paste-ready steps for both dev (SoftHSM) and production (real HSM like CloudHSM / YubiHSM), plus configuration snippets you can drop into your OpenBao / rotation flow. I’ll also show a small signing-proxy option if OpenBao you run can’t load PKCS#11 directly.
I’ll keep it hands-on and concrete.
1) High-level idea (like I’m teaching a beginner)
- The root private key is the most sensitive secret for your PKI. Keep it inside hardware (HSM) so it never appears on disk.
- PKCS#11 is the standard API HSMs expose. You talk to the HSM through a vendor PKCS#11 library (a .so).
- There are two general integration patterns:
- OpenBao loads PKCS#11 directly and uses the HSM to sign CSRs (preferred if OpenBao supports it).
- Signing proxy (a small service) loads the PKCS#11 module and signs CSRs on behalf of OpenBao via a simple HTTP API (useful if OpenBao lacks direct PKCS#11 support or you want isolation).
- For dev/test you use SoftHSM (software HSM) so you can practice the flow.
2) Prep: dev environment with SoftHSM (quick, repeatable)
Commands below assume a Linux machine.
- Install SoftHSM & pkcs11 tools (Debian/Ubuntu example):
sudo apt update
sudo apt install -y softhsm2 pcscd opensc pkcs11-tools
- Initialize a token and slot (example values):
export SOFTHSM2_CONF=/etc/softhsm/softhsm2.conf # default path
softhsm2-util --init-token --slot 0 --label "ROOT_TOKEN" \
--pin 1234 --so-pin 987654
- Find the PKCS#11 module path:
# common path for SoftHSM
ls -l /usr/lib/softhsm/libsofthsm2.so
MODULE_PATH=/usr/lib/softhsm/libsofthsm2.so
- Create an RSA keypair inside the HSM (private key never leaves):
# create an RSA key in the token using pkcs11-tool
pkcs11-tool --module $MODULE_PATH --login --pin 1234 \
--keypairgen --key-type rsa:4096 --label root-key
- Export the public key (so you can build the root certificate):
pkcs11-tool --module $MODULE_PATH --login --pin 1234 \
--read-object --type pubkey --label root-key -y rsa > root_pub.pem
- Create a self-signed root certificate using the HSM to sign (using OpenSSL + engine_pkcs11 or p11tool):
- If you have p11tool (from GnuTLS) you can create a self-signed cert:
p11tool --provider=$MODULE_PATH --login --label=root-key \
--outfile=root_ca.crt --generate-self-signed --dn="CN=My HSM Root CA"
- Or use openssl with the PKCS#11 engine (example depends on engine availability). Many distros provide engine_pkcs11 or openssl-pkcs11 wrappers.
Now you have root_ca.crt (PEM) and the root private key stays inside SoftHSM.
3) Option A — Configure OpenBao to use PKCS#11 directly (preferred)
Status note: OpenBao is Vault-compatible in API and many deployments follow Vault patterns. If your OpenBao distribution supports PKCS#11 configuration (many Vault-like servers do), the config will mention the module path and key label/token. If you aren’t sure whether your OpenBao build supports PKCS#11, check its docs or use the signing-proxy pattern below.
Here’s a Vault-style PKCS#11 snippet (adapt to OpenBao config format; variables are placeholders):
# Example: openbao server config block (conceptual / Vault-like)
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "false"
}
pkcs11 {
module = "/usr/lib/softhsm/libsofthsm2.so" # PKCS#11 module
token_label = "ROOT_TOKEN"
key_label = "root-key"
pin = "1234"
}
When configured, OpenBao will use the HSM via PKCS#11 to sign intermediate CSRs. Your rotate-intermediate.sh can keep calling OpenBao’s sign-intermediate endpoint — OpenBao will perform signing inside the HSM.
How to confirm: after startup, call the OpenBao sign endpoint with a CSR and verify openssl verify with the root cert.
4) Option B — Signing proxy (universal, simple, minimal privileges)
If OpenBao can’t load PKCS#11 directly, run a tiny signing service that:
- Accepts a CSR over HTTPS from OpenBao (or the rotation job).
- Uses PKCS#11 to sign it with the root key inside the HSM.
- Returns the signed cert PEM bundle.
Advantages:
- Keeps HSM access separate from OpenBao process.
- Easy to audit (single purpose).
- Can enforce ACLs, mTLS, and rate limits.
Minimal Python example using python-pkcs11 (conceptual—libraries and install steps omitted):
# sign_proxy.py (concept)
from flask import Flask, request, jsonify
from pkcs11 import PKCS11Lib, KeyType, Attribute, ObjectClass
from cryptography import x509
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
import base64
app = Flask(__name__)
lib = PKCS11Lib('/usr/lib/softhsm/libsofthsm2.so')
token = lib.get_token(token_label='ROOT_TOKEN')
session = token.open(user_pin='1234')
@app.route("/sign", methods=["POST"])
def sign():
csr_pem = request.files['csr'].read()
csr = x509.load_pem_x509_csr(csr_pem)
# find private key object in HSM by label
priv = session.get_key(label='root-key', key_type=KeyType.RSA, object_class=ObjectClass.PRIVATE_KEY)
# sign the CSR's TBSCertificate using HSM (you need to construct the certificate structure
# and use PKCS#11 sign; exact code depends on library support).
signed_cert_pem = do_hsm_signing(priv, csr) # implement carefully
return signed_cert_pem, 200, {'Content-Type': 'application/pem-certificate-chain'}
# secure this endpoint: mTLS, IP allowlist, auth, etc.
You’d run this proxy in Docker and expose it only to OpenBao (private network). The rotate-intermediate.sh can call OpenBao’s API or call the proxy directly when rotation is needed.
5) Integration with the rotate flow you already have
If OpenBao uses HSM via PKCS#11 directly, no change to the rotate-intermediate.sh is required: the script creates CSR -> POST to OpenBao sign-intermediate -> OpenBao signs inside HSM -> script installs intermediate -> kill -HUP 1.
If using a signing proxy, change the perform_rotation() step to POST the CSR to the proxy instead of OpenBao:
# in rotate-intermediate.sh (sign-with-proxy)
curl -sS -X POST -F "csr=@${CSR_TMP}" https://sign-proxy.internal/sign -o "$CRT_TMP"
Then mv and kill -HUP 1 as before.
6) Production examples & hardening
- HSM vendors: AWS CloudHSM, Azure Dedicated HSM, YubiHSM2, Thales Luna, etc. Each has its own PKCS#11 library path and initialization process. Replace MODULE_PATH and PINs accordingly.
- Device / container placement:
- For software HSM (SoftHSM) you can run in the same container for dev only.
- For hardware HSMs, mount the vendor driver library into the container (volumes:) and give the container access to the device. Prefer a dedicated signing proxy container per HSM host to reduce attack surface.
- Auth & network:
- Use mTLS between OpenBao (or rotator) and the signing proxy.
- Restrict who can call sign endpoints (AppRole policies, ACLs).
- Auditing:
- Enable HSM audit logs and OpenBao audit logs so every signature operation is recorded.
- Key lifecycle:
- Keep root key offline as long as possible; many orgs use the HSM only for signing during an automated rotation window, then keep keys in an HSM that is offline otherwise.
- Backups & DR:
- Use HSM cluster features or secure backups of configuration/certificates (not private keys).
7) Troubleshooting tips
- If pkcs11-tool --list-mechanisms shows no mechanisms, driver/module path may be wrong — check vendor docs.
- For OpenSSL + PKCS#11 engines, versions and engine names vary; many distros now use engine_pkcs11 or openssl-pkcs11. Search installed packages for pkcs11 or pcks11.
- If signing fails, confirm HSM policies permit SIGN with key and verify PIN is correct.
- Test by signing a small message with pkcs11-tool --sign then verifying with public key.
8) Concrete checklist / commands summary (dev -> production)
Dev (SoftHSM):
- Install SoftHSM; init token; generate key: pkcs11-tool --keypairgen ...
- Create self-signed root cert with p11tool or openssl + engine.
- Point OpenBao config to PKCS#11 module or run signing proxy.
Prod (HSM):
- Provision HSM and create key in secure environment (follow vendor docs).
- Export public key or certificate signing request for the root cert; create root cert signed by HSM.
- Configure OpenBao to use PKCS#11 module path + token + label OR run a hardened signing-proxy that uses PKCS#11.
- Test CSR signing: ask OpenBao (or proxy) to sign a CSR and validate chain.
- Hook rotate-intermediate.sh to call OpenBao sign endpoint (no change if OpenBao handles PKCS#11).
9) Example OpenBao / Vault-like PKCS#11 hint (if supported)
If OpenBao supports the same PKCS#11 syntax as Vault, the provider config might look like this (conceptual):
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "false"
}
kms "pkcs11" {
module = "/usr/lib/softhsm/libsofthsm2.so"
token_label = "ROOT_TOKEN"
key_label = "root-key"
pin = "1234"
}
If that works, your rotate script is unchanged and OpenBao performs signing in-HSM.
10) Final practical recommendation (what I’d do)
- Use PKCS#11-backed HSM for root keys in production. Don’t mount root key files.
- If OpenBao can be configured to use PKCS#11 directly, do that — simpler.
- Otherwise, implement a small signing-proxy (mTLS + logging) that uses PKCS#11. Keep it narrow-scoped and heavily audited.
- Test the whole flow in SoftHSM before switching to a real HSM.
If you want, I’ll:
- produce a full rotate-intermediate.sh specifically adjusted to call an OpenBao instance that uses PKCS#11 (unchanged) or to call a signing-proxy;
- or generate a docker-compose dev example with SoftHSM, OpenBao (or mock), and the signing-proxy so you can test locally.
Which would you like next?
==============================
Great request — let’s Feynman this into a dev lab so you can simulate a hardware-backed CA signing flow with PKCS#11, OpenBao, and a signing-proxy inside Docker Compose.
🧠 Step 1: Explain Like I’m 5
We want to test PKCS#11-based key storage without a real HSM.
So:
- Use SoftHSM: software simulates an HSM that supports PKCS#11.
- Use OpenBao: mock Vault/OpenBao instance configured to use PKCS#11 via SoftHSM.
- Use a signing-proxy: A service that talks PKCS#11 and signs CSRs for step-ca.
- Compose them together in Docker so you have a fully working lab.
🛠 Step 2: Architecture
+-------------------+ PKCS#11 calls +-------------+
| signing-proxy | <-----------------> | SoftHSM2 |
| (REST/GRPC API) | +-------------+
+-------------------+ ^
| HTTP API |
v |
+-------------------+ PKCS#11 driver +-----------+
| OpenBao (Vault) | <------------------> | PKCS#11 .so|
+-------------------+ +-----------+
This lets you simulate hardware-backed root signing locally.
📂 Step 3: File Layout
docker-compose.yml
softhsm/
tokens/
openbao/
config.hcl
signing-proxy/
Dockerfile
config.yaml
rotate-intermediate.sh
📜 Step 4: docker-compose.yml
version: "3.9"
services:
softhsm:
image: ghcr.io/opendnssec/softhsm:2.6.1
container_name: softhsm
environment:
- SOFTHSM2_CONF=/etc/softhsm/softhsm2.conf
volumes:
- ./softhsm/tokens:/var/lib/softhsm/tokens
- ./softhsm/softhsm2.conf:/etc/softhsm/softhsm2.conf
command: sleep infinity
openbao:
image: openbao/openbao:latest
container_name: openbao
environment:
- VAULT_DEV_ROOT_TOKEN_ID=root
- VAULT_ADDR=http://0.0.0.0:8200
- VAULT_API_ADDR=http://openbao:8200
ports:
- "8200:8200"
volumes:
- ./openbao:/vault/config
command: server -config=/vault/config/config.hcl
depends_on:
- softhsm
signing-proxy:
build: ./signing-proxy
container_name: signing-proxy
environment:
- PKCS11_MODULE=/usr/lib/softhsm/libsofthsm2.so
- PKCS11_SLOT=0
- PKCS11_PIN=1234
ports:
- "8443:8443"
depends_on:
- softhsm
volumes:
softhsm_data:
📜 Step 5: softhsm/softhsm2.conf
directories.tokendir = /var/lib/softhsm/tokens
objectstore.backend = file
slots.removable = false
📜 Step 6: openbao/config.hcl
storage "file" {
path = "/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
# PKCS#11 integration example
pkcs11 {
module_path = "/usr/lib/softhsm/libsofthsm2.so"
pin = "1234"
token_label = "bao-root"
slot = "0"
}
📜 Step 7: signing-proxy/config.yaml
pkcs11:
module: "/usr/lib/softhsm/libsofthsm2.so"
slot: 0
pin: "1234"
server:
listen: ":8443"
tls:
cert: "/etc/proxy/certs/proxy.crt"
key: "/etc/proxy/certs/proxy.key"
📜 Step 8: Example rotate-intermediate.sh
#!/bin/bash
set -euo pipefail
OPENBAO_ADDR=${OPENBAO_ADDR:-http://openbao:8200}
OPENBAO_TOKEN=${OPENBAO_TOKEN:-root}
SIGNING_PROXY=${SIGNING_PROXY:-https://signing-proxy:8443}
# Fetch intermediate CSR from step-ca
csr=$(curl -s -X POST http://step-ca:9000/new-intermediate | jq -r .csr)
# Ask signing-proxy to sign with PKCS#11 root
signed_cert=$(curl -s -X POST $SIGNING_PROXY/sign \
-H "Content-Type: application/json" \
-d "{\"csr\": \"$csr\"}" | jq -r .certificate)
# Push signed cert to step-ca
curl -s -X POST http://step-ca:9000/install-intermediate \
-d "{\"certificate\": \"$signed_cert\"}"
This script assumes:
- step-ca is another service you spin up in the same network.
- Signing-proxy signs the CSR using the PKCS#11-backed root key.
🐳 Step 9: SoftHSM Init
Before running:
mkdir -p softhsm/tokens
docker-compose up -d softhsm
docker exec -it softhsm softhsm2-util --init-token --slot 0 --label "bao-root" --pin 1234 --so-pin 1234
🚀 Step 10: Run Entire Lab
docker-compose up -d
- OpenBao listens at http://localhost:8200
- Signing proxy at https://localhost:8443
- PKCS#11 calls go through SoftHSM
- Rotate script signs new intermediates with the simulated HSM key
🔥 Why This Rocks
✅ Lets you test real PKCS#11 flows without an actual HSM.
✅ Simulates OpenBao (Vault) PKCS#11 mode and a REST proxy.
✅ Fully containerized for dev/test pipelines.
✅ Easy to swap SoftHSM with a real HSM later.
Would you like me to add step-ca itself into this Compose file so the whole rotation cycle runs end-to-end inside one docker-compose up?
========================
Perfect — let’s extend the dev lab so step-ca is in the same docker-compose.yml. This will make your rotation pipeline fully runnable locally with one docker-compose up.
🧠 Step 1: Concept Recap
We already have:
- SoftHSM: Simulates a hardware HSM via PKCS#11.
- OpenBao: Mock Vault/OpenBao instance integrated with PKCS#11.
- signing-proxy: Talks to PKCS#11 (via SoftHSM) to sign CSRs.
- rotate-intermediate.sh: Requests a CSR from step-ca, gets it signed, and pushes the cert back.
We now add step-ca so the CA runs inside the same lab, with its root key “secured” in HSM (via signing-proxy).
🛠 Step 2: Updated docker-compose.yml
Here’s the complete Compose file:
version: "3.9"
services:
softhsm:
image: ghcr.io/opendnssec/softhsm:2.6.1
container_name: softhsm
environment:
- SOFTHSM2_CONF=/etc/softhsm/softhsm2.conf
volumes:
- ./softhsm/tokens:/var/lib/softhsm/tokens
- ./softhsm/softhsm2.conf:/etc/softhsm/softhsm2.conf
command: sleep infinity
openbao:
image: openbao/openbao:latest
container_name: openbao
environment:
- VAULT_DEV_ROOT_TOKEN_ID=root
- VAULT_ADDR=http://0.0.0.0:8200
- VAULT_API_ADDR=http://openbao:8200
ports:
- "8200:8200"
volumes:
- ./openbao:/vault/config
command: server -config=/vault/config/config.hcl
depends_on:
- softhsm
signing-proxy:
build: ./signing-proxy
container_name: signing-proxy
environment:
- PKCS11_MODULE=/usr/lib/softhsm/libsofthsm2.so
- PKCS11_SLOT=0
- PKCS11_PIN=1234
ports:
- "8443:8443"
depends_on:
- softhsm
step-ca:
image: smallstep/step-ca:latest
container_name: step-ca
environment:
- DOCKER_STEPCA_INIT_NAME=Local Dev CA
- DOCKER_STEPCA_INIT_DNS=step-ca
- DOCKER_STEPCA_INIT_REMOTE_MANAGEMENT=true
- DOCKER_STEPCA_PASSWORD=changeit
volumes:
- ./step:/home/step
ports:
- "9000:9000"
command: ["step-ca", "/home/step/config/ca.json"]
depends_on:
- signing-proxy
volumes:
softhsm_data:
🗂 Supporting Files
softhsm/softhsm2.conf
directories.tokendir = /var/lib/softhsm/tokens
objectstore.backend = file
slots.removable = false
openbao/config.hcl
storage "file" {
path = "/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
pkcs11 {
module_path = "/usr/lib/softhsm/libsofthsm2.so"
pin = "1234"
token_label = "bao-root"
slot = "0"
}
signing-proxy/config.yaml
pkcs11:
module: "/usr/lib/softhsm/libsofthsm2.so"
slot: 0
pin: "1234"
server:
listen: ":8443"
tls:
cert: "/etc/proxy/certs/proxy.crt"
key: "/etc/proxy/certs/proxy.key"
step/config/ca.json
A minimal config that references signing-proxy for signing requests:
{
"address": ":9000",
"dnsNames": ["step-ca"],
"db": {
"type": "badgerV2",
"dataSource": "/home/step/db"
},
"authority": {
"provisioners": [
{
"type": "ACME",
"name": "acme"
}
],
"signingProxy": {
"url": "https://signing-proxy:8443"
}
}
}
rotate-intermediate.sh
#!/bin/bash
set -euo pipefail
OPENBAO_ADDR=${OPENBAO_ADDR:-http://openbao:8200}
OPENBAO_TOKEN=${OPENBAO_TOKEN:-root}
SIGNING_PROXY=${SIGNING_PROXY:-https://signing-proxy:8443}
STEP_CA=${STEP_CA:-http://step-ca:9000}
echo "[*] Requesting new intermediate CSR from step-ca..."
csr=$(curl -s -X POST ${STEP_CA}/new-intermediate | jq -r .csr)
echo "[*] Asking signing-proxy to sign..."
signed_cert=$(curl -s -X POST $SIGNING_PROXY/sign \
-H "Content-Type: application/json" \
-d "{\"csr\": \"$csr\"}" | jq -r .certificate)
echo "[*] Installing new intermediate cert into step-ca..."
curl -s -X POST ${STEP_CA}/install-intermediate \
-H "Content-Type: application/json" \
-d "{\"certificate\": \"$signed_cert\"}"
echo "[+] Rotation complete."
🧪 Step 3: Initialize and Run
- Initialize SoftHSM slot and token:
- mkdir -p softhsm/tokens
- docker-compose up -d softhsm
- docker exec -it softhsm softhsm2-util --init-token --slot 0 \
- --label "bao-root" --pin 1234 --so-pin 1234
- Bring up everything:
- docker-compose up -d
- Test step-ca:
- curl http://localhost:9000/health
- Run a manual rotation:
- ./rotate-intermediate.sh
🔥 Why This is Powerful
- Full PKCS#11 simulation using SoftHSM (swap with a real HSM later).
- step-ca integrated directly with signing-proxy for external root signing.
- OpenBao in PKCS#11 mode for token management.
- One docker-compose up gives a realistic PKI lab for dev/test.
Would you like me to add Prometheus scraping & Grafana dashboards here so you can see rotation health metrics live inside this same Docker Compose stack?
=============================
Perfect — let’s level this up with the Feynman Technique so we:
✅ Use Caddy with DNS challenges for wildcard domains (like *.example.com)
✅ Let Caddy terminate HTTPS for both the CA’s public endpoint and its admin interface (keeping step-ca internal HTTP-only).
This makes your setup production-grade:
- Wildcard certs = no need to request per-host certs
- Admin UI/API access is locked behind TLS without exposing raw step-ca
🧠 Step 1: Explain Simply
Right now:
- Caddy just proxies ca.example.com over HTTPS with Let’s Encrypt HTTP challenges.
- Step-ca listens on port 9000 internally.
- Admin API is also on the same endpoint, no extra security layer.
We want:
- Caddy to use DNS-based ACME challenges (instead of HTTP) so we can issue wildcard certs (like *.example.com).
- Split public ACME endpoint from admin interface:
- ca.example.com → For issuing certs (ACME clients)
- admin.example.com → For admin access (protected with TLS and optional auth).
- Keep step-ca behind the reverse proxy with internal HTTP only.
🛠 Step 2: Key Pieces
- Caddy DNS challenge plugin: Lets Caddy prove domain ownership via DNS.
- API tokens: For your DNS provider (Cloudflare, Route53, DigitalOcean, etc.).
- Separate vhosts in Caddyfile: One for public CA ACME, one for admin UI.
- Caddy TLS termination: Clients see HTTPS; step-ca stays internal HTTP.
🧩 Step 3: Build It
🔹 1. Install Caddy with DNS Plugin
We’ll use a pre-built Caddy with the right DNS plugin (example: Cloudflare):
caddy:
image: caddy:latest
# For Cloudflare DNS challenge, use: caddy:latest-builder + xcaddy build
If you want to build manually:
xcaddy build \
--with github.com/caddy-dns/cloudflare
Replace cloudflare with your DNS provider.
🔹 2. Set DNS API Token
In .env file (to keep secrets safe):
CF_API_TOKEN=your-cloudflare-dns-token
🔹 3. Update Caddyfile
Here’s the new Caddyfile:
{
email admin@example.com
}
# Public ACME endpoint
ca.example.com {
tls {
dns cloudflare {env.CF_API_TOKEN}
}
reverse_proxy step-ca:9000
}
# Admin interface over TLS
admin.example.com {
tls {
dns cloudflare {env.CF_API_TOKEN}
}
reverse_proxy step-ca:9000
basicauth / {
admin JDJhJDE0JHJvZzNQUmM0cmw3U2o4cGZjc2EwbU5KbFN2R25uVnNoT3pxZmxBSW9sd1N2Q0Q3d0xuMkhm # bcrypt hash for password
}
}
What’s happening:
- ca.example.com → Used by ACME clients to request certs.
- admin.example.com → Admin UI/API endpoint with BasicAuth (use bcrypt-hashed passwords).
- Both domains get wildcard-ready certs from Let’s Encrypt via DNS challenge.
🔹 4. Updated docker-compose.yml
version: "3.8"
services:
step-ca:
build: .
container_name: step-ca
restart: unless-stopped
environment:
- CA_NAME=My Public CA
- CA_DNS=ca.example.com
- CA_PROVISIONER=admin@example.com
- CA_PASSWORD=changeit
volumes:
- ./step-ca/config:/home/step/config
- ./step-ca/data:/home/step/data
networks:
- stepnet
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
env_file:
- .env
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./caddy_data:/data
- ./caddy_config:/config
networks:
- stepnet
networks:
stepnet:
driver: bridge
🔹 5. Folder Structure
.
├── Dockerfile
├── docker-compose.yml
├── Caddyfile
├── .env
├── init-step-ca.sh
└── step-ca/
├── config/
└── data/
🔍 Step 4: Simplify & Review
Now your system works like this:
- docker-compose up -d spins everything up.
- Caddy gets wildcard certs for both *.example.com and admin.example.com.
- Step-ca only talks HTTP internally, Caddy handles TLS.
- Admin UI/API is now at https://admin.example.com and password-protected.
- ACME clients talk to https://ca.example.com/acme/acme/directory to request certs.
🔒 Security Wins:
- Separation of concerns: Step-ca stays internal; Caddy is the public TLS front.
- Wildcard TLS: Easy to serve multiple services under your domain.
- Admin protection: BasicAuth + TLS keeps admin endpoints safe.
Would you also like me to automate wildcard cert issuance for your own services (like web apps) using this CA’s ACME endpoint — so everything in your domain auto-gets internal certs?
========================