implementing-tls

Original source / Raw SKILL.md

---
name: implementing-tls
description: Configure TLS certificates and encryption for secure communications. Use when setting up HTTPS, securing service-to-service connections, implementing mutual TLS (mTLS), or debugging certificate issues.
---

# Implementing TLS

## Purpose

Implement Transport Layer Security (TLS) for encrypting network communications and authenticating services. Generate certificates, automate certificate lifecycle management with Let's Encrypt or internal CAs, configure TLS 1.3, implement mutual TLS for service authentication, and debug common certificate issues.

## When to Use This Skill

Trigger this skill when:
- Setting up HTTPS for web applications or APIs
- Securing service-to-service communication in microservices
- Implementing mutual TLS (mTLS) for zero-trust networks
- Generating certificates for development or production
- Automating certificate renewal and rotation
- Debugging certificate validation errors
- Configuring TLS termination at load balancers
- Setting up internal PKI for corporate networks

## Quick Start

### For Development (Local HTTPS)

Use mkcert for trusted local certificates:

```bash
# Install mkcert
brew install mkcert  # macOS
# sudo apt install mkcert  # Linux

# Install local CA
mkcert -install

# Generate certificate
mkcert example.com localhost 127.0.0.1
# Creates: example.com+2.pem and example.com+2-key.pem
```

### For Production (Public HTTPS)

**Kubernetes with cert-manager:**
```bash
# Install cert-manager
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

# Create Let's Encrypt issuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx
EOF
```

**Traditional servers with Certbot:**
```bash
# Install certbot
sudo apt install certbot

# Obtain certificate
sudo certbot certonly --standalone -d example.com -d www.example.com
# Certificates saved to /etc/letsencrypt/live/example.com/
```

### For Internal Services (Internal PKI)

Generate internal CA with CFSSL:

```bash
# Install CFSSL
brew install cfssl  # macOS

# Create CA
cfssl genkey -initca ca-csr.json | cfssljson -bare ca

# Generate server certificate
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=server \
  server-csr.json | cfssljson -bare server
```

See `examples/cfssl-ca/` for complete configuration files.

## TLS 1.3 Configuration Best Practices

### Protocol Versions

Enable TLS 1.3 and 1.2 only:
```nginx
# Nginx
ssl_protocols TLSv1.3 TLSv1.2;
ssl_prefer_server_ciphers off;  # Let client choose
```

Disable obsolete protocols: SSLv3, TLS 1.0, TLS 1.1.

### Cipher Suites

**TLS 1.3 (5 cipher suites):**
```
TLS_AES_256_GCM_SHA384           # Recommended
TLS_CHACHA20_POLY1305_SHA256     # Mobile-optimized
TLS_AES_128_GCM_SHA256           # Performance
```

**TLS 1.2 fallback:**
```nginx
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305';
```

### Security Features

- **Perfect Forward Secrecy (PFS)**: Use ephemeral key exchanges (ECDHE)
- **OCSP Stapling**: Enable for performance and privacy
- **HSTS**: Force HTTPS with `Strict-Transport-Security` header
- **Disable compression**: Prevent CRIME attacks

For detailed TLS 1.3 configuration, see `references/tls13-best-practices.md`.

## Decision Framework

### Certificate Type Selection

```
Need TLS certificate?
│
├─ Public-facing (internet users)?
│  │
│  ├─ Single domain → Let's Encrypt with HTTP-01
│  │  Tools: certbot, cert-manager
│  │  Challenge: HTTP verification
│  │
│  └─ Multiple subdomains → Let's Encrypt with DNS-01
│     Tools: certbot with DNS plugin, cert-manager
│     Challenge: DNS TXT records
│     Supports: Wildcard certificates (*.example.com)
│
└─ Internal (corporate network)?
   │
   ├─ Development → mkcert or self-signed
   │  Tools: mkcert (trusted), openssl (basic)
   │  No automation needed
   │
   └─ Production → Internal CA
      │
      ├─ Small scale (<10 services) → CFSSL
      │  Manual management acceptable
      │
      └─ Large scale (100+ services) → Vault PKI or cert-manager
         Dynamic secrets, automatic rotation
```

### Automation Tool Selection

```
Environment?
│
├─ Kubernetes → cert-manager
│  Native CRDs, Ingress integration
│  Supports: Let's Encrypt, Vault, CA, self-signed
│
├─ Traditional servers (VMs) → Certbot (public) or CFSSL (internal)
│  Plugins: nginx, apache, DNS providers
│  Automated renewal via cron/systemd
│
├─ Microservices (any platform) → HashiCorp Vault PKI
│  Dynamic secrets, short-lived certs
│  API-driven, service mesh integration
│
└─ Developer workstation → mkcert
   Trusted by browser automatically
```

### Standard TLS vs Mutual TLS (mTLS)

**Use Standard TLS (server-only authentication) when:**
- Public websites (users trust server)
- APIs with bearer tokens (separate auth layer)
- Services behind API gateway
- Simple architectures (<5 services)

**Use Mutual TLS (both authenticate) when:**
- Service-to-service in microservices
- High security requirements (financial, healthcare)
- Machine-to-machine APIs
- Zero-trust networks
- No shared network trust

See `references/mtls-guide.md` for mTLS implementation patterns.

## Common Workflows

### Generate Self-Signed Certificate

**Quick generation with SANs:**
```bash
# Create OpenSSL config
cat > san.cnf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[dn]
CN = example.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = api.example.com
IP.1 = 192.168.1.100
EOF

# Generate key and certificate
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout server-key.pem -out server-cert.pem \
  -days 365 -config san.cnf -extensions v3_req

# Verify SANs
openssl x509 -in server-cert.pem -noout -text | grep -A 3 "Subject Alternative Name"
```

For detailed examples including CFSSL and mkcert, see `references/certificate-generation.md` and `examples/self-signed/`.

### Setup Let's Encrypt Automation

**With Certbot (traditional servers):**
```bash
# Standalone mode (port 80 must be free)
sudo certbot certonly --standalone -d example.com -d www.example.com

# Webroot mode (no service interruption)
sudo certbot certonly --webroot -w /var/www/html -d example.com

# DNS challenge (wildcard support)
sudo certbot certonly --manual --preferred-challenges dns \
  -d example.com -d "*.example.com"

# Test renewal
sudo certbot renew --dry-run
```

**With cert-manager (Kubernetes):**
```yaml
# Ingress with automatic certificate
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80
```

See `references/automation-patterns.md` for complete automation guides.

### Configure Mutual TLS (mTLS)

**Server configuration (Nginx):**
```nginx
server {
    listen 443 ssl;
    server_name api.example.com;

    # Server certificate
    ssl_certificate /etc/ssl/certs/server.crt;
    ssl_certificate_key /etc/ssl/private/server.key;

    # CA to verify client certificates
    ssl_client_certificate /etc/ssl/certs/ca.crt;
    ssl_verify_client on;
    ssl_verify_depth 2;

    # TLS 1.3
    ssl_protocols TLSv1.3;

    location / {
        proxy_pass http://backend;
        # Pass client cert info to backend
        proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
        proxy_set_header X-SSL-Client-S-DN $ssl_client_s_dn;
    }
}
```

**Client request with certificate:**
```bash
curl https://api.example.com/endpoint \
  --cert client.crt \
  --key client.key \
  --cacert ca.crt
```

See `references/mtls-guide.md` and `examples/mtls-nginx/` for complete mTLS implementations.

### Debug TLS Issues

**Test TLS connection:**
```bash
# Basic connection test
openssl s_client -connect example.com:443

# Show certificate chain
openssl s_client -connect example.com:443 -showcerts

# Test specific TLS version
openssl s_client -connect example.com:443 -tls1_3

# Test with client certificate (mTLS)
openssl s_client -connect api.example.com:443 \
  -cert client.crt -key client.key -CAfile ca.crt
```

**Examine certificate:**
```bash
# View certificate details
openssl x509 -in cert.pem -noout -text

# Check expiration
openssl x509 -in cert.pem -noout -dates

# Check Subject Alternative Names
openssl x509 -in cert.pem -noout -text | grep -A 1 "Subject Alternative Name"

# Verify certificate chain
openssl verify -CAfile ca.crt cert.pem
```

**Verify key and certificate match:**
```bash
# Certificate modulus
openssl x509 -in cert.pem -noout -modulus | md5sum

# Key modulus (must match)
openssl rsa -in key.pem -noout -modulus | md5sum
```

**Common errors and solutions:**

| Error | Cause | Solution |
|-------|-------|----------|
| `certificate has expired` | Certificate validity passed | Renew certificate, check system clock |
| `unable to get local issuer certificate` | CA not in trust store | Add CA cert to system trust store |
| `Hostname mismatch` | CN/SAN doesn't match hostname | Regenerate cert with correct SANs |
| `handshake failure` | TLS version/cipher mismatch | Enable TLS 1.2+, check cipher suites |
| `certificate signed by unknown authority` | Missing intermediate certs | Include full chain in server config |

See `references/debugging-tls.md` for comprehensive troubleshooting guide.

## Tool Selection Guide

| Use Case | Environment | Recommended Tool | Alternative |
|----------|-------------|------------------|-------------|
| Public HTTPS | Kubernetes | cert-manager | External Secrets Operator |
| Public HTTPS | VMs/Bare Metal | Certbot | acme.sh |
| Internal PKI | Any | HashiCorp Vault | CFSSL, Smallstep |
| mTLS (K8s) | Kubernetes | cert-manager + Istio | Linkerd, Consul |
| mTLS (VMs) | Traditional | Vault PKI | CFSSL |
| Local Dev | Workstation | mkcert | Self-signed (OpenSSL) |
| Debugging | Any | OpenSSL s_client | curl -v |
| Automation | CI/CD | CFSSL API | Vault API |

## Certificate Lifecycle

```
1. Generate
   ├─ Development: mkcert, self-signed (OpenSSL)
   ├─ Production: Let's Encrypt, commercial CA
   └─ Internal: CFSSL, Vault PKI

2. Deploy
   ├─ Kubernetes: Mount as Secret volume
   ├─ VMs: Copy to /etc/ssl/ or application directory
   └─ Containers: Mount via Docker volumes

3. Monitor
   ├─ Check expiry: openssl x509 -noout -dates
   ├─ Prometheus: blackbox_exporter (probe_ssl_earliest_cert_expiry)
   └─ Alert: < 7 days before expiry

4. Renew
   ├─ Automated: certbot renew, cert-manager, Vault Agent
   ├─ Manual: Generate new CSR, reissue from CA
   └─ Timing: Renew 30 days before expiry

5. Rotate
   ├─ Zero-downtime: Load new cert, graceful reload
   ├─ Kubernetes: Update Secret, rolling restart
   └─ Service mesh: Automatic rotation (Istio, Linkerd)
```

## Certificate Formats

**PEM (most common):**
- Extensions: .pem, .crt, .cer, .key
- Base64 encoded, ASCII text
- Used by: Apache, Nginx, OpenSSL

**DER (binary):**
- Extensions: .der, .cer
- Binary format
- Used by: Java, Windows

**PKCS#12 / PFX (container):**
- Extensions: .p12, .pfx
- Contains certificate + private key (password protected)
- Used by: Windows, Java keystores, browsers

**Convert formats:**
```bash
# PEM to DER
openssl x509 -in cert.pem -outform DER -out cert.der

# PEM to PKCS#12
openssl pkcs12 -export -out cert.p12 -inkey key.pem -in cert.pem

# PKCS#12 to PEM
openssl pkcs12 -in cert.p12 -out cert.pem -nodes
```

See `scripts/convert-formats.sh` for automated conversion.

## References

### Detailed Guides
- **references/certificate-generation.md** - Comprehensive generation examples (OpenSSL, CFSSL, mkcert)
- **references/automation-patterns.md** - Automation deep-dive (Certbot, cert-manager, Vault PKI)
- **references/mtls-guide.md** - mTLS implementation patterns and architecture
- **references/debugging-tls.md** - Troubleshooting guide with common errors and solutions
- **references/tls13-best-practices.md** - TLS 1.3 configuration and security features

## Examples

### Working Code
- **examples/self-signed/** - Self-signed certificate generation scripts
- **examples/cfssl-ca/** - Internal CA setup with CFSSL (complete configuration)
- **examples/certbot/** - Let's Encrypt automation (standalone, webroot, DNS challenges)
- **examples/cert-manager/** - Kubernetes certificate management (ClusterIssuer, Ingress)
- **examples/mtls-nginx/** - Mutual TLS with Nginx (server + client configuration)
- **examples/vault-pki/** - Vault PKI integration and dynamic certificates

## Scripts

### Utility Tools
- **scripts/check-cert-expiry.sh** - Monitor certificate expiration across multiple domains
- **scripts/validate-chain.sh** - Verify certificate chain integrity
- **scripts/test-tls-connection.sh** - Test TLS connections with various options
- **scripts/convert-formats.sh** - Convert between PEM, DER, and PKCS#12 formats

## Related Skills

**Security and Authentication:**
- **secret-management** - Store private keys securely (Vault, Kubernetes Secrets, HSM)
- **auth-security** - Application-level authentication (OAuth, OIDC, JWT)
- **security-hardening** - System security configuration
- **security-architecture** - Holistic security design and threat modeling

**Infrastructure:**
- **kubernetes-operations** - Kubernetes cluster TLS configuration
- **load-balancing-patterns** - TLS termination at load balancers
- **network-architecture** - Network security design

**Operations:**
- **deploying-applications** - Inject certificates at runtime
- **observability** - Monitor certificate health and expiry
- **building-ci-pipelines** - Automate certificate generation in CI/CD


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### references/tls13-best-practices.md

```markdown
# TLS 1.3 Best Practices and Configuration

Guide to configuring TLS 1.3 for optimal security and performance.

## Table of Contents

1. [Why TLS 1.3](#why-tls-13)
2. [Protocol Configuration](#protocol-configuration)
3. [Cipher Suite Selection](#cipher-suite-selection)
4. [Security Features](#security-features)
5. [Performance Optimization](#performance-optimization)

## Why TLS 1.3

### Key Improvements Over TLS 1.2

**Performance:**
- **Faster handshake**: 1-RTT instead of 2-RTT (50% faster)
- **0-RTT resumption**: Zero round-trip time for repeat connections
- **Reduced latency**: 100-200ms saved per connection

**Security:**
- **Forward secrecy mandatory**: All cipher suites use ephemeral keys
- **Encrypted handshake**: More of the handshake is encrypted
- **Simplified cipher suites**: Removed weak algorithms
- **Downgrade protection**: Prevents protocol downgrade attacks

**Removed Vulnerabilities:**
- No RSA key transport (forward secrecy required)
- No static DH/ECDH
- No CBC mode ciphers (BEAST, Lucky13 attacks)
- No MD5, SHA-1, RC4, 3DES
- No compression (CRIME attack)
- No renegotiation

### TLS 1.3 vs TLS 1.2 Handshake

**TLS 1.2 (2-RTT):**
```
Client                          Server
  |--- ClientHello ------------->|
  |<-- ServerHello, Certificate -|
  |<-- ServerHelloDone ----------|
  |--- ClientKeyExchange ------->|
  |--- Finished ---------------->|
  |<-- Finished -----------------|
  |=== Application Data ========>|

Total: 2 round trips before data
```

**TLS 1.3 (1-RTT):**
```
Client                          Server
  |--- ClientHello + KeyShare -->|
  |<-- ServerHello, Certificate -|
  |<-- Finished -----------------|
  |=== Application Data ========>|
  |--- Finished ---------------->|

Total: 1 round trip before data
```

**TLS 1.3 with 0-RTT Resumption:**
```
Client                          Server
  |--- ClientHello + Early Data >|
  |=== Application Data ========>|  (sent immediately!)
  |<-- ServerHello ---------------|
  |<-- Finished -----------------|

Total: 0 round trips (but replay risk)
```

## Protocol Configuration

### Nginx Configuration

**Recommended TLS 1.3 configuration:**

```nginx
server {
    listen 443 ssl http2;
    server_name example.com;

    # Certificate files
    ssl_certificate /etc/ssl/certs/example.com.crt;
    ssl_certificate_key /etc/ssl/private/example.com.key;

    # Protocol versions (TLS 1.3 + TLS 1.2 fallback)
    ssl_protocols TLSv1.3 TLSv1.2;

    # Let client choose cipher (modern best practice)
    ssl_prefer_server_ciphers off;

    # TLS 1.3 cipher suites (optional - defaults are good)
    ssl_ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256;

    # TLS 1.2 cipher suites (fallback)
    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305';

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/ssl/certs/chain.pem;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;

    # Session resumption
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;  # Disable for perfect forward secrecy

    # HSTS (force HTTPS)
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    location / {
        proxy_pass http://backend;
    }
}

# HTTP to HTTPS redirect
server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}
```

### Apache Configuration

```apache
<VirtualHost *:443>
    ServerName example.com

    # Certificate files
    SSLCertificateFile /etc/ssl/certs/example.com.crt
    SSLCertificateKeyFile /etc/ssl/private/example.com.key
    SSLCertificateChainFile /etc/ssl/certs/chain.pem

    # Protocol versions
    SSLProtocol -all +TLSv1.3 +TLSv1.2

    # Cipher suites
    SSLCipherSuite TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
    SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305

    # Prefer client cipher order (modern)
    SSLHonorCipherOrder off

    # OCSP stapling
    SSLUseStapling on
    SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

    # Session resumption
    SSLSessionCache "shmcb:logs/ssl_scache(512000)"
    SSLSessionCacheTimeout 300

    # HSTS
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
</VirtualHost>
```

### HAProxy Configuration

```haproxy
frontend https_front
    bind *:443 ssl crt /etc/ssl/certs/example.com.pem ssl-min-ver TLSv1.2 alpn h2,http/1.1

    # TLS 1.3 cipher suites
    ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

    # TLS 1.2 ciphers
    ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305

    # Security options
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

    # HSTS
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

    default_backend web_servers
```

## Cipher Suite Selection

### TLS 1.3 Cipher Suites

Only 5 cipher suites in TLS 1.3 (all AEAD):

```
TLS_AES_256_GCM_SHA384           - AES-256 with GCM mode
TLS_CHACHA20_POLY1305_SHA256     - ChaCha20-Poly1305 (mobile-optimized)
TLS_AES_128_GCM_SHA256           - AES-128 with GCM mode (performance)
TLS_AES_128_CCM_SHA256           - AES-128 with CCM (constrained devices)
TLS_AES_128_CCM_8_SHA256         - AES-128 with CCM-8 (IoT)
```

**Recommended order:**

```nginx
# Let client choose (modern approach)
ssl_prefer_server_ciphers off;
ssl_ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256;
```

**Cipher characteristics:**

| Cipher | Security | Performance | Mobile | Notes |
|--------|----------|-------------|--------|-------|
| TLS_AES_256_GCM_SHA384 | Highest | Good (AES-NI) | Good | Strongest encryption |
| TLS_CHACHA20_POLY1305_SHA256 | High | Excellent | Best | Fast on ARM/mobile |
| TLS_AES_128_GCM_SHA256 | High | Best (AES-NI) | Good | Balanced choice |
| TLS_AES_128_CCM_* | High | Moderate | Good | IoT/constrained devices |

### TLS 1.2 Cipher Suites (Fallback)

**Recommended for compatibility:**

```
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-CHACHA20-POLY1305
```

**What to disable:**

```
# Disable weak ciphers
ssl_ciphers '!RC4:!DES:!3DES:!MD5:!SHA1:!aNULL:!eNULL:!EXPORT';

# Specific ciphers to avoid:
- RC4 (broken)
- DES, 3DES (too weak)
- MD5, SHA-1 (collision attacks)
- CBC mode (BEAST, Lucky13)
- RSA key transport (no forward secrecy)
- Export ciphers (intentionally weak)
- Anonymous ciphers (no authentication)
```

### Verify Cipher Configuration

```bash
# Test with OpenSSL
openssl s_client -connect example.com:443 -tls1_3

# Check negotiated cipher
# Look for "Cipher    : TLS_AES_256_GCM_SHA384"

# Test specific cipher
openssl s_client -connect example.com:443 \
  -tls1_3 -ciphersuites 'TLS_AES_256_GCM_SHA384'

# Test with nmap
nmap --script ssl-enum-ciphers -p 443 example.com

# Test with testssl.sh
./testssl.sh example.com
```

## Security Features

### OCSP Stapling

**What it does:**
- Server fetches OCSP response from CA
- Staples (includes) response in TLS handshake
- Client doesn't need to contact CA (privacy + performance)

**Nginx configuration:**

```nginx
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/certs/chain.pem;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
```

**Verify OCSP stapling:**

```bash
openssl s_client -connect example.com:443 -status

# Look for:
# OCSP Response Status: successful (0x0)
# Cert Status: good
```

### HSTS (HTTP Strict Transport Security)

**Force HTTPS for all connections:**

```nginx
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
```

**Parameters:**
- `max-age=31536000`: Cache for 1 year (seconds)
- `includeSubDomains`: Apply to all subdomains
- `preload`: Submit to HSTS preload list (browsers)

**Preload list submission:**
Visit: https://hstspreload.org/

### Session Resumption

**Session cache (server-side):**

```nginx
# Shared cache across workers
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# Disable session tickets (better forward secrecy)
ssl_session_tickets off;
```

**Why disable session tickets:**
- Session tickets stored client-side
- Encrypted with server's ticket key
- If ticket key compromised, past sessions compromised
- Disabling improves forward secrecy

### Perfect Forward Secrecy (PFS)

**Ensured in TLS 1.3** (all cipher suites use ephemeral keys).

**TLS 1.2 requires ECDHE or DHE:**

```nginx
# PFS-enabled ciphers only
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256';
```

**Verify PFS:**

```bash
openssl s_client -connect example.com:443 | grep "Server Temp Key"

# Should show: Server Temp Key: ECDH, X25519, 253 bits
# Not: No server temp key (no PFS)
```

### Certificate Transparency

**Required for public certificates (2018+):**

All certificates issued by public CAs are logged to public CT logs.

**Check CT logs:**
- https://crt.sh/
- https://transparencyreport.google.com/https/certificates

**Nginx expects CT via OCSP stapling:**
```nginx
ssl_stapling on;
ssl_stapling_verify on;
```

### 0-RTT Resumption (Use with Caution)

**Benefits:**
- Zero round-trip time for repeat connections
- Fastest possible resumption

**Risks:**
- Vulnerable to replay attacks
- Early data can be replayed by attacker
- Only use for idempotent requests (GET, not POST)

**Nginx configuration:**

```nginx
ssl_early_data on;

# Pass header to backend
location / {
    proxy_pass http://backend;
    proxy_set_header Early-Data $ssl_early_data;
}
```

**Application must check:**

```python
# Python/Flask example
@app.before_request
def check_early_data():
    if request.headers.get('Early-Data') == '1':
        # Only allow safe methods
        if request.method not in ['GET', 'HEAD', 'OPTIONS']:
            abort(425, "Too Early")
```

**When to disable 0-RTT:**
- APIs with state-changing operations
- Authentication endpoints
- Payment processing
- High-security applications

## Performance Optimization

### HTTP/2 and HTTP/3

**HTTP/2 over TLS 1.3:**

```nginx
listen 443 ssl http2;
ssl_protocols TLSv1.3 TLSv1.2;
```

**HTTP/3 (QUIC) support:**

```nginx
# Nginx 1.25.0+
listen 443 quic reuseport;
listen 443 ssl http2;

# Advertise HTTP/3 availability
add_header Alt-Svc 'h3=":443"; ma=86400';
```

### Session Cache Tuning

```nginx
# Shared cache for multiple workers
ssl_session_cache shared:SSL:10m;  # 10MB = ~40,000 sessions
ssl_session_timeout 10m;

# Monitor cache usage
# Check Nginx error log for "session cache size" messages
```

### Hardware Acceleration

**Enable AES-NI (Intel/AMD CPUs):**

```bash
# Check if AES-NI available
grep -o aes /proc/cpuinfo | wc -l
# > 0 means available

# Nginx automatically uses AES-NI if available
# OpenSSL 1.0.1+ required
openssl version
```

**Benchmark:**

```bash
# Test AES-128-GCM performance
openssl speed -evp aes-128-gcm

# Test ChaCha20-Poly1305 performance
openssl speed chacha20-poly1305
```

### Connection Reuse

```nginx
# Keep connections alive
keepalive_timeout 65;
keepalive_requests 100;

# Upstream keepalive (to backend)
upstream backend {
    server 10.0.0.1:8080;
    keepalive 32;  # Keep 32 connections per worker
}
```

## Testing and Validation

### SSL Labs Test

```
Visit: https://www.ssllabs.com/ssltest/analyze.html?d=example.com

Target rating: A+ (with HSTS preload)
```

### testssl.sh (Command-Line)

```bash
git clone https://github.com/drwetter/testssl.sh.git
cd testssl.sh

# Full test
./testssl.sh example.com

# Quick test
./testssl.sh --fast example.com

# Check specific features
./testssl.sh --protocols example.com
./testssl.sh --cipher example.com
./testssl.sh --pfs example.com
```

### Verify Configuration

```bash
# Check TLS 1.3 support
openssl s_client -connect example.com:443 -tls1_3

# Verify cipher
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  grep "Cipher"

# Check certificate chain
openssl s_client -connect example.com:443 -showcerts

# Verify OCSP stapling
openssl s_client -connect example.com:443 -status | \
  grep "OCSP Response Status"

# Check HSTS header
curl -I https://example.com | grep -i strict-transport
```

## Best Practices Summary

1. **Enable TLS 1.3 and 1.2 only** (disable 1.1, 1.0, SSLv3)
2. **Let client choose cipher** (ssl_prefer_server_ciphers off)
3. **Use strong cipher suites** (AES-GCM, ChaCha20-Poly1305)
4. **Enable OCSP stapling** (privacy + performance)
5. **Implement HSTS** (force HTTPS)
6. **Disable session tickets** (better forward secrecy)
7. **Use HTTP/2** (multiplexing, header compression)
8. **Short certificate lifetimes** (90 days or less)
9. **Monitor certificate expiry** (alert 7-30 days before)
10. **Regular security testing** (SSL Labs, testssl.sh)
11. **Keep OpenSSL updated** (security patches)
12. **Use 0-RTT cautiously** (replay attack risk)

## Common Mistakes to Avoid

1. **Enabling TLS 1.0/1.1** (insecure, deprecated)
2. **Weak cipher suites** (RC4, 3DES, CBC mode)
3. **Missing intermediate certificates** (chain incomplete)
4. **Long certificate lifetimes** (>1 year is suspicious)
5. **No HSTS** (allows downgrade attacks)
6. **Self-signed certs in production** (browser warnings)
7. **Ignoring OCSP/CRL** (revoked certs still work)
8. **Static session keys** (no forward secrecy)
9. **Compression enabled** (CRIME attack)
10. **Renegotiation allowed** (DoS risk)

## Migration Checklist

Migrating from TLS 1.2 to TLS 1.3:

- [ ] Update OpenSSL to 1.1.1+ (TLS 1.3 support)
- [ ] Update web server (Nginx 1.13+, Apache 2.4.37+)
- [ ] Configure ssl_protocols (add TLSv1.3)
- [ ] Test with modern clients (Chrome, Firefox, curl)
- [ ] Monitor for connection failures (old clients)
- [ ] Verify performance improvement (1-RTT handshake)
- [ ] Consider 0-RTT for read-only endpoints
- [ ] Update monitoring (track TLS version usage)

For certificate generation and renewal, see `certificate-generation.md` and `automation-patterns.md`.

```

### references/mtls-guide.md

```markdown
# Mutual TLS (mTLS) Implementation Guide

Comprehensive guide to implementing mutual TLS for service-to-service authentication.

## Table of Contents

1. [Understanding mTLS](#understanding-mtls)
2. [Use Cases and Architecture](#use-cases-and-architecture)
3. [mTLS with Nginx](#mtls-with-nginx)
4. [mTLS with Application Code](#mtls-with-application-code)
5. [Certificate Distribution](#certificate-distribution)
6. [Service Mesh Integration](#service-mesh-integration)

## Understanding mTLS

### Standard TLS vs Mutual TLS

**Standard TLS (Server Authentication Only):**
- Client verifies server identity via certificate
- Server does not verify client
- Common for HTTPS websites

**Mutual TLS (Both Authenticate):**
- Both client and server present certificates
- Bidirectional authentication
- Used for service-to-service communication

### mTLS Handshake Flow

```
Client                          Server
  |                               |
  |--- ClientHello -------------->|
  |<-- ServerHello ---------------|
  |<-- Certificate ---------------|  Server proves identity
  |<-- CertificateRequest --------|  Server requests client cert
  |<-- ServerHelloDone ----------|
  |                               |
  |--- Certificate -------------->|  Client proves identity
  |--- ClientKeyExchange -------->|
  |--- CertificateVerify -------->|  Client signs with private key
  |--- ChangeCipherSpec --------->|
  |--- Finished ----------------->|
  |<-- ChangeCipherSpec ----------|
  |<-- Finished ------------------|
  |                               |
  |=== Encrypted Communication ===|
```

## Use Cases and Architecture

### Service-to-Service Authentication

**Microservices architecture:**

```
┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│ API Gateway │◄──mTLS──►│ User Service│◄──mTLS──►│ DB Service  │
└─────────────┘         └─────────────┘         └─────────────┘
                              ▲
                            mTLS
                              ▼
                        ┌─────────────┐
                        │Order Service│
                        └─────────────┘
```

**Benefits:**
- No shared secrets or passwords
- Certificate-based authorization (CN, O, OU fields)
- Automatic rotation via short-lived certificates
- Audit trail (certificate serial numbers in logs)

### Zero-Trust Networks

All services require mTLS, even on internal networks:

```
Principle: Never trust, always verify
├─ Every connection authenticated
├─ Certificate-based service identity
├─ Network location irrelevant
└─ Defense in depth
```

### API Authentication

External partners authenticate via client certificates:

```
Partner System ──[client cert]──► API Gateway ──► Internal Services
```

**Advantages over API keys:**
- Stronger cryptographic authentication
- Non-repudiation (signed requests)
- Automatic expiry (time-bound access)
- Harder to steal (requires private key + certificate)

## mTLS with Nginx

### Server Configuration (Require Client Certificates)

```nginx
server {
    listen 443 ssl;
    server_name api.example.com;

    # Server certificate
    ssl_certificate /etc/ssl/certs/server.crt;
    ssl_certificate_key /etc/ssl/private/server.key;

    # CA certificate to verify client certificates
    ssl_client_certificate /etc/ssl/certs/ca.crt;

    # Require client certificate
    ssl_verify_client on;

    # Certificate verification depth
    ssl_verify_depth 2;

    # TLS protocol and ciphers
    ssl_protocols TLSv1.3 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256';
    ssl_prefer_server_ciphers off;

    # Pass client certificate information to backend
    location / {
        proxy_pass http://backend;

        # Certificate in PEM format
        proxy_set_header X-SSL-Client-Cert $ssl_client_cert;

        # Subject Distinguished Name
        proxy_set_header X-SSL-Client-S-DN $ssl_client_s_dn;

        # Issuer Distinguished Name
        proxy_set_header X-SSL-Client-I-DN $ssl_client_i_dn;

        # Verification status (SUCCESS, FAILED:reason, NONE)
        proxy_set_header X-SSL-Client-Verify $ssl_client_verify;

        # Certificate serial number
        proxy_set_header X-SSL-Client-Serial $ssl_client_serial;
    }
}
```

### Optional Client Certificate

Make client certificate optional, fallback to other authentication:

```nginx
server {
    listen 443 ssl;
    server_name api.example.com;

    ssl_certificate /etc/ssl/certs/server.crt;
    ssl_certificate_key /etc/ssl/private/server.key;
    ssl_client_certificate /etc/ssl/certs/ca.crt;

    # Optional client certificate
    ssl_verify_client optional;
    ssl_verify_depth 2;

    location / {
        # Allow if client cert valid OR has API key
        set $auth_pass 0;

        if ($ssl_client_verify = "SUCCESS") {
            set $auth_pass 1;
        }

        if ($http_x_api_key != "") {
            set $auth_pass 1;
        }

        if ($auth_pass = 0) {
            return 401;
        }

        proxy_pass http://backend;
        proxy_set_header X-SSL-Client-Verify $ssl_client_verify;
        proxy_set_header X-SSL-Client-S-DN $ssl_client_s_dn;
    }
}
```

### Certificate-Based Authorization

Route based on client certificate attributes:

```nginx
# Extract organization from client cert
map $ssl_client_s_dn $client_org {
    ~O=PartnerA $client_org "partnera";
    ~O=PartnerB $client_org "partnerb";
    default "unknown";
}

server {
    listen 443 ssl;
    ssl_verify_client on;
    ssl_client_certificate /etc/ssl/certs/ca.crt;

    location /api/partnera/ {
        if ($client_org != "partnera") {
            return 403;
        }
        proxy_pass http://partnera-backend;
    }

    location /api/partnerb/ {
        if ($client_org != "partnerb") {
            return 403;
        }
        proxy_pass http://partnerb-backend;
    }
}
```

### Testing with curl

**Basic mTLS request:**

```bash
curl https://api.example.com/endpoint \
  --cert client.crt \
  --key client.key \
  --cacert ca.crt
```

**With verbose output:**

```bash
curl -v https://api.example.com/endpoint \
  --cert client.crt \
  --key client.key \
  --cacert ca.crt
```

**Using PKCS#12 format:**

```bash
# Combine cert and key into PKCS#12
openssl pkcs12 -export -out client.p12 \
  -inkey client.key -in client.crt

# Use with curl
curl https://api.example.com/endpoint \
  --cert-type P12 \
  --cert client.p12:password \
  --cacert ca.crt
```

## mTLS with Application Code

### Go Example

```go
package main

import (
    "crypto/tls"
    "crypto/x509"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    // Load CA certificate
    caCert, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }
    caCertPool := x509.NewCertPool()
    caCertPool.AppendCertsFromPEM(caCert)

    // Load client certificate and key
    clientCert, err := tls.LoadX509KeyPair("client.crt", "client.key")
    if err != nil {
        log.Fatal(err)
    }

    // Configure TLS
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{clientCert},
        RootCAs:      caCertPool,
        MinVersion:   tls.VersionTLS13,
    }

    // Create HTTP client with mTLS
    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: tlsConfig,
        },
    }

    // Make request
    resp, err := client.Get("https://api.example.com/endpoint")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    log.Printf("Response status: %s", resp.Status)
}
```

### Python Example

```python
import requests

# Make mTLS request
response = requests.get(
    'https://api.example.com/endpoint',
    cert=('client.crt', 'client.key'),
    verify='ca.crt'
)

print(f"Status: {response.status_code}")
print(f"Response: {response.text}")
```

### Node.js Example

```javascript
const https = require('https');
const fs = require('fs');

const options = {
  hostname: 'api.example.com',
  port: 443,
  path: '/endpoint',
  method: 'GET',
  cert: fs.readFileSync('client.crt'),
  key: fs.readFileSync('client.key'),
  ca: fs.readFileSync('ca.crt')
};

const req = https.request(options, (res) => {
  console.log(`Status: ${res.statusCode}`);
  res.on('data', (d) => {
    process.stdout.write(d);
  });
});

req.on('error', (e) => {
  console.error(e);
});

req.end();
```

## Certificate Distribution

### Pattern 1: Manual Distribution (Small Scale)

Suitable for <10 services:

```bash
# 1. Generate CA
cfssl genkey -initca ca-csr.json | cfssljson -bare ca

# 2. Distribute CA to all services (trust store)
# Services must trust this CA to verify peers

# 3. Generate certificates for each service
for service in api user order; do
  cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
    -config=ca-config.json -profile=peer \
    ${service}-csr.json | cfssljson -bare ${service}
done

# 4. Deploy certificates to services
# service.crt, service.key → /etc/ssl/
```

### Pattern 2: cert-manager (Kubernetes)

Automated certificate issuance for pods:

```yaml
# Internal CA
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: Internal CA
  secretName: internal-ca-key-pair
  duration: 87600h
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer

---
# CA Issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-key-pair

---
# Service certificate (automatic)
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-service-cert
  namespace: production
spec:
  secretName: api-service-tls
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
  - api-service
  - api-service.production.svc.cluster.local
  duration: 2160h  # 90 days
  renewBefore: 720h  # Renew 30 days before expiry
  usages:
  - server auth
  - client auth
```

**Mount certificate in pod:**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      containers:
      - name: api
        volumeMounts:
        - name: tls
          mountPath: /etc/tls
          readOnly: true
      volumes:
      - name: tls
        secret:
          secretName: api-service-tls
```

### Pattern 3: HashiCorp Vault PKI (Dynamic)

Short-lived certificates issued on demand:

```bash
# Service requests certificate on startup
vault write pki_int/issue/service-role \
  common_name="api-service.internal" \
  ttl="24h"

# Vault Agent auto-renews before expiry
# Service reloads certificate without restart
```

**Vault Agent sidecar in Kubernetes:**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-service"
        vault.hashicorp.com/agent-inject-secret-cert: "pki_int/issue/service-role"
        vault.hashicorp.com/agent-inject-template-cert: |
          {{- with secret "pki_int/issue/service-role" "common_name=api-service.internal" "ttl=24h" }}
          {{ .Data.certificate }}
          {{ .Data.ca_chain }}
          {{- end }}
    spec:
      containers:
      - name: api
        # Application reads cert from /vault/secrets/cert
```

### Pattern 4: Service Mesh (Istio/Linkerd)

Automatic mTLS via sidecar proxies:

```yaml
# Istio PeerAuthentication (require mTLS)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
```

**How it works:**
1. Sidecar proxy intercepts all traffic
2. Certificates issued by mesh CA (automatically)
3. mTLS negotiation handled by proxy (transparent to app)
4. Automatic rotation (short TTL, typically 24h)
5. Policy-based access control (which services can communicate)

## Service Mesh Integration

### Istio mTLS

**Enable strict mTLS for namespace:**

```yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
```

**Authorization policy (certificate-based):**

```yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-service-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/production/sa/user-service"]
    to:
    - operation:
        methods: ["GET", "POST"]
```

### Linkerd mTLS

mTLS enabled by default in Linkerd:

```bash
# Install Linkerd
linkerd install | kubectl apply -f -

# Inject sidecar into deployment
kubectl get deploy api-service -o yaml | \
  linkerd inject - | \
  kubectl apply -f -

# Verify mTLS
linkerd viz stat deploy/api-service
# Shows secured/unsecured connection ratio
```

### Consul Connect

```hcl
# Service definition with Connect
service {
  name = "api-service"
  port = 8080

  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "database"
          local_bind_port  = 5432
        }
      }
    }
  }
}
```

## Best Practices

1. **Short-lived certificates**: 24 hours to 90 days maximum
2. **Automatic rotation**: Never rely on manual renewal
3. **Certificate monitoring**: Alert before expiry
4. **Least privilege**: Issue certificates with minimal permissions
5. **Certificate revocation**: Have revocation process ready (CRL, OCSP)
6. **Audit logging**: Log certificate serial numbers for non-repudiation
7. **Separate CAs**: Different CAs for server and client certificates
8. **Service mesh for scale**: Use Istio/Linkerd for 50+ services
9. **Test certificate validation**: Ensure services reject invalid certs
10. **Emergency access**: Have break-glass procedure if certificates fail

## Troubleshooting

### Common mTLS Errors

**"peer did not return a certificate"**
- Client not sending certificate
- Verify client has cert/key files
- Check curl command includes --cert and --key

**"certificate verify failed: unable to get local issuer certificate"**
- CA certificate not in trust store
- Server: Add CA to ssl_client_certificate
- Client: Add CA to --cacert or system trust store

**"tlsv1 alert unknown ca"**
- Server doesn't trust client's CA
- Verify ssl_client_certificate contains correct CA

**"tlsv1 alert certificate required"**
- Server requires client certificate (ssl_verify_client on)
- Client must provide certificate

**"tlsv1 alert access denied"**
- Certificate verification succeeded but authorization failed
- Check Nginx location rules or application logic

### Debug mTLS with OpenSSL

```bash
# Test server mTLS requirement
openssl s_client -connect api.example.com:443

# Verify server requests client certificate
# Look for "Acceptable client certificate CA names"

# Test with client certificate
openssl s_client -connect api.example.com:443 \
  -cert client.crt -key client.key -CAfile ca.crt \
  -showcerts

# Verify handshake successful:
# "Verify return code: 0 (ok)"
```

### Verify Certificate Chain

```bash
# Verify server certificate against CA
openssl verify -CAfile ca.crt server.crt

# Verify client certificate against CA
openssl verify -CAfile ca.crt client.crt

# Check certificate purpose
openssl x509 -in server.crt -noout -purpose
# Should include: "SSL server : Yes"

openssl x509 -in client.crt -noout -purpose
# Should include: "SSL client : Yes"
```

For additional debugging, see `debugging-tls.md`.

```

### references/certificate-generation.md

```markdown
# Certificate Generation Guide

Comprehensive examples for generating TLS certificates across different tools and use cases.

## Table of Contents

1. [Self-Signed Certificates with OpenSSL](#self-signed-certificates-with-openssl)
2. [Internal CA with CFSSL](#internal-ca-with-cfssl)
3. [Local Development with mkcert](#local-development-with-mkcert)
4. [Subject Alternative Names (SANs)](#subject-alternative-names-sans)

## Self-Signed Certificates with OpenSSL

### Quick Single-Line Generation

```bash
# Generate self-signed certificate + key (valid 365 days)
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout server-key.pem -out server-cert.pem \
  -days 365 -subj "/CN=example.com"
```

### Production-Quality with SANs

Create configuration file with Subject Alternative Names:

```bash
cat > san.cnf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[dn]
C = US
ST = California
L = San Francisco
O = Example Corp
OU = IT Department
CN = example.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = api.example.com
IP.1 = 192.168.1.100
EOF
```

Generate certificate:

```bash
# Generate private key
openssl genrsa -out server-key.pem 2048

# Generate CSR
openssl req -new -key server-key.pem -out server.csr -config san.cnf

# Self-sign the certificate
openssl x509 -req -in server.csr -signkey server-key.pem \
  -out server-cert.pem -days 365 -extensions v3_req -extfile san.cnf

# Verify SANs
openssl x509 -in server-cert.pem -noout -text | grep -A 3 "Subject Alternative Name"
```

## Internal CA with CFSSL

### Installation

```bash
# Linux/macOS
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -o /usr/local/bin/cfssl
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -o /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

# Package managers
# Ubuntu: apt install golang-cfssl
# macOS: brew install cfssl
```

### Create CA Configuration

**ca-config.json** - Signing profiles:

```json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "peer": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}
```

**ca-csr.json** - CA certificate request:

```json
{
  "CN": "Example Corp Internal CA",
  "key": {
    "algo": "rsa",
    "size": 4096
  },
  "names": [
    {
      "C": "US",
      "ST": "California",
      "L": "San Francisco",
      "O": "Example Corp",
      "OU": "IT Security"
    }
  ],
  "ca": {
    "expiry": "87600h"
  }
}
```

### Generate CA Certificate

```bash
cfssl genkey -initca ca-csr.json | cfssljson -bare ca
# Produces: ca.pem (cert), ca-key.pem (private key), ca.csr
```

### Generate Server Certificate

**server-csr.json**:

```json
{
  "CN": "api.example.com",
  "hosts": [
    "api.example.com",
    "api.internal.example.com",
    "192.168.1.100"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "ST": "California",
      "L": "San Francisco",
      "O": "Example Corp",
      "OU": "Engineering"
    }
  ]
}
```

Generate and sign:

```bash
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=server \
  server-csr.json | cfssljson -bare server
# Produces: server.pem (cert), server-key.pem (private key)
```

### Generate Client Certificate (mTLS)

**client-csr.json**:

```json
{
  "CN": "client.example.com",
  "hosts": [""],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "ST": "California",
      "O": "Example Corp",
      "OU": "Engineering"
    }
  ]
}
```

Generate and sign:

```bash
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
  -config=ca-config.json -profile=client \
  client-csr.json | cfssljson -bare client
# Produces: client.pem (cert), client-key.pem (private key)
```

## Local Development with mkcert

### Installation

```bash
# macOS
brew install mkcert
brew install nss  # For Firefox support

# Linux (Debian/Ubuntu)
sudo apt install libnss3-tools
wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.4/mkcert-v1.4.4-linux-amd64
sudo mv mkcert-v1.4.4-linux-amd64 /usr/local/bin/mkcert
sudo chmod +x /usr/local/bin/mkcert

# Windows (via Chocolatey)
choco install mkcert
```

### Generate Trusted Local Certificates

```bash
# Install local CA in system trust store
mkcert -install

# Generate certificate for local domains
mkcert example.com "*.example.com" localhost 127.0.0.1 ::1
# Produces: example.com+4.pem (cert) and example.com+4-key.pem (key)
# Already trusted by browser and system!

# Use with web server
nginx -c nginx.conf  # Reference the generated .pem files
```

### View CA Location

```bash
# Show CA certificate location
mkcert -CAROOT
# Example: /Users/username/Library/Application Support/mkcert
```

## Subject Alternative Names (SANs)

### Why SANs Matter

Modern browsers ignore the Common Name (CN) field and require SANs. Without SANs, certificate validation fails even if CN matches.

### SAN Configuration Examples

**For multiple domains:**

```ini
[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = api.example.com
DNS.4 = app.example.com
```

**For wildcard + specific domains:**

```ini
[alt_names]
DNS.1 = example.com
DNS.2 = *.example.com
DNS.3 = *.internal.example.com
```

**For IP addresses:**

```ini
[alt_names]
IP.1 = 192.168.1.100
IP.2 = 10.0.0.5
```

**Combined (domains + IPs):**

```ini
[alt_names]
DNS.1 = example.com
DNS.2 = localhost
IP.1 = 127.0.0.1
IP.2 = ::1
```

### Verify SANs in Certificate

```bash
# View SANs
openssl x509 -in cert.pem -noout -text | grep -A 3 "Subject Alternative Name"

# Or with verbose output
openssl x509 -in cert.pem -noout -text | grep -A 10 "X509v3 extensions"
```

## Certificate Verification

### Verify Certificate Contents

```bash
# View all certificate details
openssl x509 -in cert.pem -noout -text

# Check expiration dates
openssl x509 -in cert.pem -noout -dates

# Check issuer
openssl x509 -in cert.pem -noout -issuer

# Check subject
openssl x509 -in cert.pem -noout -subject

# Check fingerprint
openssl x509 -in cert.pem -noout -fingerprint -sha256
```

### Verify Certificate Chain

```bash
# Verify against CA
openssl verify -CAfile ca.crt cert.pem

# Verify with intermediate certificates
openssl verify -CAfile root-ca.crt -untrusted intermediate.crt cert.pem
```

### Verify Key and Certificate Match

```bash
# Certificate modulus
cert_modulus=$(openssl x509 -in cert.pem -noout -modulus | md5sum)

# Key modulus
key_modulus=$(openssl rsa -in key.pem -noout -modulus | md5sum)

# Compare (must match)
if [ "$cert_modulus" = "$key_modulus" ]; then
  echo "Certificate and key match"
else
  echo "Mismatch - cert and key do not pair"
fi
```

## Advanced: Certificate Signing Request (CSR)

### Generate CSR for Commercial CA

```bash
# Generate private key
openssl genrsa -out server-key.pem 2048

# Create CSR config
cat > csr.cnf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[dn]
C = US
ST = California
L = San Francisco
O = Example Corp
OU = IT Department
CN = example.com
emailAddress = [email protected]

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
EOF

# Generate CSR
openssl req -new -key server-key.pem -out server.csr -config csr.cnf

# Verify CSR
openssl req -in server.csr -noout -text

# Submit CSR to CA (DigiCert, GlobalSign, etc.)
# CA will return signed certificate
```

## Best Practices

1. **Key Size**: Use RSA 2048-bit minimum (4096-bit for CAs)
2. **Validity Period**:
   - Development: 1-5 years
   - Production public: 90 days (Let's Encrypt standard)
   - Internal: 1-2 years
3. **Always Include SANs**: Required by modern browsers
4. **Protect Private Keys**:
   - File permissions: 600 (owner read/write only)
   - Never commit to version control
   - Use secret management tools (Vault, Sealed Secrets)
5. **Certificate Transparency**: Public certificates are logged
6. **Hash Algorithm**: Use SHA-256 minimum (SHA-1 deprecated)

## Troubleshooting

### Common Generation Errors

**"unable to write random state"**
- Solution: Ensure write permissions to home directory

**"No subject given"**
- Solution: Provide -subj flag or configuration file

**"req_extensions not found"**
- Solution: Verify [v3_req] section in config file

**"SANs not present in certificate"**
- Solution: Ensure -extensions v3_req -extfile san.cnf in signing command

For runtime certificate issues, see `debugging-tls.md`.

```

### references/automation-patterns.md

```markdown
# TLS Automation Patterns

Comprehensive guide to automating certificate lifecycle management across different platforms.

## Table of Contents

1. [Let's Encrypt with Certbot](#lets-encrypt-with-certbot)
2. [cert-manager in Kubernetes](#cert-manager-in-kubernetes)
3. [HashiCorp Vault PKI](#hashicorp-vault-pki)
4. [Renewal Strategies](#renewal-strategies)

## Let's Encrypt with Certbot

### Installation

```bash
# Ubuntu/Debian
sudo apt update
sudo apt install certbot

# RHEL/CentOS/Fedora
sudo yum install certbot
# or
sudo dnf install certbot

# macOS
brew install certbot

# Verify installation
certbot --version
```

### Standalone Mode

Requires port 80 to be free (no web server running):

```bash
# Obtain certificate
sudo certbot certonly --standalone \
  -d example.com \
  -d www.example.com \
  --email [email protected] \
  --agree-tos \
  --no-eff-email

# Certificates saved to:
# /etc/letsencrypt/live/example.com/fullchain.pem
# /etc/letsencrypt/live/example.com/privkey.pem
# /etc/letsencrypt/live/example.com/chain.pem
# /etc/letsencrypt/live/example.com/cert.pem
```

### Webroot Mode

Web server keeps running, validation files served from document root:

```bash
# Obtain certificate
sudo certbot certonly --webroot \
  -w /var/www/html \
  -d example.com \
  -d www.example.com \
  --email [email protected] \
  --agree-tos

# Certbot writes challenge files to:
# /var/www/html/.well-known/acme-challenge/
```

### Nginx Plugin

Automatically configures Nginx:

```bash
# Install nginx plugin
sudo apt install python3-certbot-nginx

# Obtain certificate AND configure Nginx
sudo certbot --nginx -d example.com -d www.example.com

# Certbot modifies nginx config automatically
# Adds ssl_certificate and ssl_certificate_key directives
# Creates redirect from HTTP to HTTPS
```

### Apache Plugin

Automatically configures Apache:

```bash
# Install apache plugin
sudo apt install python3-certbot-apache

# Obtain certificate AND configure Apache
sudo certbot --apache -d example.com -d www.example.com
```

### DNS Challenge (Wildcard Support)

Required for wildcard certificates:

```bash
# Manual DNS challenge
sudo certbot certonly --manual \
  --preferred-challenges dns \
  -d example.com \
  -d "*.example.com"

# Certbot prompts to add TXT record:
# _acme-challenge.example.com TXT "abc123..."

# Verify DNS propagation before continuing:
dig _acme-challenge.example.com TXT
```

**Automated DNS with plugins:**

```bash
# Cloudflare
sudo apt install python3-certbot-dns-cloudflare
cat > ~/.secrets/cloudflare.ini <<EOF
dns_cloudflare_api_token = YOUR_API_TOKEN
EOF
chmod 600 ~/.secrets/cloudflare.ini

sudo certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials ~/.secrets/cloudflare.ini \
  -d example.com -d "*.example.com"

# Other DNS providers: Route53, Google Cloud DNS, Digital Ocean, etc.
# See: https://eff-certbot.readthedocs.io/en/stable/using.html#dns-plugins
```

### Automatic Renewal

**Cron job (traditional):**

```bash
# Certbot package usually installs cron job automatically
# Check: cat /etc/cron.d/certbot

# Manual cron entry (runs twice daily):
0 0,12 * * * root certbot renew --quiet

# Test renewal without actually renewing:
sudo certbot renew --dry-run
```

**Systemd timer (modern systems):**

```bash
# Check timer status
systemctl status certbot.timer

# Enable timer
sudo systemctl enable certbot.timer
sudo systemctl start certbot.timer

# View timer schedule
systemctl list-timers certbot.timer
```

**Post-renewal hooks:**

```bash
# Reload web server after renewal
sudo certbot renew --deploy-hook "systemctl reload nginx"

# Or create hook script
sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy
sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'EOF'
#!/bin/bash
systemctl reload nginx
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
```

### Staging Environment

Test with Let's Encrypt staging to avoid rate limits:

```bash
# Use staging server
sudo certbot certonly --standalone \
  --staging \
  -d example.com \
  -d www.example.com

# Staging certificates are NOT trusted by browsers
# Use for testing only
```

**Rate Limits:**
- 50 certificates per registered domain per week
- 5 duplicate certificates per week
- Staging has much higher limits

## cert-manager in Kubernetes

### Installation via Helm

```bash
# Add Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install cert-manager with CRDs
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.0 \
  --set installCRDs=true

# Verify installation
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager
```

### ClusterIssuer (Let's Encrypt)

**Production issuer:**

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    # HTTP-01 solver
    - http01:
        ingress:
          class: nginx
```

**Staging issuer (for testing):**

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
```

**DNS-01 solver (wildcard support):**

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
        - example.com
```

### Ingress with Automatic Certificates

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - example.com
    - www.example.com
    secretName: example-com-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80
```

cert-manager automatically:
1. Detects Ingress with cert-manager.io/cluster-issuer annotation
2. Creates Certificate resource
3. Solves ACME challenge (HTTP-01)
4. Stores certificate in Secret (example-com-tls)
5. Renews before expiry

### Manual Certificate Resource

For non-Ingress use cases:

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-cert
  namespace: production
spec:
  secretName: api-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - api.example.com
  - api.internal.example.com
  duration: 2160h  # 90 days
  renewBefore: 720h  # 30 days before expiry
```

### Internal CA with cert-manager

**Create self-signed CA:**

```yaml
# Self-signed issuer (bootstrap)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

---
# CA certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: Internal CA
  secretName: internal-ca-key-pair
  duration: 87600h  # 10 years
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer

---
# CA issuer (uses CA cert above)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-key-pair
```

**Issue certificates from internal CA:**

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: service-cert
  namespace: production
spec:
  secretName: service-tls
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
  - service.production.svc.cluster.local
  duration: 8760h  # 1 year
  renewBefore: 2160h  # 90 days
  usages:
  - server auth
  - client auth
```

### Monitor Certificates

```bash
# List all certificates
kubectl get certificate -A

# Check certificate status
kubectl describe certificate example-com-cert -n production

# Check certificate request
kubectl get certificaterequest -A

# View certificate secret
kubectl get secret example-com-tls -n production -o yaml

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager
```

## HashiCorp Vault PKI

### Enable PKI Secrets Engine

```bash
# Enable PKI at pki/
vault secrets enable pki

# Set max TTL to 10 years
vault secrets tune -max-lease-ttl=87600h pki

# Enable intermediate CA (best practice)
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int
```

### Generate Root CA

```bash
# Generate internal root CA
vault write -field=certificate pki/root/generate/internal \
  common_name="Example Corp Root CA" \
  issuer_name="root-2025" \
  ttl=87600h > root_ca.crt

# Configure CA and CRL URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com:8200/v1/pki/crl"
```

### Generate Intermediate CA

```bash
# Generate intermediate CSR
vault write -format=json pki_int/intermediate/generate/internal \
  common_name="Example Corp Intermediate CA" \
  issuer_name="example-intermediate" \
  | jq -r '.data.csr' > pki_intermediate.csr

# Sign intermediate with root CA
vault write -format=json pki/root/sign-intermediate \
  issuer_ref="root-2025" \
  csr=@pki_intermediate.csr \
  format=pem_bundle \
  ttl="43800h" \
  | jq -r '.data.certificate' > intermediate.cert.pem

# Import signed intermediate
vault write pki_int/intermediate/set-signed [email protected]
```

### Create Role for Certificate Issuance

```bash
# Create role for web servers
vault write pki_int/roles/example-dot-com \
  allowed_domains="example.com" \
  allow_subdomains=true \
  max_ttl="720h" \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ServerAuth"

# Create role for client certificates (mTLS)
vault write pki_int/roles/client-cert \
  allowed_domains="example.com" \
  allow_subdomains=true \
  max_ttl="24h" \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ClientAuth"
```

### Issue Certificate

```bash
# Issue server certificate
vault write pki_int/issue/example-dot-com \
  common_name="api.example.com" \
  ttl="24h"

# Returns: certificate, private_key, ca_chain, serial_number

# Issue with explicit SANs
vault write pki_int/issue/example-dot-com \
  common_name="api.example.com" \
  alt_names="api.internal.example.com,api.prod.example.com" \
  ip_sans="192.168.1.100" \
  ttl="24h"
```

### Vault Agent Auto-Renewal

**vault-agent.hcl:**

```hcl
pid_file = "/var/run/vault-agent.pid"

vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method {
    type = "kubernetes"
    config = {
      role = "web-service"
    }
  }

  sink {
    type = "file"
    config = {
      path = "/var/run/secrets/vault-token"
    }
  }
}

template {
  source      = "/etc/vault/cert.tpl"
  destination = "/etc/ssl/certs/server.crt"
  command     = "systemctl reload nginx"
}

template {
  source      = "/etc/vault/key.tpl"
  destination = "/etc/ssl/private/server.key"
  perms       = 0600
  command     = "systemctl reload nginx"
}
```

**cert.tpl:**

```
{{- with secret "pki_int/issue/example-dot-com" "common_name=api.example.com" "ttl=24h" }}
{{ .Data.certificate }}
{{ .Data.ca_chain }}
{{- end }}
```

**key.tpl:**

```
{{- with secret "pki_int/issue/example-dot-com" "common_name=api.example.com" "ttl=24h" }}
{{ .Data.private_key }}
{{- end }}
```

### Revoke Certificate

```bash
# Revoke by serial number
vault write pki_int/revoke serial_number="39:dd:2e:90:b7:23..."

# Check CRL
curl https://vault.example.com:8200/v1/pki_int/crl | openssl crl -inform DER -text
```

## Renewal Strategies

### Renewal Timing

**Best practice:** Renew at 2/3 of certificate lifetime

| Validity | Renew After | Example |
|----------|-------------|---------|
| 90 days | 60 days | Let's Encrypt (30 days before expiry) |
| 30 days | 20 days | Short-lived internal certs |
| 1 year | 8 months | Long-lived internal certs |
| 24 hours | 16 hours | Dynamic secrets (Vault) |

### Zero-Downtime Renewal

**Load new certificate without restart:**

```bash
# Nginx (reload config)
nginx -s reload

# Apache
apachectl graceful

# HAProxy
systemctl reload haproxy
```

**Kubernetes rolling restart:**

```bash
# Update Secret (cert-manager does this automatically)
kubectl create secret tls example-tls \
  --cert=new-cert.pem --key=new-key.pem \
  --dry-run=client -o yaml | kubectl apply -f -

# Rolling restart deployment
kubectl rollout restart deployment web-app -n production
```

### Monitoring Certificate Expiry

**Check expiry with OpenSSL:**

```bash
# Local certificate
openssl x509 -in cert.pem -noout -enddate

# Remote server
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -enddate

# Check if expires within 30 days
openssl x509 -in cert.pem -noout -checkend 2592000
# Exit 0: valid for 30+ days, Exit 1: expires within 30 days
```

**Prometheus monitoring:**

```yaml
# blackbox_exporter config
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4

# Prometheus scrape config
scrape_configs:
  - job_name: 'ssl_expiry'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter:9115

# Alert rule
groups:
  - name: ssl_expiry
    rules:
    - alert: SSLCertExpiring
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
      for: 1h
      annotations:
        summary: "SSL certificate expiring in <7 days: {{ $labels.instance }}"
```

### Renewal Failure Handling

**Certbot renewal failure:**

```bash
# Check renewal logs
cat /var/log/letsencrypt/letsencrypt.log

# Manual renewal attempt
sudo certbot renew --force-renewal -d example.com

# Common issues:
# - Port 80 blocked (firewall)
# - DNS not pointing to server
# - Rate limit exceeded (use staging)
```

**cert-manager renewal failure:**

```bash
# Check Certificate status
kubectl describe certificate example-cert -n production

# Common issues in status:
# - ACME challenge failed (Ingress misconfigured)
# - DNS propagation timeout (DNS-01)
# - Rate limit (use staging ClusterIssuer)

# Check CertificateRequest
kubectl get certificaterequest -n production
kubectl describe certificaterequest <name> -n production

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager --tail=100
```

## Best Practices

1. **Use staging environment** for testing (avoid rate limits)
2. **Automate from day one** - manual renewal will be forgotten
3. **Monitor expiry** with alerting (7-30 days before expiry)
4. **Test renewal process** regularly (dry-run)
5. **Short-lived certificates** are more secure (90 days or less)
6. **Separate concerns**: Certificate automation ≠ key storage (use secret management)
7. **Log renewal events** for audit trail
8. **Plan for failure**: What happens if renewal fails?

```

### references/debugging-tls.md

```markdown
# TLS Debugging and Troubleshooting Guide

Comprehensive guide for debugging TLS issues and common certificate problems.

## Table of Contents

1. [Essential Debugging Commands](#essential-debugging-commands)
2. [Common TLS Errors](#common-tls-errors)
3. [Certificate Validation](#certificate-validation)
4. [TLS Handshake Analysis](#tls-handshake-analysis)
5. [Monitoring Certificate Expiry](#monitoring-certificate-expiry)

## Essential Debugging Commands

### OpenSSL s_client

**Basic connection test:**

```bash
# Test TLS connection
openssl s_client -connect example.com:443

# Key indicators in output:
# - "Verify return code: 0 (ok)" = certificate valid
# - "Verify return code: 20" = unable to get local issuer certificate
# - "Verify return code: 10" = certificate has expired
```

**Show certificate chain:**

```bash
openssl s_client -connect example.com:443 -showcerts

# Displays full certificate chain:
# - Server certificate (first)
# - Intermediate certificates
# - Root CA (usually not included)
```

**Test specific TLS version:**

```bash
# TLS 1.3
openssl s_client -connect example.com:443 -tls1_3

# TLS 1.2
openssl s_client -connect example.com:443 -tls1_2

# TLS 1.1 (should fail on modern servers)
openssl s_client -connect example.com:443 -tls1_1
```

**Test with SNI (Server Name Indication):**

```bash
# Required for shared hosting / multiple domains
openssl s_client -connect example.com:443 -servername example.com

# Without SNI, server may present wrong certificate
```

**Test specific cipher suite:**

```bash
# Test if server supports specific cipher
openssl s_client -connect example.com:443 \
  -cipher 'ECDHE-RSA-AES256-GCM-SHA384'

# Test TLS 1.3 cipher
openssl s_client -connect example.com:443 \
  -tls1_3 -ciphersuites 'TLS_AES_256_GCM_SHA384'
```

**Test mTLS (client certificate):**

```bash
openssl s_client -connect api.example.com:443 \
  -cert client.crt \
  -key client.key \
  -CAfile ca.crt

# Verify output includes:
# "Acceptable client certificate CA names" (server requests client cert)
# "Verify return code: 0 (ok)"
```

**Save server certificate:**

```bash
# Extract and save certificate
openssl s_client -connect example.com:443 </dev/null 2>/dev/null | \
  openssl x509 -out server.crt

# Verify saved certificate
openssl x509 -in server.crt -noout -text
```

### curl for TLS Testing

**Verbose TLS handshake:**

```bash
curl -v https://example.com

# Look for:
# * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
# * Server certificate:
# *  subject: CN=example.com
# *  issuer: C=US; O=Let's Encrypt; CN=R3
```

**Very verbose (show all TLS details):**

```bash
curl -vvv https://example.com 2>&1 | grep -E "SSL|TLS"
```

**Test with specific CA:**

```bash
curl --cacert ca.crt https://example.com
```

**Test mTLS:**

```bash
curl --cert client.crt --key client.key --cacert ca.crt \
  https://api.example.com/endpoint
```

**Ignore certificate errors (INSECURE - testing only):**

```bash
curl -k https://example.com
# or
curl --insecure https://example.com
```

### Certificate Examination

**View certificate details:**

```bash
# Full certificate details
openssl x509 -in cert.pem -noout -text

# Specific fields only
openssl x509 -in cert.pem -noout -subject
openssl x509 -in cert.pem -noout -issuer
openssl x509 -in cert.pem -noout -dates
openssl x509 -in cert.pem -noout -serial
openssl x509 -in cert.pem -noout -fingerprint -sha256
```

**Check Subject Alternative Names (SANs):**

```bash
openssl x509 -in cert.pem -noout -text | grep -A 1 "Subject Alternative Name"

# Example output:
# X509v3 Subject Alternative Name:
#     DNS:example.com, DNS:www.example.com, IP Address:192.168.1.100
```

**Check certificate purpose:**

```bash
openssl x509 -in cert.pem -noout -purpose

# Shows allowed purposes:
# - SSL client : Yes/No
# - SSL server : Yes/No
# - S/MIME signing : Yes/No
```

**Check certificate expiration:**

```bash
# Show dates
openssl x509 -in cert.pem -noout -dates
# Output:
# notBefore=Jan  1 00:00:00 2025 GMT
# notAfter=Apr  1 00:00:00 2025 GMT

# Check if expired
openssl x509 -in cert.pem -noout -checkend 0
# Exit 0: not expired, Exit 1: expired

# Check if expires within 30 days (2592000 seconds)
openssl x509 -in cert.pem -noout -checkend 2592000
```

## Common TLS Errors

### Error: "certificate has expired"

**Cause:** Certificate validity period has passed.

**Diagnosis:**

```bash
# Check expiration date
openssl x509 -in cert.pem -noout -enddate

# Check system clock
date
timedatectl  # systemd systems
```

**Solution:**

```bash
# 1. Renew certificate
certbot renew  # Let's Encrypt
# or regenerate self-signed certificate

# 2. Verify system clock is correct
sudo ntpdate pool.ntp.org
# or
sudo timedatectl set-ntp true
```

### Error: "unable to get local issuer certificate"

**Cause:** CA certificate not in trust store.

**Diagnosis:**

```bash
# Check certificate chain
openssl s_client -connect example.com:443 -showcerts

# Verify issuer
openssl x509 -in cert.pem -noout -issuer
```

**Solution:**

Add CA certificate to system trust store:

```bash
# Ubuntu/Debian
sudo cp ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# RHEL/CentOS/Fedora
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Or specify CA in application
curl --cacert ca.crt https://example.com
openssl s_client -CAfile ca.crt -connect example.com:443
```

### Error: "certificate verify failed: Hostname mismatch"

**Cause:** Certificate CN/SAN doesn't match requested hostname.

**Diagnosis:**

```bash
# Check certificate SANs
openssl x509 -in cert.pem -noout -text | grep -A 3 "Subject Alternative Name"

# Check what hostname you're requesting
curl -v https://example.com  # Look at "Host:" header
```

**Solution:**

```bash
# 1. Regenerate certificate with correct SANs
# See certificate-generation.md

# 2. Or use correct hostname
curl https://correct-hostname.com  # Match certificate

# 3. For testing with IP (if cert has IP in SAN)
curl --resolve example.com:443:192.168.1.100 https://example.com
```

### Error: "SSL handshake failed: sslv3 alert handshake failure"

**Cause:** TLS version or cipher suite mismatch.

**Diagnosis:**

```bash
# Test TLS versions
openssl s_client -connect example.com:443 -tls1_3
openssl s_client -connect example.com:443 -tls1_2
openssl s_client -connect example.com:443 -tls1_1

# Test cipher support
openssl s_client -connect example.com:443 \
  -cipher 'ECDHE-RSA-AES256-GCM-SHA384'
```

**Solution:**

Update configuration to support TLS 1.2+ and strong ciphers:

```nginx
# Nginx
ssl_protocols TLSv1.3 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256';
```

### Error: "certificate signed by unknown authority"

**Cause:** Intermediate certificates missing from chain.

**Diagnosis:**

```bash
# Check what server sends
openssl s_client -connect example.com:443 -showcerts

# Verify chain locally
openssl verify -CAfile ca.crt cert.pem
# If this fails, chain is incomplete
```

**Solution:**

Include full certificate chain in server configuration:

```bash
# Build full chain
cat cert.pem intermediate.pem root.pem > fullchain.pem

# Configure server
# Nginx
ssl_certificate /etc/ssl/certs/fullchain.pem;

# Apache
SSLCertificateFile /etc/ssl/certs/fullchain.pem
```

### Error: "tlsv1 alert unknown ca" (mTLS)

**Cause:** Server doesn't trust client certificate's CA.

**Diagnosis:**

```bash
# Verify client certificate issuer
openssl x509 -in client.crt -noout -issuer

# Check server's trusted CAs
# Nginx: ssl_client_certificate directive
# Check file contains correct CA
openssl x509 -in /etc/ssl/certs/ca.crt -noout -subject
```

**Solution:**

Add client CA to server trust store:

```nginx
# Nginx
ssl_client_certificate /etc/ssl/certs/client-ca.crt;
ssl_verify_client on;
```

### Error: "peer did not return a certificate" (mTLS)

**Cause:** Client not sending certificate when server requires it.

**Diagnosis:**

```bash
# Verify server requests client cert
openssl s_client -connect api.example.com:443
# Look for "Acceptable client certificate CA names"
```

**Solution:**

```bash
# Client must provide certificate
curl --cert client.crt --key client.key https://api.example.com

# Or in application code (Go example)
clientCert, _ := tls.LoadX509KeyPair("client.crt", "client.key")
tlsConfig := &tls.Config{Certificates: []tls.Certificate{clientCert}}
```

## Certificate Validation

### Verify Certificate Chain

```bash
# Verify certificate against CA
openssl verify -CAfile ca.crt cert.pem

# Verify with intermediate certificates
openssl verify -CAfile root-ca.crt -untrusted intermediate.crt cert.pem

# Verify full chain
cat intermediate.crt root.crt > chain.pem
openssl verify -CAfile chain.pem cert.pem
```

### Verify Private Key and Certificate Match

```bash
# Extract modulus from certificate
cert_modulus=$(openssl x509 -in cert.pem -noout -modulus | openssl md5)
echo "Certificate modulus: $cert_modulus"

# Extract modulus from private key
key_modulus=$(openssl rsa -in key.pem -noout -modulus | openssl md5)
echo "Private key modulus: $key_modulus"

# Compare (must match)
if [ "$cert_modulus" = "$key_modulus" ]; then
    echo "✓ Certificate and key match"
else
    echo "✗ Certificate and key DO NOT match"
fi
```

### Verify Certificate Purpose

```bash
# Check Extended Key Usage
openssl x509 -in cert.pem -noout -text | grep -A 1 "Extended Key Usage"

# Server certificates should have:
# TLS Web Server Authentication

# Client certificates should have:
# TLS Web Client Authentication
```

## TLS Handshake Analysis

### Capture TLS Handshake with tcpdump

```bash
# Capture traffic on port 443
sudo tcpdump -i any -w capture.pcap port 443

# Analyze with Wireshark or tshark
tshark -r capture.pcap -Y tls.handshake

# View certificate in handshake
tshark -r capture.pcap -Y tls.handshake.certificate -V
```

### Analyze TLS with ssllabs

```bash
# Online SSL/TLS testing
# Visit: https://www.ssllabs.com/ssltest/

# Or use testssl.sh (command-line)
git clone https://github.com/drwetter/testssl.sh.git
cd testssl.sh
./testssl.sh example.com
```

### Test OCSP Stapling

```bash
# Test if server supports OCSP stapling
openssl s_client -connect example.com:443 -status

# Look for "OCSP Response Status: successful"
```

## Monitoring Certificate Expiry

### Check Expiry for Single Domain

```bash
# Extract expiry date
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -enddate

# Human-readable time remaining
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -enddate | \
  awk -F'=' '{print $2}' | \
  xargs -I {} date -d {} +%s | \
  awk '{print "Days remaining: " int(($1 - systime()) / 86400)}'
```

### Check Multiple Domains

```bash
# domains.txt contains one domain per line
cat domains.txt | while read domain; do
    expiry=$(echo | openssl s_client -connect $domain:443 2>/dev/null | \
      openssl x509 -noout -enddate | cut -d= -f2)
    echo "$domain expires: $expiry"
done
```

### Prometheus Monitoring

**blackbox_exporter configuration:**

```yaml
# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      fail_if_ssl: false
      fail_if_not_ssl: true
```

**Prometheus scrape config:**

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'ssl_expiry'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115
```

**Alert rule:**

```yaml
# alerts.yml
groups:
  - name: ssl_expiry
    rules:
    - alert: SSLCertExpiringSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate expiring in <7 days"
        description: "{{ $labels.instance }} certificate expires in {{ $value | humanizeDuration }}"

    - alert: SSLCertExpired
      expr: probe_ssl_earliest_cert_expiry - time() <= 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "SSL certificate expired"
        description: "{{ $labels.instance }} certificate has expired"
```

### Nagios/Icinga Check

```bash
#!/bin/bash
# check_ssl_cert.sh

DOMAIN=$1
WARN_DAYS=30
CRIT_DAYS=7

# Get expiry date
EXPIRY=$(echo | openssl s_client -connect $DOMAIN:443 2>/dev/null | \
         openssl x509 -noout -enddate | cut -d= -f2)

# Convert to epoch
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( ($EXPIRY_EPOCH - $NOW_EPOCH) / 86400 ))

# Check thresholds
if [ $DAYS_LEFT -le $CRIT_DAYS ]; then
    echo "CRITICAL: Certificate expires in $DAYS_LEFT days"
    exit 2
elif [ $DAYS_LEFT -le $WARN_DAYS ]; then
    echo "WARNING: Certificate expires in $DAYS_LEFT days"
    exit 1
else
    echo "OK: Certificate expires in $DAYS_LEFT days"
    exit 0
fi
```

## Advanced Debugging

### Debug TLS with strace

```bash
# Trace openssl command
strace -e trace=read,write,open,connect \
  openssl s_client -connect example.com:443

# Trace application
strace -e trace=network,read,write -p PID
```

### Enable TLS Debug Logging

**Nginx:**

```nginx
error_log /var/log/nginx/error.log debug;

# Generates verbose TLS handshake logs
```

**Apache:**

```apache
LogLevel ssl:trace6

# Logs to error_log with detailed TLS info
```

**Go application:**

```go
// Set GODEBUG environment variable
// GODEBUG=x509roots=1 ./app
// Shows certificate verification details
```

### Test Certificate Revocation

**Check CRL (Certificate Revocation List):**

```bash
# Extract CRL URL from certificate
openssl x509 -in cert.pem -noout -text | grep -A 4 "CRL Distribution"

# Download and check CRL
wget http://crl.example.com/ca.crl
openssl crl -in ca.crl -inform DER -text

# Check if certificate is revoked
openssl crl -in ca.crl -inform DER -noout -text | grep -A 2 "Serial Number: ABC123"
```

**Check OCSP (Online Certificate Status Protocol):**

```bash
# Extract OCSP URL
openssl x509 -in cert.pem -noout -ocsp_uri

# Check OCSP status
openssl ocsp -issuer ca.crt -cert cert.pem \
  -url http://ocsp.example.com -resp_text
```

## Troubleshooting Checklist

When debugging TLS issues, check:

- [ ] Certificate not expired (`openssl x509 -noout -dates`)
- [ ] Hostname matches certificate SANs
- [ ] Certificate chain complete (intermediate certs included)
- [ ] CA certificate in trust store
- [ ] Private key matches certificate
- [ ] TLS 1.2 or 1.3 enabled
- [ ] Strong cipher suites configured
- [ ] Firewall allows port 443
- [ ] SNI configured (shared hosting)
- [ ] Certificate not revoked (CRL/OCSP)
- [ ] System clock correct
- [ ] File permissions correct (key: 600, cert: 644)

For certificate generation issues, see `certificate-generation.md`.
For mTLS-specific debugging, see `mtls-guide.md`.

```