managing-secrets

Original source / Raw SKILL.md

---
name: managing-secrets
description: Managing secrets (API keys, database credentials, certificates) with Vault, cloud providers, and Kubernetes. Use when storing sensitive data, rotating credentials, syncing secrets to Kubernetes, implementing dynamic secrets, or scanning code for leaked secrets.
---

# Managing Secrets

Secure storage, rotation, and delivery of secrets (API keys, database credentials, TLS certificates) for applications and infrastructure.

## When to Use This Skill

Use when:
- Storing API keys, database credentials, or encryption keys
- Implementing secret rotation (manual or automatic)
- Syncing secrets from external stores to Kubernetes
- Setting up dynamic secrets (database, cloud providers)
- Scanning code for leaked secrets
- Implementing zero-knowledge patterns
- Meeting compliance requirements (SOC 2, ISO 27001, PCI DSS)

## Quick Decision Frameworks

### Framework 1: Choosing a Secret Store

| Scenario | Primary Choice | Alternative |
|----------|----------------|-------------|
| Kubernetes + Multi-Cloud | Vault + ESO | Cloud Secret Manager + ESO |
| Kubernetes + Single Cloud | Cloud Secret Manager + ESO | Vault + ESO |
| Serverless (AWS Lambda) | AWS Secrets Manager | AWS Parameter Store |
| Multi-Cloud Enterprise | HashiCorp Vault | Doppler (SaaS) |
| Small Team (<10 apps) | Doppler, Infisical | 1Password Secrets Automation |
| GitOps-Centric | SOPS (git-encrypted) | Sealed Secrets (K8s-only) |

**Decision Tree:**
- Kubernetes? → External Secrets Operator (ESO) with chosen backend
- Single cloud? → Cloud-native (AWS/GCP/Azure)
- Multi-cloud/on-prem? → HashiCorp Vault
- GitOps? → SOPS or Sealed Secrets

### Framework 2: Static vs. Dynamic Secrets

| Secret Type | Use Dynamic? | TTL | Solution |
|-------------|-------------|-----|----------|
| Database credentials | YES | 1 hour | Vault DB engine |
| Cloud IAM (AWS/GCP) | YES | 15 min | Vault cloud engine |
| SSH/RDP access | YES | 5 min | Vault SSH engine |
| TLS certificates | YES | 24 hours | Vault PKI / cert-manager |
| Third-party API keys | NO | Quarterly | Vault KV v2 (manual rotation) |

### Framework 3: Kubernetes Secret Delivery

| Method | Use Case | Rotation | Restart Required |
|--------|----------|----------|------------------|
| **External Secrets Operator** | Static secrets, periodic sync | Polling (1h) | Yes |
| **Secrets Store CSI Driver** | File-based, watch rotation | inotify | No |
| **Vault Secrets Operator** | Vault-specific, dynamic | Automatic renewal | Optional |

## HashiCorp Vault Fundamentals

### Core Components

- **Secrets Engines**: KV v2 (static), Database (dynamic), AWS, PKI, SSH
- **Auth Methods**: Kubernetes, JWT/OIDC, AppRole, LDAP
- **Policies**: HCL-based access control (least privilege)
- **Leases**: TTL for secrets, auto-renewal, auto-revocation

### Static Secrets (KV v2)

```bash
# Create secret
vault kv put secret/myapp/config api_key=sk_live_EXAMPLE

# Read secret
vault kv get secret/myapp/config

# List versions
vault kv metadata get secret/myapp/config
```

### Dynamic Database Credentials

```bash
# Configure PostgreSQL
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb"

# Create role
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\"..." \
  default_ttl="1h"

# Generate credentials
vault read database/creds/app-role
```

For detailed Vault architecture, see `references/vault-architecture.md`.

## Kubernetes Integration

### External Secrets Operator (ESO)

Syncs secrets from 30+ providers to Kubernetes Secrets.

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          role: "app-role"
```

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/database/config
```

### Vault Secrets Operator (VSO)

Kubernetes-native Vault integration with automatic lease renewal.

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    name: dynamic-db-creds
```

For ESO vs CSI vs VSO comparison, see `references/kubernetes-integration.md`.

## Secret Rotation Patterns

### Pattern 1: Versioned Static Secrets (Blue/Green)

1. Create new secret version in Vault
2. Update staging environment
3. Monitor for errors (24-48 hours)
4. Gradual production rollout (10% → 50% → 100%)
5. Revoke old secret (after 7 days)

### Pattern 2: Dynamic Database Credentials

Vault auto-generates credentials with short TTL:
- App fetches credentials from Vault
- Vault automatically renews lease (at 67% of TTL)
- On expiration, Vault revokes access
- On renewal failure, app requests new credentials

### Pattern 3: TLS Certificate Rotation

Using cert-manager + Vault PKI:
- cert-manager requests certificate from Vault
- Automatically renews before expiration (default: 67% of duration)
- Updates Kubernetes Secret on renewal
- Optional pod restart (via Reloader)

For detailed rotation workflows, see `references/rotation-patterns.md`.

## Multi-Language Integration

### Python (hvac)

```python
import hvac

client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes(role='app-role', jwt=jwt)

# Fetch dynamic credentials
response = client.secrets.database.generate_credentials(name='postgres-role')
username = response['data']['username']
password = response['data']['password']
```

### Go (Vault API)

```go
import vault "github.com/hashicorp/vault/api"

client, _ := vault.NewClient(vault.DefaultConfig())
k8sAuth, _ := auth.NewKubernetesAuth("app-role")
client.Auth().Login(context.Background(), k8sAuth)

secret, _ := client.Logical().Read("database/creds/postgres-role")
```

### TypeScript (node-vault)

```typescript
import vault from 'node-vault';

const client = vault({ endpoint: 'https://vault.example.com' });
await client.kubernetesLogin({ role: 'app-role', jwt });

const response = await client.read('database/creds/postgres-role');
```

For complete examples, see `examples/dynamic-db-credentials/`.

## Secret Scanning

### Pre-Commit Hooks (Gitleaks)

```bash
# Install Gitleaks
brew install gitleaks

# Run on staged files
gitleaks protect --staged --verbose
```

Pre-commit hook prevents secrets from being committed.
For setup, see `examples/secret-scanning/pre-commit`.

### CI/CD Integration

```yaml
# GitHub Actions
- name: Run Gitleaks
  uses: gitleaks/gitleaks-action@v2
```

### Remediation Workflow

When a secret is leaked:
1. **Rotate immediately** (within 1 hour)
2. **Revoke at provider**
3. **Remove from Git history** (BFG Repo-Cleaner)
4. **Force push** (notify team)
5. **Audit access** (who had access during leak window)
6. **Document incident**

For detailed remediation, see `references/secret-scanning.md`.

## Zero-Knowledge Patterns

### Client-Side Encryption (E2EE)

User password → PBKDF2 → encryption key → encrypt secret → send to server

Server stores only encrypted blobs (cannot decrypt).

### Shamir's Secret Sharing

Split secret into N shares, require M to reconstruct (e.g., 3 of 5).

```bash
# Initialize Vault with Shamir shares
vault operator init -key-shares=5 -key-threshold=3

# Unseal requires 3 of 5 key shares
vault operator unseal <KEY_1>
vault operator unseal <KEY_2>
vault operator unseal <KEY_3>
```

For implementations, see `references/zero-knowledge.md`.

## Library Recommendations (2025)

### Secret Stores

| Library | Use Case | Trust Score |
|---------|----------|-------------|
| HashiCorp Vault | Enterprise, multi-cloud | High (73.3/100) |
| External Secrets Operator | Kubernetes integration | High (85.0/100) |
| AWS Secrets Manager | AWS workloads | High |
| GCP Secret Manager | GCP workloads | High |
| Azure Key Vault | Azure workloads | High |

### Secret Scanning

| Library | Use Case | Trust Score |
|---------|----------|-------------|
| Gitleaks | Pre-commit, CI/CD | High (89.9/100) |
| TruffleHog | Git history scanning | Medium |

### Client Libraries

| Language | Library | Version |
|----------|---------|---------|
| Python | `hvac` | 2.2.0+ |
| Go | `vault/api` | Latest |
| TypeScript | `node-vault` | 0.10.2+ |
| Rust | `vaultrs` | 0.7+ |

## Common Workflows

### Workflow 1: Vault + ESO on Kubernetes

1. Install Vault (Helm chart)
2. Initialize and unseal Vault
3. Enable Kubernetes auth
4. Install External Secrets Operator
5. Create SecretStore (Vault connection)
6. Create ExternalSecret (secret mapping)

For step-by-step guide, see `examples/vault-eso-setup/`.

### Workflow 2: Dynamic Database Credentials

1. Enable database secrets engine
2. Configure database connection
3. Create role with TTL
4. App fetches credentials
5. Vault auto-renews lease

For implementation, see `examples/dynamic-db-credentials/`.

### Workflow 3: Secret Scanning Remediation

1. Gitleaks detects secret
2. Block commit (pre-commit hook)
3. Developer removes secret
4. Developer stores in Vault
5. Developer references Vault path

For setup, see `examples/secret-scanning/`.

## Integration with Related Skills

- **auth-security**: OAuth client secrets, JWT signing keys
- **databases-***: Dynamic database credentials
- **deploying-applications**: Container registry credentials
- **observability**: Grafana/Datadog API keys
- **infrastructure-as-code**: Cloud provider credentials

## Security Best Practices

1. Never commit secrets to Git (use Gitleaks pre-commit hook)
2. Use dynamic secrets where possible
3. Rotate secrets regularly (quarterly for static, hourly for dynamic)
4. Implement least privilege (Vault policies, RBAC)
5. Enable audit logging
6. Encrypt at rest (Vault storage, etcd encryption)
7. Use short TTLs (< 24 hours for dynamic secrets)
8. Monitor failed access attempts

## Common Pitfalls

### Secrets in Environment Variables

Environment variables visible in process lists.
**Solution:** Use file-based secrets (Kubernetes volumes, CSI driver).

### Hardcoded Secrets in Manifests

Base64 is not encryption.
**Solution:** Use External Secrets Operator.

### No Secret Rotation

Stale credentials increase breach risk.
**Solution:** Use dynamic secrets or automate rotation.

### Root Token in Production

Unlimited permissions.
**Solution:** Use auth methods with least privilege policies.

## For Detailed Information, See

- `references/vault-architecture.md` - Vault internals, HA setup, policies
- `references/kubernetes-integration.md` - ESO, CSI driver, VSO comparison
- `references/rotation-patterns.md` - Detailed rotation workflows
- `references/secret-scanning.md` - Gitleaks, remediation procedures
- `references/zero-knowledge.md` - E2EE, Shamir's secret sharing
- `references/cloud-providers.md` - AWS, GCP, Azure secret managers
- `examples/vault-eso-setup/` - Complete Kubernetes setup
- `examples/dynamic-db-credentials/` - Multi-language examples
- `examples/secret-scanning/` - Pre-commit hooks, CI/CD
- `scripts/setup_vault.sh` - Automated Vault installation


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### references/vault-architecture.md

```markdown
# HashiCorp Vault Architecture

Comprehensive guide to Vault internals, high availability setup, policies, and production deployment patterns.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Secrets Engines](#secrets-engines)
3. [Authentication Methods](#authentication-methods)
4. [Policies and Access Control](#policies-and-access-control)
5. [Storage Backends](#storage-backends)
6. [High Availability Setup](#high-availability-setup)
7. [Audit Logging](#audit-logging)
8. [Production Deployment](#production-deployment)
9. [Common Pitfalls](#common-pitfalls)

## Architecture Overview

### Core Components

```
┌─────────────────────────────────────────────────────────┐
│              HashiCorp Vault Architecture                │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌────────────────────────────────────────────┐         │
│  │         API Layer (HTTP/HTTPS)             │         │
│  │  ├── RESTful API                           │         │
│  │  ├── CLI (vault command)                   │         │
│  │  └── UI (Web interface)                    │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Authentication Methods                │         │
│  │  ├── Kubernetes (service accounts)         │         │
│  │  ├── JWT/OIDC (external identity)          │         │
│  │  ├── AppRole (machines, CI/CD)             │         │
│  │  ├── LDAP/Active Directory                 │         │
│  │  └── Token (direct auth)                   │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │         Core (Policy Engine)               │         │
│  │  ├── Request routing                       │         │
│  │  ├── Policy evaluation (HCL)               │         │
│  │  ├── Lease management                      │         │
│  │  └── Token generation                      │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │         Secrets Engines                    │         │
│  │  ├── KV v2 (versioned key-value)           │         │
│  │  ├── Database (dynamic credentials)        │         │
│  │  ├── AWS (dynamic IAM)                     │         │
│  │  ├── PKI (TLS certificates)                │         │
│  │  └── SSH (dynamic certificates)            │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Barrier (Encryption Layer)            │         │
│  │  ├── AES-256-GCM encryption                │         │
│  │  ├── Unsealing mechanism                   │         │
│  │  └── Master key protection                 │         │
│  └────────────────┬───────────────────────────┘         │
│                   │                                      │
│                   ▼                                      │
│  ┌────────────────────────────────────────────┐         │
│  │      Storage Backend                       │         │
│  │  ├── Consul (HA, recommended)              │         │
│  │  ├── etcd (Kubernetes native)              │         │
│  │  ├── S3 (cost-effective, no HA)            │         │
│  │  └── PostgreSQL (relational)               │         │
│  └────────────────────────────────────────────┘         │
│                                                          │
└─────────────────────────────────────────────────────────┘
```

### Data Flow

1. **Request**: Client sends request (CLI, API, SDK)
2. **Authentication**: Vault validates identity (token, K8s SA, etc.)
3. **Authorization**: Policy engine evaluates permissions
4. **Secrets Engine**: Generates or retrieves secret
5. **Encryption**: Barrier encrypts data
6. **Storage**: Encrypted data written to backend
7. **Response**: Decrypted secret returned to client

## Secrets Engines

### KV v2 (Versioned Key-Value)

Static secrets with versioning and soft deletes.

**Enable:**
```bash
vault secrets enable -path=secret kv-v2
```

**Write Secret:**
```bash
vault kv put secret/myapp/config \
  api_key=sk_live_123 \
  database_url=postgresql://localhost/mydb
```

**Read Secret:**
```bash
vault kv get secret/myapp/config
vault kv get -version=2 secret/myapp/config  # Specific version
```

**Versioning:**
```bash
vault kv metadata get secret/myapp/config  # View all versions
vault kv undelete -versions=2 secret/myapp/config  # Restore deleted version
vault kv destroy -versions=1 secret/myapp/config  # Permanently delete
```

**Use Cases:**
- API keys (third-party services)
- OAuth client secrets
- Encryption keys (KEK)
- Configuration values

### Database Secrets Engine (Dynamic Credentials)

Auto-generates database credentials with TTL.

**Enable:**
```bash
vault secrets enable database
```

**Configure PostgreSQL:**
```bash
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role,readonly-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=require" \
  username="vault-admin" \
  password="vault-admin-password"
```

**Create Role:**
```bash
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"
```

**Generate Credentials:**
```bash
vault read database/creds/app-role
# Output:
# Key                Value
# ---                -----
# lease_id           database/creds/app-role/abc123
# lease_duration     1h
# username           v-k8s-app-role-xyz789
# password           A1b2C3d4E5f6
```

**Rotation:**
```bash
# Rotate root credentials (Vault admin password)
vault write -force database/rotate-root/postgres
```

**Supported Databases:**
- PostgreSQL, MySQL, MongoDB
- MSSQL, Oracle, Cassandra
- Elasticsearch, InfluxDB
- Redshift, Snowflake

### AWS Secrets Engine (Dynamic IAM)

Auto-generates AWS IAM credentials.

**Enable:**
```bash
vault secrets enable aws
```

**Configure:**
```bash
vault write aws/config/root \
  access_key=AKIAIOSFODNN7EXAMPLE \
  secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  region=us-east-1
```

**Create Role:**
```bash
vault write aws/roles/app-role \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF
```

**Generate Credentials:**
```bash
vault read aws/creds/app-role
# Output:
# Key                Value
# ---                -----
# lease_id           aws/creds/app-role/def456
# lease_duration     15m
# access_key         AKIAI44QH8DHBEXAMPLE
# secret_key         je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
```

**TTL:** Default 15 minutes, auto-revoked on expiration.

### PKI Secrets Engine (TLS Certificates)

Issues TLS certificates with automatic renewal.

**Enable:**
```bash
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki
```

**Generate Root CA:**
```bash
vault write pki/root/generate/internal \
  common_name=example.com \
  ttl=8760h
```

**Configure URLs:**
```bash
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com/v1/pki/crl"
```

**Create Role:**
```bash
vault write pki/roles/web-server \
  allowed_domains=example.com \
  allow_subdomains=true \
  max_ttl=72h
```

**Issue Certificate:**
```bash
vault write pki/issue/web-server \
  common_name=app.example.com \
  ttl=24h
# Returns: certificate, issuing_ca, private_key
```

## Authentication Methods

### Kubernetes Auth

Authenticate using Kubernetes service account tokens.

**Enable:**
```bash
vault auth enable kubernetes
```

**Configure:**
```bash
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  token_reviewer_jwt="<service-account-jwt>" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
```

**Create Role:**
```bash
vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=production \
  policies=app-policy \
  ttl=1h
```

**Login (from pod):**
```bash
vault write auth/kubernetes/login \
  role=app-role \
  jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
```

### JWT/OIDC Auth

Authenticate using external identity providers (Okta, Auth0, Google).

**Enable:**
```bash
vault auth enable oidc
```

**Configure:**
```bash
vault write auth/oidc/config \
  oidc_discovery_url="https://accounts.google.com" \
  oidc_client_id="<client-id>" \
  oidc_client_secret="<client-secret>" \
  default_role="default"
```

**Create Role:**
```bash
vault write auth/oidc/role/default \
  bound_audiences="<client-id>" \
  allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
  user_claim="email" \
  policies=default
```

### AppRole Auth

Machine authentication for CI/CD, applications.

**Enable:**
```bash
vault auth enable approle
```

**Create Role:**
```bash
vault write auth/approle/role/ci-role \
  secret_id_ttl=10m \
  token_num_uses=10 \
  token_ttl=20m \
  token_max_ttl=30m \
  secret_id_num_uses=40 \
  policies=ci-policy
```

**Get RoleID:**
```bash
vault read auth/approle/role/ci-role/role-id
# role_id: 12345678-1234-1234-1234-123456789012
```

**Generate SecretID:**
```bash
vault write -f auth/approle/role/ci-role/secret-id
# secret_id: 87654321-4321-4321-4321-210987654321
```

**Login:**
```bash
vault write auth/approle/login \
  role_id=12345678-1234-1234-1234-123456789012 \
  secret_id=87654321-4321-4321-4321-210987654321
```

## Policies and Access Control

### Policy Syntax (HCL)

```hcl
# Read access to KV v2 secrets
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}

# Write access to specific secret
path "secret/data/myapp/config" {
  capabilities = ["create", "update", "read"]
}

# Dynamic database credentials
path "database/creds/app-role" {
  capabilities = ["read"]
}

# Deny all other paths
path "*" {
  capabilities = ["deny"]
}
```

**Capabilities:**
- `create`: Create new secrets
- `read`: Read existing secrets
- `update`: Update existing secrets
- `delete`: Delete secrets
- `list`: List secret paths
- `sudo`: Admin operations
- `deny`: Explicitly deny access

### Policy Templates

Dynamic policies using templates:

```hcl
# Policy for service accounts in specific namespace
path "secret/data/{{identity.entity.aliases.auth_kubernetes_abc123.metadata.service_account_namespace}}/*" {
  capabilities = ["read"]
}
```

### Writing Policies

```bash
# Write policy from file
vault policy write app-policy app-policy.hcl

# Write policy inline
vault policy write readonly-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read", "list"]
}
EOF

# List policies
vault policy list

# Read policy
vault policy read app-policy
```

### Least Privilege Example

```hcl
# Application policy (minimal permissions)
path "secret/data/myapp/config" {
  capabilities = ["read"]
}

path "database/creds/app-role" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}
```

## Storage Backends

### Consul (Recommended for HA)

**Pros:**
- High availability
- Automatic failover
- Service discovery
- Health checking

**Cons:**
- Requires separate Consul cluster
- More complex setup

**Configuration:**
```hcl
storage "consul" {
  address = "consul.example.com:8500"
  path    = "vault/"
  token   = "<consul-token>"
}
```

### etcd (Kubernetes Native)

**Pros:**
- Kubernetes native
- HA support
- Widely deployed

**Cons:**
- Performance at scale
- Requires etcd cluster

**Configuration:**
```hcl
storage "etcd" {
  address  = "https://etcd.example.com:2379"
  etcd_api = "v3"
  path     = "vault/"
  ha_enabled = "true"
}
```

### S3 (Cost-Effective)

**Pros:**
- Low cost
- Managed service
- Unlimited storage

**Cons:**
- No HA support
- Higher latency

**Configuration:**
```hcl
storage "s3" {
  bucket     = "vault-storage"
  region     = "us-east-1"
  access_key = "AKIAIOSFODNN7EXAMPLE"
  secret_key = "wJalrXUtnFEMI/K7MDENG"
}
```

### PostgreSQL

**Pros:**
- Relational database
- Familiar tooling
- HA with replication

**Cons:**
- Not officially recommended
- Performance tuning needed

**Configuration:**
```hcl
storage "postgresql" {
  connection_url = "postgres://vault:password@postgres:5432/vault?sslmode=require"
  ha_enabled     = "true"
}
```

## High Availability Setup

### HA Architecture

```
┌────────────────────────────────────────────────┐
│         Vault HA Cluster (3 nodes)             │
├────────────────────────────────────────────────┤
│                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────┐ │
│  │   vault-0    │  │   vault-1    │  │vault-│ │
│  │   (Active)   │  │  (Standby)   │  │  2   │ │
│  │              │  │              │  │(Stand│ │
│  └──────┬───────┘  └──────┬───────┘  └──┬───┘ │
│         │                 │              │     │
│         └─────────────────┼──────────────┘     │
│                           │                    │
│                           ▼                    │
│              ┌─────────────────────┐           │
│              │   Storage Backend   │           │
│              │   (Consul/etcd)     │           │
│              └─────────────────────┘           │
│                                                 │
└────────────────────────────────────────────────┘
```

**Key Concepts:**
- One active node (handles requests)
- Multiple standby nodes (ready for failover)
- Automatic leader election
- Storage backend coordinates HA

### Kubernetes Deployment (HA)

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
spec:
  serviceName: vault
  replicas: 3
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15
        args:
        - "server"
        - "-config=/vault/config/vault.hcl"
        ports:
        - containerPort: 8200
          name: api
        - containerPort: 8201
          name: cluster
        volumeMounts:
        - name: config
          mountPath: /vault/config
        - name: data
          mountPath: /vault/data
      volumes:
      - name: config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
```

**vault.hcl:**
```hcl
storage "raft" {
  path    = "/vault/data"
  node_id = "vault-0"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/vault/tls/tls.crt"
  tls_key_file  = "/vault/tls/tls.key"
}

api_addr = "https://vault-0.vault:8200"
cluster_addr = "https://vault-0.vault:8201"

ui = true
```

### Unsealing in HA

All nodes must be unsealed independently:

```bash
# Unseal each node (requires 3 of 5 key shares)
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-0 -- vault operator unseal <KEY_SHARE_3>

kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-1 -- vault operator unseal <KEY_SHARE_3>

kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_1>
kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_2>
kubectl exec vault-2 -- vault operator unseal <KEY_SHARE_3>
```

**Auto-Unseal (Recommended for Production):**

Using cloud KMS (AWS, GCP, Azure):

```hcl
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc-def-ghi"
}
```

## Audit Logging

### Enable Audit Device

```bash
# File audit device
vault audit enable file file_path=/vault/logs/audit.log

# Syslog audit device
vault audit enable syslog tag="vault" facility="AUTH"

# Socket audit device
vault audit enable socket address="logstash.example.com:9090" socket_type="tcp"
```

### Audit Log Format

JSON format with request and response details:

```json
{
  "time": "2025-12-03T10:15:30Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:abc123...",
    "accessor": "hmac-sha256:def456...",
    "display_name": "kubernetes-production-app-sa",
    "policies": ["app-policy", "default"]
  },
  "request": {
    "operation": "read",
    "path": "database/creds/app-role",
    "remote_address": "10.244.1.5"
  },
  "response": {
    "secret": {
      "lease_id": "database/creds/app-role/xyz789"
    }
  }
}
```

**Logged Information:**
- Authentication details
- Request path and operation
- Response (secrets are HMAC-hashed)
- Remote IP address
- Timestamp

## Production Deployment

### Checklist

- [ ] **TLS enabled** (never run Vault without TLS in production)
- [ ] **HA setup** (3+ nodes with Consul/etcd/Raft storage)
- [ ] **Auto-unseal** (AWS KMS, GCP KMS, Azure Key Vault)
- [ ] **Audit logging** (multiple devices for redundancy)
- [ ] **Backup strategy** (storage backend snapshots)
- [ ] **Monitoring** (Prometheus metrics, health checks)
- [ ] **Policies reviewed** (least privilege, no root tokens)
- [ ] **Secrets rotation** (quarterly for static, automatic for dynamic)
- [ ] **Disaster recovery** (tested restore procedure)

### Resource Requirements

**Development:**
- 1 CPU, 512 MB RAM
- Single node
- File storage backend

**Production (Small - <1000 secrets):**
- 2 CPU, 2 GB RAM per node
- 3 nodes (HA)
- Consul/etcd storage

**Production (Large - >10,000 secrets):**
- 4 CPU, 8 GB RAM per node
- 5+ nodes (HA)
- Dedicated storage cluster
- Load balancer

### Monitoring

**Prometheus Metrics:**
```yaml
# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}
```

**Key Metrics:**
- `vault_core_unsealed`: Vault seal status (1 = unsealed)
- `vault_core_active`: Active node (1 = active, 0 = standby)
- `vault_token_count`: Total active tokens
- `vault_secret_lease_creation`: Secret lease generation rate
- `vault_runtime_alloc_bytes`: Memory usage

## Common Pitfalls

### Pitfall 1: Running Without TLS

**Problem:** Secrets transmitted in plaintext.

**Solution:**
```hcl
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0  # NEVER set to 1 in production
  tls_cert_file = "/vault/tls/tls.crt"
  tls_key_file  = "/vault/tls/tls.key"
}
```

### Pitfall 2: Using Root Token

**Problem:** Unlimited permissions, no audit trail.

**Solution:** Delete root token after initial setup, use auth methods.

```bash
# Revoke root token
vault token revoke <ROOT_TOKEN>
```

### Pitfall 3: No Backup Strategy

**Problem:** Data loss on storage backend failure.

**Solution:** Regular snapshots of storage backend.

```bash
# Consul snapshot
consul snapshot save backup.snap

# etcd snapshot
ETCDCTL_API=3 etcdctl snapshot save backup.db
```

### Pitfall 4: Unsealing Delays

**Problem:** Manual unsealing after restart delays recovery.

**Solution:** Use auto-unseal with cloud KMS.

### Pitfall 5: Single Audit Device

**Problem:** Audit log failure blocks all requests.

**Solution:** Configure multiple audit devices (file + syslog).

```bash
vault audit enable file file_path=/vault/logs/audit.log
vault audit enable syslog tag="vault"
```

### Pitfall 6: No Lease Renewal

**Problem:** Dynamic secrets expire, causing app failures.

**Solution:** Implement lease renewal in application code.

```python
import time
import threading

def renew_lease(client, lease_id, lease_duration):
    renewal_time = lease_duration * 0.67  # Renew at 67% of TTL
    time.sleep(renewal_time)
    client.sys.renew_lease(lease_id)
```

### Pitfall 7: Hardcoded Policies in Code

**Problem:** Policy changes require code deployment.

**Solution:** Store policies as files, reference in Vault.

```bash
vault policy write app-policy /path/to/app-policy.hcl
```

```

### references/kubernetes-integration.md

```markdown
# Kubernetes Secret Integration

Comprehensive guide to integrating secret management with Kubernetes using External Secrets Operator, Secrets Store CSI Driver, and Vault Secrets Operator.

## Table of Contents

1. [Integration Approaches](#integration-approaches)
2. [External Secrets Operator (ESO)](#external-secrets-operator-eso)
3. [Secrets Store CSI Driver](#secrets-store-csi-driver)
4. [Vault Secrets Operator (VSO)](#vault-secrets-operator-vso)
5. [Comparison Matrix](#comparison-matrix)
6. [Best Practices](#best-practices)

## Integration Approaches

### Native Kubernetes Secrets (Baseline)

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=  # base64("admin")
  password: cGFzc3dvcmQ=  # base64("password")
```

**Problems:**
- Base64 is NOT encryption
- Secrets stored in etcd (encrypted at rest if enabled)
- Manual rotation required
- No audit trail
- Secrets in Git (if committed)

**Solution:** Use external secret stores + sync operators.

### External Secret Stores

```
┌─────────────────────────────────────────────────────┐
│         External Secret Store Integrations          │
├─────────────────────────────────────────────────────┤
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   External Secrets Operator (ESO)      │         │
│  │  ├── Syncs from 30+ providers          │         │
│  │  ├── Creates Kubernetes Secrets        │         │
│  │  └── Polling-based refresh             │         │
│  └────────────────────────────────────────┘         │
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   Secrets Store CSI Driver             │         │
│  │  ├── Mounts secrets as files           │         │
│  │  ├── No Kubernetes Secret creation     │         │
│  │  └── Watch-based refresh               │         │
│  └────────────────────────────────────────┘         │
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │   Vault Secrets Operator (VSO)         │         │
│  │  ├── Vault-specific CRDs               │         │
│  │  ├── Dynamic secret renewal            │         │
│  │  └─ Kubernetes-native experience       │         │
│  └────────────────────────────────────────┘         │
│                                                      │
└─────────────────────────────────────────────────────┘
```

## External Secrets Operator (ESO)

### Installation

```bash
# Helm installation
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace

# Verify installation
kubectl get pods -n external-secrets-system
```

### Core Resources

**SecretStore** (Namespace-scoped)
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"
          serviceAccountRef:
            name: app-sa
```

**ClusterSecretStore** (Cluster-wide)
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-global
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "cluster-admin-role"
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets-system
```

**ExternalSecret**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h  # Poll every hour
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials  # Kubernetes Secret name
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: secret/data/database/config
      property: username
  - secretKey: password
    remoteRef:
      key: secret/data/database/config
      property: password
```

### Multi-Provider Examples

**AWS Secrets Manager**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: app-sa
```

**GCP Secret Manager**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secrets
spec:
  provider:
    gcpsm:
      projectID: "my-project-123"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production-cluster
          serviceAccountRef:
            name: app-sa
```

**Azure Key Vault**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secrets
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-vault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: app-sa
```

### Advanced Features

**Secret Templating**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-config
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: app-config
    template:
      type: Opaque
      data:
        config.yaml: |
          database:
            url: postgresql://{{ .username }}:{{ .password }}@postgres:5432/mydb
          api_key: {{ .api_key }}
  data:
  - secretKey: username
    remoteRef:
      key: secret/data/database/config
      property: username
  - secretKey: password
    remoteRef:
      key: secret/data/database/config
      property: password
  - secretKey: api_key
    remoteRef:
      key: secret/data/api/keys
      property: stripe_key
```

**DataFrom (Fetch All Keys)**
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: all-secrets
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: all-secrets
  dataFrom:
  - extract:
      key: secret/data/myapp/config  # Fetch all keys from this path
```

**PushSecret (Sync TO External Store)**
```yaml
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-credentials
spec:
  refreshInterval: 10m
  secretStoreRefs:
  - name: vault-backend
    kind: SecretStore
  selector:
    secret:
      name: local-secret  # Kubernetes Secret to push
  data:
  - match:
      secretKey: username
      remoteRef:
        remoteKey: secret/data/synced/credentials
        property: username
```

## Secrets Store CSI Driver

### Installation

```bash
# Install CSI driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

# Install Vault provider
kubectl apply -f https://raw.githubusercontent.com/hashicorp/vault-csi-provider/main/deployment/vault-csi-provider.yaml
```

### SecretProviderClass (Vault)

```yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "app-role"
    vaultSkipTLSVerify: "false"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/database/config"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/database/config"
        secretKey: "password"
```

### Pod with CSI Volume

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: production
spec:
  serviceAccountName: app-sa
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets-store
      mountPath: "/mnt/secrets"
      readOnly: true
    env:
    - name: DB_USERNAME_FILE
      value: /mnt/secrets/db-username
    - name: DB_PASSWORD_FILE
      value: /mnt/secrets/db-password
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"
```

### Sync to Kubernetes Secret (Optional)

```yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  secretObjects:
  - secretName: db-credentials
    type: Opaque
    data:
    - objectName: db-username
      key: username
    - objectName: db-password
      key: password
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "app-role"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/database/config"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/database/config"
        secretKey: "password"
```

### Auto-Rotation Detection

Application code to watch for file changes:

```python
import time
from pathlib import Path
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class SecretReloader(FileSystemEventHandler):
    def on_modified(self, event):
        if event.src_path == "/mnt/secrets/db-password":
            print("Secret rotated, reloading database connection...")
            reload_database_connection()

observer = Observer()
observer.schedule(SecretReloader(), "/mnt/secrets", recursive=False)
observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()
```

## Vault Secrets Operator (VSO)

### Installation

```bash
# Helm installation
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace
```

### VaultConnection

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: production
spec:
  address: "https://vault.example.com:8200"
  skipTLSVerify: false
  caCertSecretRef: vault-ca-cert
```

### VaultAuth

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: production
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa
```

### VaultStaticSecret (KV v2)

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: api-keys
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: secret
  path: myapp/api-keys
  type: kv-v2
  refreshAfter: 1h
  destination:
    create: true
    name: api-keys
```

### VaultDynamicSecret (Database)

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/postgres-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    create: true
    name: dynamic-db-creds
  rolloutRestartTargets:
  - kind: Deployment
    name: app
```

**Automatic Renewal:**
- VSO renews lease at 67% of TTL
- On renewal failure, requests new credentials
- Optionally triggers pod restart (rolloutRestartTargets)

### VaultPKISecret (TLS Certificates)

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: app-tls
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: pki
  role: web-server
  commonName: app.example.com
  altNames:
  - www.app.example.com
  ttl: 24h
  destination:
    create: true
    name: app-tls-cert
  rolloutRestartTargets:
  - kind: Deployment
    name: app
```

## Comparison Matrix

| Feature | ESO | CSI Driver | VSO |
|---------|-----|------------|-----|
| **Multi-Provider** | Yes (30+) | Yes (AWS, GCP, Azure, Vault) | No (Vault only) |
| **Secret Type** | Kubernetes Secret | Files (volume mount) | Kubernetes Secret |
| **Rotation** | Polling (refresh interval) | Watch (inotify) | Lease renewal (automatic) |
| **Dynamic Secrets** | Limited | Yes | Yes (native support) |
| **Pod Restart** | Manual | Manual | Automatic (optional) |
| **Complexity** | Low | Medium | Low (for Vault users) |
| **Performance** | Polling overhead | Low overhead | Low overhead |
| **Maturity** | High (CNCF) | High (Kubernetes SIG) | Medium (HashiCorp) |

### When to Use Each

**External Secrets Operator (ESO):**
- Multi-cloud environments
- Need to support multiple secret stores
- Static secrets with hourly refresh acceptable
- Team familiar with Kubernetes operators

**Secrets Store CSI Driver:**
- Need file-based secret delivery
- Automatic rotation without pod restart
- TLS certificates (frequent rotation)
- Applications that watch files for changes

**Vault Secrets Operator (VSO):**
- Vault-centric infrastructure
- Dynamic secrets (database, cloud)
- Automatic lease renewal required
- Prefer Kubernetes-native CRDs

## Best Practices

### 1. Namespace Isolation

Use namespace-scoped SecretStores:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production  # Isolated to production namespace
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret/production"  # Namespace-specific path
      auth:
        kubernetes:
          role: "production-app-role"
          serviceAccountRef:
            name: app-sa
```

### 2. Least Privilege Service Accounts

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["db-credentials"]  # Specific secret only
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-secret-reader-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-sa
roleRef:
  kind: Role
  name: app-secret-reader
  apiGroup: rbac.authorization.k8s.io
```

### 3. Refresh Interval Tuning

```yaml
# Frequent rotation (TLS certs)
refreshInterval: 10m

# Moderate rotation (API keys)
refreshInterval: 1h

# Infrequent rotation (static configs)
refreshInterval: 6h
```

### 4. Monitoring and Alerting

**Prometheus Metrics (ESO):**
```yaml
- external_secrets_sync_calls_total
- external_secrets_sync_calls_error
- external_secrets_status_condition
```

**Alerts:**
```yaml
- alert: ExternalSecretSyncFailure
  expr: external_secrets_sync_calls_error > 0
  for: 5m
  annotations:
    summary: "ExternalSecret sync failing"
```

### 5. Encryption at Rest (etcd)

Enable etcd encryption for Kubernetes Secrets:

```yaml
# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <BASE64_ENCODED_SECRET>
  - identity: {}
```

```yaml
# kube-apiserver flag
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml
```

### 6. Secret Versioning

Track secret versions for rollback:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: versioned-secret
  annotations:
    external-secrets.io/secret-version: "v2"
spec:
  secretStoreRef:
    name: vault-backend
  target:
    name: app-config
  data:
  - secretKey: api_key
    remoteRef:
      key: secret/data/api-keys
      version: 2  # Specific version
```

### 7. Testing Secret Rotation

```bash
# Update secret in Vault
vault kv put secret/myapp/config api_key=new_key_v2

# Wait for refresh interval
sleep 3600

# Verify Kubernetes Secret updated
kubectl get secret db-credentials -o jsonpath='{.data.api_key}' | base64 -d

# Check pod logs for reload
kubectl logs -f deployment/app
```

### 8. Disaster Recovery

Backup SecretStore configurations:

```bash
# Export all ExternalSecrets
kubectl get externalsecrets -A -o yaml > externalsecrets-backup.yaml

# Export all SecretStores
kubectl get secretstores -A -o yaml > secretstores-backup.yaml

# Restore
kubectl apply -f externalsecrets-backup.yaml
kubectl apply -f secretstores-backup.yaml
```

```

### references/rotation-patterns.md

```markdown
# Secret Rotation Patterns

Detailed workflows for rotating static secrets, dynamic credentials, and TLS certificates.

## Table of Contents

1. [Why Rotate Secrets](#why-rotate-secrets)
2. [Versioned Static Secrets](#versioned-static-secrets)
3. [Dynamic Database Credentials](#dynamic-database-credentials)
4. [TLS Certificate Rotation](#tls-certificate-rotation)
5. [Cloud Provider Credentials](#cloud-provider-credentials)
6. [Automation Scripts](#automation-scripts)

## Why Rotate Secrets

**Security Benefits:**
- Limits blast radius of compromised credentials
- Reduces window of unauthorized access
- Meets compliance requirements (SOC 2, ISO 27001, PCI DSS)
- Detects dormant credential usage

**Industry Standards:**
- **PCI DSS**: Rotate passwords every 90 days
- **SOC 2**: Document rotation policy and evidence
- **ISO 27001**: Regular credential review and rotation
- **NIST**: Recommend rotation on suspicion of compromise

## Versioned Static Secrets

### Pattern: Blue/Green Rotation

For third-party API keys with no auto-rotation support.

**Steps:**

1. **Create New Secret Version**
```bash
# Vault KV v2 (versioned)
vault kv put secret/api-keys/stripe \
  key=sk_live_NEW_KEY_v2 \
  created_at="2025-12-03T10:00:00Z" \
  rotated_by="ops-team"

# Verify version created
vault kv metadata get secret/api-keys/stripe
# current_version: 2
```

2. **Update Staging Environment**
```bash
# Update Kubernetes Secret in staging
kubectl set env deployment/api-service -n staging \
  STRIPE_API_KEY=sk_live_NEW_KEY_v2

# Or update ExternalSecret to fetch version 2
kubectl patch externalsecret stripe-key -n staging --type merge -p '
spec:
  data:
  - secretKey: key
    remoteRef:
      key: secret/data/api-keys/stripe
      property: key
      version: 2
'
```

3. **Monitor for Errors (24-48 Hours)**
```bash
# Check application logs
kubectl logs -f deployment/api-service -n staging | grep -i "stripe\|error"

# Monitor error rates
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\"}[5m])"

# Verify API calls succeed
curl https://api.stripe.com/v1/charges \
  -u sk_live_NEW_KEY_v2: \
  -d amount=100 \
  -d currency=usd \
  -d source=tok_test
```

4. **Gradual Production Rollout**
```bash
# Update 10% of pods
kubectl patch deployment api-service -n production -p '
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
'
kubectl set env deployment/api-service -n production STRIPE_API_KEY=sk_live_NEW_KEY_v2

# Wait 1 hour, check metrics
sleep 3600
kubectl top pods -n production -l app=api-service

# Update remaining pods
kubectl rollout status deployment/api-service -n production
```

5. **Revoke Old Secret (After 7 Days)**
```bash
# Revoke at provider
# Stripe: Dashboard → API Keys → Revoke old key

# Delete old version from Vault
vault kv metadata delete secret/api-keys/stripe -versions=1

# Document rotation
vault kv put secret/rotation-log/stripe-2025-12 \
  old_key=sk_live_OLD_KEY_v1 \
  new_key=sk_live_NEW_KEY_v2 \
  rotated_at="2025-12-03T10:00:00Z" \
  revoked_at="2025-12-10T10:00:00Z"
```

### Automation Script

```bash
#!/bin/bash
# rotate-static-secret.sh

SECRET_PATH=$1  # e.g., "secret/api-keys/stripe"
NEW_VALUE=$2

echo "Rotating secret: $SECRET_PATH"

# Write new version
vault kv put "$SECRET_PATH" \
  key="$NEW_VALUE" \
  created_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  rotated_by="$(whoami)"

# Get new version number
NEW_VERSION=$(vault kv metadata get -format=json "$SECRET_PATH" | jq -r '.current_version')
echo "New version: $NEW_VERSION"

# Update staging
kubectl set env deployment/api-service -n staging \
  API_KEY="$NEW_VALUE"

echo "Secret rotated. Monitor staging for 24-48 hours before production rollout."
```

## Dynamic Database Credentials

### Pattern: Automatic Lease Renewal

Vault auto-generates credentials with short TTL (1 hour).

**Initial Setup:**

```bash
# 1. Enable database engine
vault secrets enable database

# 2. Configure PostgreSQL connection
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=require" \
  username="vault-admin" \
  password="vault-admin-password"

# 3. Create role with TTL
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"
```

**Application Integration (Python):**

```python
import hvac
import time
import threading
from sqlalchemy import create_engine
from sqlalchemy.pool import NullPool

class VaultDatabaseClient:
    def __init__(self, vault_url, role):
        self.client = hvac.Client(url=vault_url)
        self.role = role
        self.lease_id = None
        self.engine = None

        # Authenticate
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
            jwt = f.read()
        self.client.auth.kubernetes(role='app-role', jwt=jwt)

        # Get initial credentials
        self._refresh_credentials()

        # Start renewal thread
        threading.Thread(target=self._renewal_loop, daemon=True).start()

    def _refresh_credentials(self):
        response = self.client.secrets.database.generate_credentials(name=self.role)
        username = response['data']['username']
        password = response['data']['password']
        self.lease_id = response['lease_id']
        self.lease_duration = response['lease_duration']

        # Create new database engine
        if self.engine:
            self.engine.dispose()
        self.engine = create_engine(
            f"postgresql://{username}:{password}@postgres:5432/mydb",
            poolclass=NullPool  # Don't pool connections (credentials rotate)
        )

        print(f"Credentials refreshed. Lease ID: {self.lease_id}, TTL: {self.lease_duration}s")

    def _renewal_loop(self):
        while True:
            # Renew at 67% of lease duration
            renewal_time = self.lease_duration * 0.67
            time.sleep(renewal_time)

            try:
                # Attempt renewal
                self.client.sys.renew_lease(self.lease_id)
                print(f"Lease renewed: {self.lease_id}")
            except Exception as e:
                print(f"Renewal failed: {e}. Requesting new credentials...")
                self._refresh_credentials()

    def query(self, sql):
        with self.engine.connect() as conn:
            return conn.execute(sql).fetchall()

# Usage
db_client = VaultDatabaseClient('https://vault.example.com', 'app-role')
users = db_client.query("SELECT * FROM users")
```

**Kubernetes with VSO:**

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
  namespace: production
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    create: true
    name: dynamic-db-creds
  rolloutRestartTargets:
  - kind: Deployment
    name: app  # Auto-restart on credential change
```

**Monitoring:**

```bash
# Check active leases
vault list sys/leases/lookup/database/creds/app-role

# View lease details
vault lease lookup database/creds/app-role/<LEASE_ID>

# Force revoke (emergency)
vault lease revoke database/creds/app-role/<LEASE_ID>

# Revoke all leases for role
vault lease revoke -prefix database/creds/app-role
```

## TLS Certificate Rotation

### Pattern: cert-manager + Vault PKI

Automatic certificate issuance and renewal.

**Vault PKI Setup:**

```bash
# 1. Enable PKI engine
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki

# 2. Generate root CA
vault write pki/root/generate/internal \
  common_name=example.com \
  ttl=8760h

# 3. Configure URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com/v1/pki/crl"

# 4. Create role
vault write pki/roles/web-server \
  allowed_domains=example.com \
  allow_subdomains=true \
  max_ttl=72h \
  key_type=rsa \
  key_bits=2048
```

**cert-manager Integration:**

```yaml
# 1. Issuer (Vault-backed)
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: production
spec:
  vault:
    server: https://vault.example.com
    path: pki/sign/web-server
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: cert-manager-vault-token
          key: token

---
# 2. Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: app-tls
  namespace: production
spec:
  secretName: app-tls-secret
  duration: 24h
  renewBefore: 8h  # Renew 8 hours before expiration (67% of 24h)
  issuerRef:
    name: vault-issuer
  dnsNames:
  - app.example.com
  - www.app.example.com
```

**Automatic Renewal:**

cert-manager automatically:
1. Monitors certificate expiration
2. Requests new certificate 8 hours before expiry
3. Updates Kubernetes Secret (app-tls-secret)
4. Triggers pod reload (via Reloader or similar)

**Pod Reload with Reloader:**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  annotations:
    reloader.stakater.com/auto: "true"  # Auto-reload on Secret change
spec:
  template:
    spec:
      containers:
      - name: app
        volumeMounts:
        - name: tls
          mountPath: /etc/tls
      volumes:
      - name: tls
        secret:
          secretName: app-tls-secret
```

## Cloud Provider Credentials

### Pattern: AWS IAM with Vault

**Vault AWS Engine Setup:**

```bash
# 1. Enable AWS engine
vault secrets enable aws

# 2. Configure root credentials
vault write aws/config/root \
  access_key=AKIAIOSFODNN7EXAMPLE \
  secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  region=us-east-1

# 3. Create role with inline policy
vault write aws/roles/s3-access \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF
  default_ttl=15m \
  max_ttl=1h
```

**Application Integration:**

```python
import boto3
import hvac
import time
import threading

class VaultAWSClient:
    def __init__(self, vault_url, aws_role):
        self.client = hvac.Client(url=vault_url)
        self.aws_role = aws_role
        self.lease_id = None

        # Authenticate with Vault
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
            jwt = f.read()
        self.client.auth.kubernetes(role='app-role', jwt=jwt)

        # Get initial AWS credentials
        self._refresh_credentials()

        # Start renewal thread
        threading.Thread(target=self._renewal_loop, daemon=True).start()

    def _refresh_credentials(self):
        response = self.client.secrets.aws.generate_credentials(name=self.aws_role)
        self.access_key = response['data']['access_key']
        self.secret_key = response['data']['secret_key']
        self.lease_id = response['lease_id']
        self.lease_duration = response['lease_duration']

        # Update boto3 session
        self.session = boto3.Session(
            aws_access_key_id=self.access_key,
            aws_secret_access_key=self.secret_key
        )
        self.s3 = self.session.client('s3')

        print(f"AWS credentials refreshed. Lease: {self.lease_id}, TTL: {self.lease_duration}s")

    def _renewal_loop(self):
        while True:
            renewal_time = self.lease_duration * 0.67
            time.sleep(renewal_time)

            try:
                self.client.sys.renew_lease(self.lease_id)
                print(f"Lease renewed: {self.lease_id}")
            except Exception as e:
                print(f"Renewal failed. Requesting new credentials...")
                self._refresh_credentials()

    def upload_file(self, bucket, key, file_path):
        self.s3.upload_file(file_path, bucket, key)

# Usage
aws_client = VaultAWSClient('https://vault.example.com', 's3-access')
aws_client.upload_file('my-bucket', 'data.csv', '/tmp/data.csv')
```

## Automation Scripts

### Comprehensive Rotation Script

```python
#!/usr/bin/env python3
"""
rotate_secrets.py

Automated secret rotation script for multiple secret types.

Usage:
  python rotate_secrets.py --type static --path secret/api-keys/stripe --value sk_live_NEW
  python rotate_secrets.py --type database --role app-role --ttl 1h
"""

import argparse
import hvac
import time
from datetime import datetime

class SecretRotator:
    def __init__(self, vault_url, vault_token):
        self.client = hvac.Client(url=vault_url, token=vault_token)

    def rotate_static(self, path, new_value):
        """Rotate static secret with versioning."""
        print(f"Rotating static secret: {path}")

        # Write new version
        self.client.secrets.kv.v2.create_or_update_secret(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            secret={'key': new_value},
            mount_point='secret'
        )

        # Get metadata
        metadata = self.client.secrets.kv.v2.read_secret_metadata(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            mount_point='secret'
        )

        version = metadata['data']['current_version']
        print(f"✓ New version created: {version}")
        print(f"✓ Previous version available for rollback")

        return version

    def rotate_database(self, role, ttl='1h'):
        """Generate new dynamic database credentials."""
        print(f"Generating new database credentials for role: {role}")

        response = self.client.secrets.database.generate_credentials(name=role)

        print(f"✓ Username: {response['data']['username']}")
        print(f"✓ Lease ID: {response['lease_id']}")
        print(f"✓ TTL: {response['lease_duration']}s")

        return response

    def revoke_old_version(self, path, versions_to_keep=2):
        """Delete old versions of static secret."""
        print(f"Cleaning up old versions: {path}")

        metadata = self.client.secrets.kv.v2.read_secret_metadata(
            path=path.replace('secret/data/', '').replace('secret/', ''),
            mount_point='secret'
        )

        current_version = metadata['data']['current_version']
        versions_to_delete = list(range(1, current_version - versions_to_keep + 1))

        if versions_to_delete:
            self.client.secrets.kv.v2.delete_secret_versions(
                path=path.replace('secret/data/', '').replace('secret/', ''),
                versions=versions_to_delete,
                mount_point='secret'
            )
            print(f"✓ Deleted versions: {versions_to_delete}")
        else:
            print("✓ No old versions to delete")

def main():
    parser = argparse.ArgumentParser(description='Rotate secrets in Vault')
    parser.add_argument('--type', required=True, choices=['static', 'database'])
    parser.add_argument('--path', help='Secret path (for static secrets)')
    parser.add_argument('--value', help='New secret value (for static secrets)')
    parser.add_argument('--role', help='Database role name (for dynamic secrets)')
    parser.add_argument('--ttl', default='1h', help='TTL for dynamic secrets')
    parser.add_argument('--vault-url', default='https://vault.example.com')
    parser.add_argument('--vault-token', required=True)

    args = parser.parse_args()

    rotator = SecretRotator(args.vault_url, args.vault_token)

    if args.type == 'static':
        if not args.path or not args.value:
            parser.error("--path and --value required for static secrets")
        rotator.rotate_static(args.path, args.value)

        # Wait 7 days before cleanup (manual trigger)
        print("\nRun the following command in 7 days to clean up old versions:")
        print(f"  python rotate_secrets.py --type cleanup --path {args.path} --vault-token <TOKEN>")

    elif args.type == 'database':
        if not args.role:
            parser.error("--role required for database secrets")
        rotator.rotate_database(args.role, args.ttl)

if __name__ == '__main__':
    main()
```

### Validation Script

```bash
#!/bin/bash
# validate_rotation.sh

set -e

SECRET_PATH=$1
EXPECTED_VERSION=$2

echo "Validating secret rotation: $SECRET_PATH"

# Check Vault
CURRENT_VERSION=$(vault kv metadata get -format=json "$SECRET_PATH" | jq -r '.current_version')

if [ "$CURRENT_VERSION" -eq "$EXPECTED_VERSION" ]; then
  echo "✓ Vault version correct: $CURRENT_VERSION"
else
  echo "✗ Vault version mismatch. Expected: $EXPECTED_VERSION, Got: $CURRENT_VERSION"
  exit 1
fi

# Check Kubernetes Secret (if using ESO)
K8S_SECRET_NAME=$(echo "$SECRET_PATH" | sed 's/\//-/g')
K8S_VERSION=$(kubectl get secret "$K8S_SECRET_NAME" -o jsonpath='{.metadata.annotations.external-secrets\.io/secret-version}')

if [ "$K8S_VERSION" -eq "$EXPECTED_VERSION" ]; then
  echo "✓ Kubernetes Secret synced: $K8S_VERSION"
else
  echo "✗ Kubernetes Secret not synced. Expected: $EXPECTED_VERSION, Got: $K8S_VERSION"
  exit 1
fi

echo "✓ Rotation validated successfully"
```

```

### references/secret-scanning.md

```markdown
# Secret Scanning and Remediation

Comprehensive guide to detecting, preventing, and remediating leaked secrets using Gitleaks and other tools.

## Table of Contents

1. [Why Scan for Secrets](#why-scan-for-secrets)
2. [Gitleaks](#gitleaks)
3. [Pre-Commit Hooks](#pre-commit-hooks)
4. [CI/CD Integration](#cicd-integration)
5. [Remediation Workflow](#remediation-workflow)
6. [Alternative Tools](#alternative-tools)

## Why Scan for Secrets

**The Problem:**
- 10M+ secrets exposed on GitHub in 2024 (GitGuardian)
- Average breach cost: $4.45M (IBM 2025)
- 63% of breaches from leaked credentials (Verizon DBIR)
- 95% preventable with secret scanning

**Common Leak Sources:**
- Git commits (hardcoded passwords, API keys)
- Environment files (.env, config.json)
- Configuration files (application.yml, settings.py)
- CI/CD logs (exposed secrets in build output)
- Docker images (secrets baked into layers)

## Gitleaks

High-performance secret scanner with customizable rules.

### Installation

```bash
# macOS
brew install gitleaks

# Linux
curl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/install.sh | sh

# Docker
docker pull ghcr.io/gitleaks/gitleaks:latest
```

### Basic Usage

```bash
# Scan current repository
gitleaks detect --verbose

# Scan specific directory
gitleaks detect --source /path/to/repo --verbose

# Scan Git history
gitleaks detect --log-opts "--all" --verbose

# Protect mode (scan uncommitted changes)
gitleaks protect --staged --verbose

# Generate report
gitleaks detect --report-format json --report-path gitleaks-report.json
```

### Configuration (.gitleaks.toml)

```toml
title = "Gitleaks Configuration"

[extend]
useDefault = true

[[rules]]
id = "generic-api-key"
description = "Generic API Key"
regex = '''(?i)(api[_-]?key|apikey)['"` ]*[:=]['"` ]*[a-zA-Z0-9]{20,}'''
tags = ["key", "API"]

[[rules]]
id = "aws-access-key"
description = "AWS Access Key"
regex = '''AKIA[0-9A-Z]{16}'''
tags = ["AWS", "key"]

[[rules]]
id = "private-key"
description = "Private Key"
regex = '''-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----'''
tags = ["key", "private"]

[[rules]]
id = "jwt"
description = "JSON Web Token"
regex = '''eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}'''
tags = ["JWT"]

[allowlist]
description = "Allowlist for test files and examples"
paths = [
  '''.*/test/.*''',
  '''.*/examples/.*''',
  '''.*_test\.go''',
  '''.*\.md'''
]
regexes = [
  '''sk_test_''',  # Stripe test keys
  '''pk_test_''',  # Stripe test keys
  '''EXAMPLE_API_KEY''',  # Placeholder examples
]
```

### Exit Codes

```bash
gitleaks detect
echo $?

# 0: No secrets found
# 1: Secrets detected
# 2: Error occurred
```

## Pre-Commit Hooks

### Install pre-commit

```bash
# Install pre-commit framework
pip install pre-commit

# Or via Homebrew
brew install pre-commit
```

### .pre-commit-config.yaml

```yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.18.1
  hooks:
  - id: gitleaks
    name: Gitleaks
    description: Detect secrets in staged files
    entry: gitleaks protect --staged --redact
    language: system
    pass_filenames: false
```

### Manual Hook (.git/hooks/pre-commit)

```bash
#!/bin/bash
# .git/hooks/pre-commit

echo "Running Gitleaks secret scan..."

# Run Gitleaks on staged files
gitleaks protect --staged --verbose --redact

if [ $? -ne 0 ]; then
  echo ""
  echo "❌ Secret detected! Commit blocked."
  echo "To fix:"
  echo "  1. Remove the secret from your code"
  echo "  2. Store it in Vault: vault kv put secret/myapp/config api_key=<YOUR_KEY>"
  echo "  3. Reference it in code: vault.read('secret/data/myapp/config')"
  echo ""
  echo "To bypass (NOT recommended): git commit --no-verify"
  exit 1
fi

echo "✅ No secrets detected. Proceeding with commit."
exit 0
```

**Install hook:**
```bash
chmod +x .git/hooks/pre-commit
```

### Husky (for Node.js projects)

```json
{
  "husky": {
    "hooks": {
      "pre-commit": "gitleaks protect --staged --verbose"
    }
  }
}
```

```bash
npx husky install
npx husky add .husky/pre-commit "gitleaks protect --staged --verbose"
```

## CI/CD Integration

### GitHub Actions

```yaml
# .github/workflows/secret-scan.yml
name: Secret Scanning

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  gitleaks:
    name: Gitleaks Secret Scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      with:
        fetch-depth: 0  # Full history for comprehensive scan

    - name: Run Gitleaks
      uses: gitleaks/gitleaks-action@v2
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}  # For commercial use

    - name: Upload results (if secrets found)
      if: failure()
      uses: actions/upload-artifact@v4
      with:
        name: gitleaks-report
        path: gitleaks-report.sarif
```

### GitLab CI

```yaml
# .gitlab-ci.yml
gitleaks:
  stage: test
  image: ghcr.io/gitleaks/gitleaks:latest
  script:
  - gitleaks detect --verbose --report-format json --report-path gitleaks-report.json
  artifacts:
    when: on_failure
    paths:
    - gitleaks-report.json
  allow_failure: false
```

### Jenkins

```groovy
pipeline {
  agent any
  stages {
    stage('Secret Scan') {
      steps {
        sh 'gitleaks detect --verbose --report-format json --report-path gitleaks-report.json'
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'gitleaks-report.json', allowEmptyArchive: true
    }
  }
}
```

### CircleCI

```yaml
# .circleci/config.yml
version: 2.1

jobs:
  gitleaks:
    docker:
    - image: ghcr.io/gitleaks/gitleaks:latest
    steps:
    - checkout
    - run:
        name: Run Gitleaks
        command: gitleaks detect --verbose
    - store_artifacts:
        path: gitleaks-report.json

workflows:
  version: 2
  build:
    jobs:
    - gitleaks
```

## Remediation Workflow

### When a Secret is Leaked

**Immediate Actions (within 1 hour):**

1. **Rotate the Exposed Secret**
```bash
# PRIORITY 1: Create new secret in Vault
vault kv put secret/api-keys/stripe \
  key=sk_live_NEW_KEY_AFTER_LEAK \
  rotated_reason="Leaked to Git on 2025-12-03" \
  previous_key=sk_live_OLD_KEY_LEAKED
```

2. **Revoke at Provider**
```bash
# Stripe: Dashboard → API Keys → Revoke
# AWS: IAM → Delete access key
# GitHub: Settings → Developer settings → Revoke token
```

3. **Update Applications**
```bash
# Update all environments immediately
kubectl set env deployment/api-service -n staging STRIPE_API_KEY=sk_live_NEW_KEY_AFTER_LEAK
kubectl set env deployment/api-service -n production STRIPE_API_KEY=sk_live_NEW_KEY_AFTER_LEAK
```

**Git History Cleanup (within 24 hours):**

4. **Remove from Git History (BFG Repo-Cleaner)**
```bash
# Install BFG
brew install bfg

# Clone fresh copy
git clone --mirror https://github.com/org/repo.git repo.git
cd repo.git

# Create file with secrets to remove
cat > secrets.txt <<EOF
sk_live_OLD_KEY_LEAKED
AKIA1234567890ABCDEF
-----BEGIN RSA PRIVATE KEY-----
EOF

# Remove secrets
bfg --replace-text secrets.txt .

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push (CAUTION: Coordinate with team)
git push --force --all
git push --force --tags
```

**Alternative: filter-repo**
```bash
pip install git-filter-repo

# Remove specific file
git filter-repo --invert-paths --path config/secrets.json

# Remove specific pattern
echo "sk_live_" > secrets-pattern.txt
git filter-repo --replace-text secrets-pattern.txt
```

5. **Audit Access**
```bash
# Check Vault audit logs
vault audit list

# Read audit log
cat /vault/logs/audit.log | jq 'select(.request.path == "secret/data/api-keys/stripe")'

# GitHub: Who cloned during leak window?
# Check repository Insights → Traffic → Git clones
```

6. **Document Incident**
```markdown
# Incident Report: Secret Leak 2025-12-03

## Summary
- **Date**: 2025-12-03 10:15 UTC
- **Secret**: Stripe API key (sk_live_OLD_KEY_LEAKED)
- **Exposure**: Committed to main branch, pushed to GitHub
- **Discovered**: Gitleaks CI/CD scan
- **Resolved**: Secret rotated within 30 minutes

## Timeline
- 10:15: Secret committed (commit abc123)
- 10:20: CI/CD pipeline detected leak, blocked merge
- 10:25: Secret rotated in Vault
- 10:30: Old key revoked at Stripe
- 11:00: Git history rewritten (BFG)
- 11:15: Force push completed

## Root Cause
- Developer accidentally committed .env file
- Pre-commit hook not installed locally

## Prevention
- [ ] Enforce pre-commit hooks (CI check)
- [ ] Add .env to .gitignore (template)
- [ ] Developer training on secret management
- [ ] Mandatory .gitleaks.toml in all repos

## Impact
- No unauthorized API usage detected
- No customer data accessed
- Leak window: 15 minutes (before rotation)
```

### False Positive Handling

**Allowlist in .gitleaks.toml:**
```toml
[allowlist]
description = "Allowlist for test files and examples"

# Ignore test files
paths = [
  '''.*/test/.*''',
  '''.*/examples/.*''',
  '''.*\.md''',  # Documentation
]

# Ignore test keys and placeholders
regexes = [
  '''sk_test_[a-zA-Z0-9]{24}''',  # Stripe test keys
  '''pk_test_[a-zA-Z0-9]{24}''',
  '''EXAMPLE_[A-Z_]+''',  # Placeholders
  '''YOUR_API_KEY_HERE''',
]

# Ignore specific commits (emergency override)
commits = [
  "abc123def456",  # Migration commit with test data
]
```

**Inline Ignore:**
```python
# gitleaks:allow
api_key = "sk_test_THIS_IS_A_TEST_KEY_FOR_EXAMPLES"
```

## Alternative Tools

### TruffleHog

Deep Git history scanning with entropy detection.

```bash
# Installation
pip install truffleHog

# Scan repository
trufflehog git https://github.com/org/repo.git

# Scan since specific commit
trufflehog git https://github.com/org/repo.git --since-commit abc123

# JSON output
trufflehog git https://github.com/org/repo.git --json
```

### detect-secrets (Yelp)

Baseline-based scanning for Python projects.

```bash
# Installation
pip install detect-secrets

# Create baseline
detect-secrets scan > .secrets.baseline

# Audit baseline (interactive)
detect-secrets audit .secrets.baseline

# Scan for new secrets
detect-secrets scan --baseline .secrets.baseline
```

### git-secrets (AWS)

Prevents committing AWS credentials.

```bash
# Installation
brew install git-secrets

# Install hooks
git secrets --install
git secrets --register-aws

# Scan repository
git secrets --scan
```

### Comparison Matrix

| Tool | Performance | Accuracy | Custom Rules | CI/CD | License |
|------|-------------|----------|--------------|-------|---------|
| **Gitleaks** | Excellent | High | Yes (regex) | Easy | MIT |
| **TruffleHog** | Good | High (entropy) | Limited | Moderate | GPL-3.0 |
| **detect-secrets** | Moderate | Moderate | Yes (plugins) | Easy | Apache-2.0 |
| **git-secrets** | Good | Moderate | Yes (regex) | Easy | Apache-2.0 |

**Recommendation:** Use Gitleaks for most projects (fast, accurate, easy CI/CD integration).

## Best Practices

1. **Layer Defenses**
   - Pre-commit hooks (local)
   - CI/CD scanning (remote)
   - Periodic full repository scans

2. **Rotate Immediately**
   - Assume leaked = compromised
   - Rotate within 1 hour
   - Revoke at provider

3. **Educate Developers**
   - Secret management training
   - Code review checklist
   - Security champions

4. **Monitor Audit Logs**
   - Track secret access patterns
   - Alert on unusual activity
   - Regular audit reviews

5. **Automate Response**
   - Auto-rotate on detection
   - Auto-revoke leaked credentials
   - Incident tickets

```

### references/zero-knowledge.md

```markdown
# Zero-Knowledge Secret Patterns

Client-side encryption and threshold cryptography for zero-knowledge secret management.

## Table of Contents

1. [Zero-Knowledge Principles](#zero-knowledge-principles)
2. [Client-Side Encryption (E2EE)](#client-side-encryption-e2ee)
3. [Shamir's Secret Sharing](#shamirs-secret-sharing)
4. [Use Cases](#use-cases)

## Zero-Knowledge Principles

**Definition:** Server never has access to decryption keys or plaintext secrets.

**Key Properties:**
- Encryption/decryption happens client-side only
- Server stores only encrypted blobs
- Master key derived from user password (never transmitted)
- Compromise of server doesn't expose secrets

**Use Cases:**
- Password managers (1Password, Bitwarden)
- Encrypted notes (Standard Notes)
- Secure messaging (Signal, WhatsApp)
- Vault root token recovery (Shamir shares)

## Client-Side Encryption (E2EE)

### Pattern: User Password → Encryption Key

```typescript
// Client-side encryption (browser)
import { pbkdf2, randomBytes, createCipheriv, createDecipheriv } from 'crypto';

class ZeroKnowledgeVault {
  // Derive encryption key from user password
  static async deriveKey(
    password: string,
    salt: Buffer
  ): Promise<Buffer> {
    return new Promise((resolve, reject) => {
      // PBKDF2 with 100k iterations
      pbkdf2(password, salt, 100000, 32, 'sha256', (err, key) => {
        if (err) reject(err);
        else resolve(key);
      });
    });
  }

  // Encrypt secret client-side
  static async encrypt(
    plaintext: string,
    password: string,
    salt: Buffer
  ): Promise<{ encrypted: string; iv: string; salt: string }> {
    const key = await this.deriveKey(password, salt);
    const iv = randomBytes(16);

    const cipher = createCipheriv('aes-256-gcm', key, iv);
    let encrypted = cipher.update(plaintext, 'utf8', 'base64');
    encrypted += cipher.final('base64');

    const authTag = cipher.getAuthTag();

    return {
      encrypted: encrypted + ':' + authTag.toString('base64'),
      iv: iv.toString('base64'),
      salt: salt.toString('base64'),
    };
  }

  // Decrypt secret client-side
  static async decrypt(
    encryptedData: { encrypted: string; iv: string; salt: string },
    password: string
  ): Promise<string> {
    const salt = Buffer.from(encryptedData.salt, 'base64');
    const key = await this.deriveKey(password, salt);
    const iv = Buffer.from(encryptedData.iv, 'base64');

    const [ciphertext, authTagB64] = encryptedData.encrypted.split(':');
    const authTag = Buffer.from(authTagB64, 'base64');

    const decipher = createDecipheriv('aes-256-gcm', key, iv);
    decipher.setAuthTag(authTag);

    let decrypted = decipher.update(ciphertext, 'base64', 'utf8');
    decrypted += decipher.final('utf8');

    return decrypted;
  }
}

// Usage
async function example() {
  const masterPassword = 'user-entered-password';
  const salt = randomBytes(32);

  // Encrypt secret (client-side)
  const encrypted = await ZeroKnowledgeVault.encrypt(
    'my-database-password',
    masterPassword,
    salt
  );

  // Send to server (server CANNOT decrypt)
  await fetch('/api/secrets', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(encrypted),
  });

  // Later: Decrypt secret (client-side)
  const decrypted = await ZeroKnowledgeVault.decrypt(encrypted, masterPassword);
  console.log(decrypted); // 'my-database-password'
}
```

### Server-Side (Zero-Knowledge)

```python
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Dict
import sqlite3

app = FastAPI()

class EncryptedSecret(BaseModel):
    encrypted: str
    iv: str
    salt: str

# Server stores ONLY encrypted data (cannot decrypt)
@app.post("/api/secrets")
async def store_secret(secret: EncryptedSecret):
    # Server has NO access to plaintext
    conn = sqlite3.connect('secrets.db')
    conn.execute('''
        INSERT INTO secrets (encrypted_data, iv, salt, created_at)
        VALUES (?, ?, ?, datetime('now'))
    ''', (secret.encrypted, secret.iv, secret.salt))
    conn.commit()
    conn.close()

    return {"status": "stored", "message": "Secret encrypted and stored"}

@app.get("/api/secrets/{id}")
async def get_secret(id: int):
    # Server returns encrypted blob (client decrypts)
    conn = sqlite3.connect('secrets.db')
    cursor = conn.execute(
        'SELECT encrypted_data, iv, salt FROM secrets WHERE id = ?',
        (id,)
    )
    row = cursor.fetchone()
    conn.close()

    if not row:
        raise HTTPException(status_code=404, detail="Secret not found")

    return {
        "encrypted": row[0],
        "iv": row[1],
        "salt": row[2]
    }
```

### Web Crypto API (Browser)

```javascript
// Modern browser-based encryption
class WebCryptoVault {
  // Derive key from password
  static async deriveKey(password, salt) {
    const encoder = new TextEncoder();
    const passwordKey = await crypto.subtle.importKey(
      'raw',
      encoder.encode(password),
      'PBKDF2',
      false,
      ['deriveKey']
    );

    return crypto.subtle.deriveKey(
      {
        name: 'PBKDF2',
        salt: salt,
        iterations: 100000,
        hash: 'SHA-256',
      },
      passwordKey,
      { name: 'AES-GCM', length: 256 },
      false,
      ['encrypt', 'decrypt']
    );
  }

  // Encrypt
  static async encrypt(plaintext, password) {
    const salt = crypto.getRandomValues(new Uint8Array(32));
    const iv = crypto.getRandomValues(new Uint8Array(12));
    const key = await this.deriveKey(password, salt);

    const encoder = new TextEncoder();
    const encrypted = await crypto.subtle.encrypt(
      { name: 'AES-GCM', iv: iv },
      key,
      encoder.encode(plaintext)
    );

    return {
      encrypted: btoa(String.fromCharCode(...new Uint8Array(encrypted))),
      iv: btoa(String.fromCharCode(...iv)),
      salt: btoa(String.fromCharCode(...salt)),
    };
  }

  // Decrypt
  static async decrypt(encryptedData, password) {
    const salt = Uint8Array.from(atob(encryptedData.salt), c => c.charCodeAt(0));
    const iv = Uint8Array.from(atob(encryptedData.iv), c => c.charCodeAt(0));
    const encrypted = Uint8Array.from(atob(encryptedData.encrypted), c => c.charCodeAt(0));

    const key = await this.deriveKey(password, salt);

    const decrypted = await crypto.subtle.decrypt(
      { name: 'AES-GCM', iv: iv },
      key,
      encrypted
    );

    const decoder = new TextDecoder();
    return decoder.decode(decrypted);
  }
}

// Usage
async function example() {
  const encrypted = await WebCryptoVault.encrypt('my-secret', 'user-password');
  console.log(encrypted); // { encrypted, iv, salt }

  const decrypted = await WebCryptoVault.decrypt(encrypted, 'user-password');
  console.log(decrypted); // 'my-secret'
}
```

## Shamir's Secret Sharing

### Pattern: Threshold Cryptography

Split a secret into N shares, require M shares to reconstruct.

**Use Cases:**
- Vault root token recovery (3 of 5 key holders)
- Multi-party approval workflows
- Distributed trust (no single point of failure)

### Python Implementation

```python
from secretsharing import PlaintextToHexSecretSharer

# Split secret into 5 shares (require 3 to reconstruct)
root_token = "your-secret-root-token-here"
shares = PlaintextToHexSecretSharer.split_secret(root_token, 3, 5)

print(f"Share 1 (CTO): {shares[0]}")
print(f"Share 2 (Security Lead): {shares[1]}")
print(f"Share 3 (DevOps Lead): {shares[2]}")
print(f"Share 4 (Compliance Officer): {shares[3]}")
print(f"Share 5 (Backup - secure storage): {shares[4]}")

# Reconstruct with any 3 of 5 shares
recovered_token = PlaintextToHexSecretSharer.recover_secret(shares[0:3])
assert recovered_token == root_token

# Only 2 shares? Cannot reconstruct
try:
    PlaintextToHexSecretSharer.recover_secret(shares[0:2])
except Exception as e:
    print(f"Error: {e}")  # "Not enough shares to reconstruct"
```

### Vault Initialization with Shamir

```bash
# Initialize Vault with Shamir shares
vault operator init \
  -key-shares=5 \
  -key-threshold=3

# Output:
# Unseal Key 1: abc123...
# Unseal Key 2: def456...
# Unseal Key 3: ghi789...
# Unseal Key 4: jkl012...
# Unseal Key 5: mno345...
# Initial Root Token: hvs.CAES...

# Distribute shares to different key holders
# - Share 1 → CTO (printed to paper, secure safe)
# - Share 2 → Security Lead (password manager)
# - Share 3 → DevOps Lead (HSM)
# - Share 4 → Compliance Officer (encrypted USB)
# - Share 5 → Backup (bank vault)
```

### Unsealing Vault (Requires 3 of 5 Shares)

```bash
# Key holder 1 (CTO) provides share
vault operator unseal abc123...
# Output: Sealed: true, Progress: 1/3

# Key holder 2 (Security Lead) provides share
vault operator unseal def456...
# Output: Sealed: true, Progress: 2/3

# Key holder 3 (DevOps Lead) provides share
vault operator unseal ghi789...
# Output: Sealed: false, Progress: 3/3

# Vault is now unsealed (3 of 5 threshold met)
```

### TypeScript Implementation (sss-wasm)

```typescript
import { split, combine } from 'sss-wasm';

// Split secret
const secret = Buffer.from('my-vault-root-token', 'utf8');
const shares = split(secret, { shares: 5, threshold: 3 });

console.log(`Share 1: ${shares[0].toString('hex')}`);
console.log(`Share 2: ${shares[1].toString('hex')}`);
console.log(`Share 3: ${shares[2].toString('hex')}`);
console.log(`Share 4: ${shares[3].toString('hex')}`);
console.log(`Share 5: ${shares[4].toString('hex')}`);

// Reconstruct with 3 shares
const recovered = combine([shares[0], shares[2], shares[4]]);
console.log(recovered.toString('utf8')); // 'my-vault-root-token'

// Only 2 shares? Cannot reconstruct
try {
  combine([shares[0], shares[1]]);
} catch (err) {
  console.error('Error: Not enough shares');
}
```

## Use Cases

### Use Case 1: Password Manager

```typescript
// User registers
const masterPassword = 'user-chosen-password';
const salt = crypto.getRandomValues(new Uint8Array(32));

// Server stores salt (NOT the password)
await fetch('/api/register', {
  method: 'POST',
  body: JSON.stringify({
    username: 'alice',
    salt: btoa(String.fromCharCode(...salt)),
  }),
});

// User stores a password
const encrypted = await WebCryptoVault.encrypt('my-gmail-password', masterPassword);
await fetch('/api/passwords', {
  method: 'POST',
  body: JSON.stringify({
    site: 'gmail.com',
    ...encrypted,
  }),
});

// Server has ZERO knowledge of:
// - Master password
// - Stored password (my-gmail-password)
// - Encryption key
```

### Use Case 2: Vault Root Token Recovery

```bash
# Scenario: All Vault nodes lost, need to recover root token

# Step 1: Gather 3 of 5 key holders
# - CTO provides share 1
# - Security Lead provides share 2
# - DevOps Lead provides share 3

# Step 2: Reconstruct root token (offline)
python3 <<EOF
from secretsharing import PlaintextToHexSecretSharer
shares = [
    "abc123...",  # Share 1 from CTO
    "def456...",  # Share 2 from Security Lead
    "ghi789..."   # Share 3 from DevOps Lead
]
root_token = PlaintextToHexSecretSharer.recover_secret(shares)
print(f"Root Token: {root_token}")
EOF

# Step 3: Use root token to initialize new Vault cluster
vault login hvs.CAES...
vault operator init -recovery-shares=5 -recovery-threshold=3
```

### Use Case 3: Multi-Party Approval

```python
# Scenario: Deploy requires approval from 2 of 3 leads

from secretsharing import PlaintextToHexSecretSharer

# Deploy key (required to trigger production deployment)
deploy_key = "prod-deploy-key-2025-12-03"

# Split into 3 shares (require 2 to reconstruct)
shares = PlaintextToHexSecretSharer.split_secret(deploy_key, 2, 3)

# Distribute shares
# - Tech Lead: shares[0]
# - Product Lead: shares[1]
# - Security Lead: shares[2]

# Deployment request: Tech Lead + Product Lead provide shares
recovered_key = PlaintextToHexSecretSharer.recover_secret([shares[0], shares[1]])

# CI/CD validates key before deploying
if recovered_key == deploy_key:
    print("✓ Deployment approved by 2 of 3 leads. Proceeding...")
    deploy_to_production()
else:
    print("✗ Invalid deployment key. Approval required.")
```

## Security Considerations

**Key Derivation:**
- Use PBKDF2 with 100k+ iterations (or Argon2id)
- Random salt (32 bytes, unique per user)
- Never transmit master password

**Encryption Algorithm:**
- AES-256-GCM (authenticated encryption)
- Random IV (12 bytes for GCM, 16 for CBC)
- Verify auth tag on decryption

**Shamir Shares:**
- Distribute shares to independent key holders
- Store in different locations (geographic separation)
- Document recovery procedure
- Test recovery annually

**Password Strength:**
- Enforce minimum entropy (80+ bits)
- Use password strength meter
- Consider passphrase (4+ random words)
- Offer 2FA for account recovery

```

### references/cloud-providers.md

```markdown
# Cloud Provider Secret Managers

Quick reference for AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault.

## Table of Contents

1. [Comparison Matrix](#comparison-matrix)
2. [AWS Secrets Manager](#aws-secrets-manager)
3. [GCP Secret Manager](#gcp-secret-manager)
4. [Azure Key Vault](#azure-key-vault)
5. [Kubernetes Integration (ESO)](#kubernetes-integration-eso)

## Comparison Matrix

| Feature | AWS Secrets Manager | GCP Secret Manager | Azure Key Vault |
|---------|--------------------|--------------------|-----------------|
| **Pricing** | $0.40/secret/month + $0.05/10k API calls | $0.06/secret version/month + $0.03/10k ops | $0.03/10k ops, Free tier |
| **Rotation** | Automatic (Lambda) | Manual (Cloud Functions) | Automatic (Event Grid) |
| **Versioning** | Yes (auto-versioned) | Yes (explicit versions) | Yes (versions) |
| **Replication** | Multi-region | Global | Multi-region |
| **IAM Integration** | Native (IAM policies) | Native (IAM policies) | Native (RBAC) |
| **Kubernetes** | ESO, CSI driver | ESO, CSI driver | ESO, CSI driver |

## AWS Secrets Manager

### CLI Commands

```bash
# Create secret
aws secretsmanager create-secret \
  --name prod/database/credentials \
  --secret-string '{"username":"admin","password":"secret123"}'

# Retrieve secret
aws secretsmanager get-secret-value \
  --secret-id prod/database/credentials \
  --query SecretString --output text

# Update secret
aws secretsmanager update-secret \
  --secret-id prod/database/credentials \
  --secret-string '{"username":"admin","password":"newsecret456"}'

# Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id prod/database/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:rotate-secret
```

### IAM Policy

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/*"
    }
  ]
}
```

### Python SDK (boto3)

```python
import boto3
import json

client = boto3.client('secretsmanager', region_name='us-east-1')

# Get secret
response = client.get_secret_value(SecretId='prod/database/credentials')
secret = json.loads(response['SecretString'])

print(f"Username: {secret['username']}")
print(f"Password: {secret['password']}")
```

## GCP Secret Manager

### gcloud Commands

```bash
# Create secret
echo -n "secret-password" | gcloud secrets create db-password \
  --data-file=- \
  --replication-policy=automatic

# Add version
echo -n "new-password" | gcloud secrets versions add db-password \
  --data-file=-

# Access secret
gcloud secrets versions access latest --secret=db-password

# Delete secret
gcloud secrets delete db-password
```

### IAM Binding

```bash
gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:[email protected]" \
  --role="roles/secretmanager.secretAccessor"
```

### Python SDK

```python
from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()

# Access secret
name = "projects/my-project/secrets/db-password/versions/latest"
response = client.access_secret_version(request={"name": name})
secret = response.payload.data.decode('UTF-8')

print(f"Secret: {secret}")
```

## Azure Key Vault

### Azure CLI Commands

```bash
# Create Key Vault
az keyvault create \
  --name my-keyvault \
  --resource-group my-rg \
  --location eastus

# Set secret
az keyvault secret set \
  --vault-name my-keyvault \
  --name db-password \
  --value "secret123"

# Get secret
az keyvault secret show \
  --vault-name my-keyvault \
  --name db-password \
  --query value --output tsv

# Enable soft-delete
az keyvault update \
  --name my-keyvault \
  --enable-soft-delete true \
  --retention-days 90
```

### Access Policy

```bash
az keyvault set-policy \
  --name my-keyvault \
  --spn <service-principal-id> \
  --secret-permissions get list
```

### Python SDK

```python
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

credential = DefaultAzureCredential()
client = SecretClient(vault_url="https://my-keyvault.vault.azure.net", credential=credential)

# Get secret
secret = client.get_secret("db-password")
print(f"Secret: {secret.value}")
```

## Kubernetes Integration (ESO)

### AWS Secrets Manager + ESO

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
  target:
    name: db-credentials
  data:
  - secretKey: username
    remoteRef:
      key: prod/database/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database/credentials
      property: password
```

### GCP Secret Manager + ESO

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secrets
spec:
  provider:
    gcpsm:
      projectID: "my-project-123"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production-cluster
          serviceAccountRef:
            name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secrets
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: db-password
      version: latest
```

### Azure Key Vault + ESO

```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secrets
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-keyvault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: app-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-secrets
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: db-password
```

```

### scripts/setup_vault.sh

```bash
#!/bin/bash
#
# setup_vault.sh - Automated Vault installation and configuration
#
# Usage:
#   ./setup_vault.sh [kubernetes|docker|local]
#
# Modes:
#   kubernetes - Deploy Vault on Kubernetes (default)
#   docker     - Run Vault in Docker container
#   local      - Install and run Vault locally

set -e

MODE=${1:-kubernetes}

echo "🔐 Vault Setup Script"
echo "Mode: $MODE"
echo ""

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

info() {
  echo -e "${GREEN}✓${NC} $1"
}

warn() {
  echo -e "${YELLOW}⚠${NC} $1"
}

error() {
  echo -e "${RED}✗${NC} $1"
  exit 1
}

# Check if Vault CLI is installed
check_vault_cli() {
  if ! command -v vault &> /dev/null; then
    warn "Vault CLI not installed. Installing..."

    if [[ "$OSTYPE" == "darwin"* ]]; then
      brew install hashicorp/tap/vault
    elif [[ "$OSTYPE" == "linux-gnu"* ]]; then
      wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
      echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
      sudo apt update && sudo apt install vault
    else
      error "Unsupported OS. Install Vault manually: https://www.vaultproject.io/downloads"
    fi

    info "Vault CLI installed"
  else
    info "Vault CLI already installed ($(vault version))"
  fi
}

# Kubernetes deployment
setup_kubernetes() {
  info "Deploying Vault on Kubernetes..."

  # Check if kubectl is available
  if ! command -v kubectl &> /dev/null; then
    error "kubectl not found. Install kubectl first."
  fi

  # Create namespace
  kubectl create namespace vault --dry-run=client -o yaml | kubectl apply -f -
  info "Namespace 'vault' created"

  # Deploy using Helm (recommended)
  if command -v helm &> /dev/null; then
    info "Using Helm for deployment..."

    helm repo add hashicorp https://helm.releases.hashicorp.com
    helm repo update

    helm install vault hashicorp/vault \
      --namespace vault \
      --set "server.dev.enabled=true" \
      --set "injector.enabled=false"

    info "Vault deployed via Helm"
  else
    # Fallback to manual deployment
    warn "Helm not found. Using manual deployment..."
    kubectl apply -f ../examples/vault-eso-setup/vault-deployment.yaml
    info "Vault deployed via kubectl"
  fi

  # Wait for pod to be ready
  info "Waiting for Vault pod to be ready..."
  kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault --timeout=120s

  # Initialize and unseal (dev mode)
  info "Vault is ready!"
  info "To access Vault:"
  echo "  kubectl port-forward -n vault svc/vault 8200:8200"
  echo "  export VAULT_ADDR='http://localhost:8200'"
  echo "  export VAULT_TOKEN='root'  # Dev mode root token"
}

# Docker deployment
setup_docker() {
  info "Running Vault in Docker (dev mode)..."

  docker run -d \
    --name vault-dev \
    --cap-add=IPC_LOCK \
    -e 'VAULT_DEV_ROOT_TOKEN_ID=root' \
    -p 8200:8200 \
    hashicorp/vault:latest

  info "Vault running in Docker"
  info "Access at: http://localhost:8200"
  info "Root token: root"
  echo ""
  echo "Set environment variables:"
  echo "  export VAULT_ADDR='http://localhost:8200'"
  echo "  export VAULT_TOKEN='root'"
}

# Local installation
setup_local() {
  check_vault_cli

  info "Starting Vault in dev mode..."

  # Create config directory
  mkdir -p ~/.vault

  # Start Vault in background
  nohup vault server -dev \
    -dev-root-token-id=root \
    -dev-listen-address=127.0.0.1:8200 \
    > ~/.vault/vault.log 2>&1 &

  VAULT_PID=$!
  echo $VAULT_PID > ~/.vault/vault.pid

  sleep 2

  info "Vault started (PID: $VAULT_PID)"
  info "Access at: http://localhost:8200"
  info "Root token: root"
  echo ""
  echo "Set environment variables:"
  echo "  export VAULT_ADDR='http://localhost:8200'"
  echo "  export VAULT_TOKEN='root'"
  echo ""
  echo "To stop Vault:"
  echo "  kill $(cat ~/.vault/vault.pid)"
}

# Configure Vault basics
configure_vault() {
  info "Configuring Vault..."

  # Enable KV v2 secrets engine
  vault secrets enable -path=secret kv-v2 || warn "KV v2 already enabled"
  info "KV v2 secrets engine enabled at secret/"

  # Enable database secrets engine
  vault secrets enable database || warn "Database engine already enabled"
  info "Database secrets engine enabled"

  # Enable Kubernetes auth (if in Kubernetes mode)
  if [[ "$MODE" == "kubernetes" ]]; then
    vault auth enable kubernetes || warn "Kubernetes auth already enabled"
    info "Kubernetes auth method enabled"
  fi

  info "Basic configuration complete!"
}

# Main execution
main() {
  check_vault_cli

  case $MODE in
    kubernetes)
      setup_kubernetes
      ;;
    docker)
      setup_docker
      ;;
    local)
      setup_local
      ;;
    *)
      error "Unknown mode: $MODE. Use: kubernetes, docker, or local"
      ;;
  esac

  echo ""
  info "Vault setup complete!"
  echo ""
  echo "Next steps:"
  echo "1. Set environment variables (see above)"
  echo "2. Verify connection: vault status"
  echo "3. Create secrets: vault kv put secret/myapp/config api_key=EXAMPLE"
  echo "4. Read secrets: vault kv get secret/myapp/config"
}

main

```