Back to skills
SkillHub ClubShip Full StackFull StackBackendTesting
ring:dev-delivery-verification
Delivery Verification Gate — verifies that what was requested is actually delivered as reachable, integrated code. Not quality review (Gate 8), not test verification (Gate 9) — this gate answers: "Is every requirement from the original task actually functioning in the running application?" Applies to ANY task type: features, refactors, fixes, infrastructure, API endpoints, middleware, business logic, integrations.
Packaged view
This page reorganizes the original catalog entry around fit, installability, and workflow context first. The original raw source lives below.
Stars
141
Hot score
96
Updated
March 20, 2026
Overall rating
C2.9
Composite score
2.9
Best-practice grade
D42.8
Install command
npx @skill-hub/cli install lerianstudio-ring-dev-delivery-verification
Repository
LerianStudio/ring
Skill path: dev-team/skills/dev-delivery-verification
Delivery Verification Gate — verifies that what was requested is actually delivered as reachable, integrated code. Not quality review (Gate 8), not test verification (Gate 9) — this gate answers: "Is every requirement from the original task actually functioning in the running application?" Applies to ANY task type: features, refactors, fixes, infrastructure, API endpoints, middleware, business logic, integrations.
Open repositoryBest for
Primary workflow: Ship Full Stack.
Technical facets: Full Stack, Backend, Testing.
Target audience: everyone.
License: Unknown.
Original source
Catalog source: SkillHub Club.
Repository owner: LerianStudio.
This is still a mirrored public skill entry. Review the repository before installing into production workflows.
What it helps with
- Install ring:dev-delivery-verification into Claude Code, Codex CLI, Gemini CLI, or OpenCode workflows
- Review https://github.com/LerianStudio/ring before adding ring:dev-delivery-verification to shared team environments
- Use ring:dev-delivery-verification for development workflows
Works across
Claude CodeCodex CLIGemini CLIOpenCode
Favorites: 0.
Sub-skills: 0.
Aggregator: No.
Original source / Raw SKILL.md
---
name: ring:dev-delivery-verification
version: 1.0.0
description: |
Delivery Verification Gate — verifies that what was requested is actually delivered
as reachable, integrated code. Not quality review (Gate 8), not test verification
(Gate 9) — this gate answers: "Is every requirement from the original task actually
functioning in the running application?" Applies to ANY task type: features, refactors,
fixes, infrastructure, API endpoints, middleware, business logic, integrations.
trigger: |
- After Gate 0 (implementation) completes, before advancing to Gate 1
- After any refactoring task claims completion
- When code is generated/scaffolded and needs integration verification
NOT_skip_when: |
- "Code compiles" → Compilation ≠ integration. Dead code compiles.
- "Tests pass" → Unit tests on isolated structs pass without wiring.
- "It's just a struct/interface" → Structs that aren't instantiated are dead code.
- "Wire will happen in next task" → Each task must deliver complete, reachable code.
- "Time pressure" → Unwired code is worse than no code — it creates false confidence.
- "It's a simple task" → Simple tasks still need verification. Partial delivery is not delivery.
sequence:
after: [ring:dev-implementation]
before: [ring:dev-devops]
related:
complementary: [ring:dev-cycle, ring:dev-implementation, ring:verification-before-completion, ring:requesting-code-review]
input_schema:
required:
- name: unit_id
type: string
description: "Task or subtask identifier being verified"
- name: requirements
type: string
description: "Original task requirements or acceptance criteria"
- name: files_changed
type: array
items: string
description: "List of files created or modified by Gate 0"
optional:
- name: gate0_handoff
type: object
description: "Full handoff from Gate 0 implementation"
output_schema:
format: markdown
required_sections:
- name: "Delivery Verification Summary"
pattern: "^## Delivery Verification Summary"
required: true
- name: "Requirement Coverage Matrix"
pattern: "^## Requirement Coverage Matrix"
required: true
- name: "Integration Verification"
pattern: "^## Integration Verification"
required: true
- name: "Dead Code Detection"
pattern: "^## Dead Code Detection"
required: true
- name: "Verdict"
pattern: "^## Verdict"
required: true
- name: "Return to Gate 0"
pattern: "^## Return to Gate 0"
required: true
description: "Mandatory when verdict is PARTIAL or FAIL. Lists specific undelivered requirements with fix instructions for Gate 0."
metrics:
- name: result
type: enum
values: [PASS, FAIL, PARTIAL]
- name: requirements_total
type: integer
- name: requirements_delivered
type: integer
- name: requirements_missing
type: integer
- name: dead_code_items
type: integer
- name: remediation_items
type: integer
description: "Number of fix instructions returned to Gate 0 (0 when PASS)"
---
# Delivery Verification Gate
## The Problem This Solves
Agents generate code that compiles and passes unit tests but doesn't actually deliver what was requested. The code exists in the repo but is never wired, never called, never reachable at runtime. This applies to **any kind of task** — not just infrastructure or observability, but features, API endpoints, business logic, integrations, refactors, everything.
**The pattern is always the same:** agent creates artifacts (structs, functions, handlers, middleware, config, migrations) that look complete in isolation, but are never connected to the application's execution path. The task appears done. It's not.
## The Core Principle
```
REQUESTED ≠ CREATED ≠ DELIVERED
- REQUESTED: What the task/requirement asks for
- CREATED: What files/structs/functions were written
- DELIVERED: What is actually functioning in the running application
Only DELIVERED counts. CREATED without DELIVERED is dead code.
This applies to EVERY task type, not just infrastructure.
```
## The Verification Process
### Step 1: Extract Requirements from the Original Task
Parse the original task (as given to Gate 0) into discrete, verifiable requirements. **Every requirement the task asks for must become a line item.** Don't filter by type — if it was requested, it must be verified.
**Examples across different task types:**
#### Feature task: "Add tenant suspension with email notification"
```
R1: Suspend endpoint accepts tenant ID and reason
R2: Tenant status changes to 'suspended' in database
R3: All active connections for tenant are closed
R4: Email notification sent to tenant admin
R5: Subsequent API calls from suspended tenant return 403
R6: Suspension is logged with audit trail
```
#### Refactor task: "Migrate authentication from custom JWT to lib-auth middleware"
```
R1: lib-auth middleware replaces custom JWT validation
R2: All protected routes use new middleware
R3: Custom JWT code removed (no dead code left)
R4: Token format backward compatible (existing tokens still work)
R5: Authorization checks use lib-auth's Authorize() method
R6: M2M flow uses lib-auth's GetApplicationToken()
```
#### Fix task: "Fix race condition in connection pool during tenant eviction"
```
R1: Mutex protects concurrent access to connections map
R2: Eviction checks connection state before closing
R3: In-flight requests complete before connection is removed
R4: Test reproduces the race condition (fails without fix)
R5: Test passes with the fix applied
```
#### Infrastructure task: "Add Redis caching to tenant config lookups"
```
R1: Cache interface defined with Get/Set/Invalidate
R2: In-memory implementation as default (zero config)
R3: Redis implementation as opt-in
R4: Client uses cache before HTTP fallback
R5: Cache TTL is configurable
R6: Cache invalidation on config change
R7: Cache metrics (hit/miss) recorded
```
**Key insight:** Every task has "code exists" requirements AND "code is integrated" requirements. Agents consistently complete the first and skip the second. This gate catches that gap.
### Step 2: Verify Each Requirement is DELIVERED (not just CREATED)
For each requirement, answer THREE questions:
1. **Does the code exist?** (file, struct, function, migration, config)
2. **Is it connected?** (called, imported, registered, wired, injected)
3. **Is it reachable at runtime?** (trace path from main() or entry point)
```
Example verification:
R1: "Suspend endpoint accepts tenant ID and reason"
1. Code exists? → handler.go:SuspendTenant() ✅
2. Connected? → RegisterRoutes() includes DELETE /tenants/:id/suspend?
→ grep -rn "Suspend" internal/bootstrap/wire*.go internal/adapters/http/
→ Found in RegisterRoutes() ✅
3. Reachable? → main→InitServers→initHTTPServer→RegisterRoutes→SuspendTenant ✅
→ Status: ✅ DELIVERED
R4: "Email notification sent to tenant admin"
1. Code exists? → notifier.go:SendSuspensionEmail() ✅
2. Connected? → Called from SuspendTenant handler?
→ grep -rn "SendSuspensionEmail" internal/ --include="*.go" | grep -v test
→ 0 matches outside tests ❌
3. Reachable? → N/A (not connected)
→ Status: ❌ CREATED NOT DELIVERED — function exists but never called
```
### Step 3: Integration Verification Checklist
For **every new artifact** created by Gate 0, verify integration based on its type:
#### Structs & Types
```
For each new struct:
[ ] Is it instantiated somewhere? (grep for NewXxx or &Xxx{})
[ ] Is the instance stored/used? (not just _ = NewXxx())
[ ] Is it reachable from main() → bootstrap → wire → handler chain?
```
#### Functions & Methods
```
For each new exported function:
[ ] Is it called from at least one non-test file?
[ ] Is the caller itself reachable from main()?
[ ] If it's a method: is the receiver struct instantiated and used?
```
#### API Endpoints & Routes
```
For each new endpoint:
[ ] Is the handler function registered in the router?
[ ] Is the route path correct and matches the spec?
[ ] Are required middleware applied (auth, validation, telemetry)?
[ ] Does a request to this endpoint reach the handler? (trace route registration)
```
#### Middleware
```
For each new middleware:
[ ] Is it registered in the router/fiber middleware chain?
[ ] Is it registered in the correct ORDER? (e.g., telemetry before auth)
[ ] Are its dependencies injected? (not nil at runtime)
```
#### Database Changes
```
For each new migration/schema change:
[ ] Is the migration file created with up AND down?
[ ] Is the new column/table used by at least one repository method?
[ ] Is the repository method called by a service?
[ ] Is the service called by a handler or consumer?
```
#### Interfaces & Implementations
```
For each new interface implementation:
[ ] Is the concrete type registered/injected via DI/wire?
[ ] Is the interface actually used at a call site?
[ ] Is the call site reachable from main()?
```
#### Config & Environment Variables
```
For each new config field:
[ ] Is it read from environment? (env tag or os.Getenv)
[ ] Is it passed to the component that uses it?
[ ] Does the component's behavior change based on the value?
```
#### Event Publishers & Consumers
```
For each new event:
[ ] Is the publisher called at the right business moment?
[ ] Is the consumer registered and listening?
[ ] Is the event schema consistent between publisher and consumer?
```
#### Dependencies (go.mod / package.json)
```
For each new dependency added:
[ ] Is it imported in at least one non-test file?
[ ] Is the import used (not just side-effect import)?
```
### Step 3.5: Automated Standards Checks (MANDATORY)
**These checks run on ALL files created/modified by Gate 0. Any failure = PARTIAL verdict.**
#### A. File Size Verification
See [shared-patterns/file-size-enforcement.md](../shared-patterns/file-size-enforcement.md)
**Go** (matches shared-patterns/file-size-enforcement.md):
```bash
find . -name "*.go" \
! -name "*_test.go" \
! -path "*/docs/*" \
! -path "*/mocks*" ! -path "*/generated/*" ! -path "*/gen/*" \
! -name "*.pb.go" ! -name "*.gen.go" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
# Also check test files separately (same threshold)
find . -name "*_test.go" \
! -path "*/mocks*" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
```
**TypeScript** (matches shared-patterns/file-size-enforcement.md):
```bash
find . \( -name "*.ts" -o -name "*.tsx" \) \
! -path "*/node_modules/*" ! -path "*/dist/*" ! -path "*/build/*" \
! -path "*/out/*" ! -path "*/.next/*" \
! -path "*/generated/*" ! -path "*/__generated__/*" \
! -path "*/__mocks__/*" ! -path "*/mocks/*" \
! -name "*.d.ts" ! -name "*.gen.ts" ! -name "*.generated.ts" ! -name "*.mock.ts" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
```
- Any modified file > 500 lines → **FAIL** (hard block, return to Gate 0 with split instructions)
- Any modified file > 300 lines → **PARTIAL** (return to Gate 0 with split instructions)
- **This check applies to ALL files in the project, not just files changed by Gate 0.** Existing oversized files are flagged but do not block unless they were modified by Gate 0.
#### B. License Header Verification
**Reference:** core.md → License Headers (MANDATORY)
```bash
# Check all files created/modified by Gate 0 for license headers
# Patterns checked: Copyright, Licensed, SPDX, License (covers Apache, MIT, etc.)
for f in $files_changed; do
if echo "$f" | grep -qE '\.(go|ts|tsx)$'; then
if ! head -10 "$f" | grep -qiE 'copyright|licensed|spdx|license'; then
echo "MISSING LICENSE HEADER: $f"
fi
fi
done
```
**Note:** Checks first 10 lines (not 5) to account for build tags, package declarations, or shebang lines that may precede the license block. The pattern is intentionally broad (`copyright|licensed|spdx|license`) to match common formats (Apache, MIT, BSD, SPDX identifiers).
- Any source file missing license header → **PARTIAL** (return to Gate 0: "Add license header per core.md")
#### C. Linting Verification
**Reference:** quality.md → Linting (MANDATORY — 14 linters)
```bash
# Go: Run golangci-lint
if [ -f .golangci.yml ] || [ -f .golangci.yaml ]; then
golangci-lint run ./...
else
echo "WARNING: Missing .golangci.yml — quality.md requires it (14 mandatory linters)"
echo "FLAG: PARTIAL — create .golangci.yml per quality.md → Linting (MANDATORY)"
fi
# TypeScript: Run eslint
if [ -f .eslintrc.js ] || [ -f .eslintrc.json ] || [ -f eslint.config.js ] || [ -f .eslintrc.yaml ] || [ -f .eslintrc.yml ]; then
npx eslint . --ext .ts,.tsx
else
echo "WARNING: Missing eslint config — typescript.md requires linting"
echo "FLAG: PARTIAL — create eslint config per typescript.md standards"
fi
```
- Lint failures → **PARTIAL** (return to Gate 0: "Fix lint issues: [errors]")
- Missing linter config in Go project → **PARTIAL** (quality.md requires .golangci.yml with 14 mandatory linters)
- Missing linter config in TypeScript project → **PARTIAL** (typescript.md requires eslint)
#### D. Migration Safety Verification
**Reference:** [migration-safety.md](../../docs/standards/golang/migration-safety.md) — Dangerous Operations Detection
This check only runs when the current branch contains new or modified SQL migration files.
```bash
# Step D.1: Detect migration files in this branch
# Only match files in migrations/ directories (not arbitrary .sql files like test fixtures)
base_branch=$(git rev-parse --abbrev-ref HEAD@{upstream} 2>/dev/null | sed 's|origin/||' || echo "main")
migration_files=$(git diff --name-only "origin/$base_branch" -- '**/migrations/*.sql' 2>/dev/null | grep -v "_test")
if [ -z "$migration_files" ]; then
echo "NO_MIGRATIONS — Step 3.5D skipped"
else
echo "Migration files found: $migration_files"
blocking=0
# Step D.2: Check for blocking operations
for f in $migration_files; do
# ADD COLUMN ... NOT NULL without DEFAULT (unsafe — table rewrite + lock)
# Allows: ADD COLUMN ... NOT NULL DEFAULT, ALTER COLUMN SET NOT NULL (constraint-only, safe after backfill)
if grep -Pin "ADD\s+COLUMN\b.*\bNOT\s+NULL\b" "$f" | grep -Piv "DEFAULT|SET\s+NOT\s+NULL"; then
echo "⛔ BLOCKING: $f — ADD COLUMN NOT NULL without DEFAULT (table rewrite + lock)"
blocking=1
fi
# DROP COLUMN
if grep -Pin "DROP\s+COLUMN" "$f"; then
echo "⛔ BLOCKING: $f — DROP COLUMN (use expand-contract: deprecate first, drop in next release)"
blocking=1
fi
# DROP TABLE without safety
if grep -Pin "DROP\s+TABLE" "$f" | grep -Piv "IF EXISTS.*deprecated"; then
echo "⛔ BLOCKING: $f — DROP TABLE (rename to _deprecated first)"
blocking=1
fi
# TRUNCATE
if grep -Pin "TRUNCATE" "$f"; then
echo "⛔ BLOCKING: $f — TRUNCATE TABLE (never in production migrations)"
blocking=1
fi
# CREATE INDEX without CONCURRENTLY
if grep -Pin "CREATE\s+(UNIQUE\s+)?INDEX\b" "$f" | grep -Piv "CONCURRENTLY"; then
echo "⛔ BLOCKING: $f — CREATE INDEX without CONCURRENTLY (locks writes)"
blocking=1
fi
# ALTER COLUMN TYPE
if grep -Pin "ALTER\s+COLUMN.*TYPE\b" "$f"; then
echo "⛔ BLOCKING: $f — ALTER COLUMN TYPE (table rewrite, use add-new-column pattern)"
blocking=1
fi
done
# Step D.3: Check DOWN migration exists
for f in $migration_files; do
base=$(basename "$f")
dir=$(dirname "$f")
if echo "$base" | grep -q "\.up\.sql$"; then
down_file="${base/.up.sql/.down.sql}"
if [ ! -f "$dir/$down_file" ]; then
echo "⛔ BLOCKING: $f — Missing DOWN migration ($down_file)"
blocking=1
elif [ ! -s "$dir/$down_file" ]; then
echo "⛔ BLOCKING: $f — DOWN migration is empty ($down_file)"
blocking=1
fi
fi
done
# Step D.4: Check idempotency (multi-tenant safety)
for f in $migration_files; do
if grep -Pin "CREATE\s+(TABLE|INDEX)" "$f" | grep -Piv "IF NOT EXISTS|CONCURRENTLY"; then
echo "⚠️ WARNING: $f — DDL without IF NOT EXISTS (not idempotent for multi-tenant re-runs)"
fi
done
if [ "$blocking" -eq 1 ]; then
echo "MIGRATION_SAFETY: ⛔ FAIL — return to Gate 0 with migration-safety.md"
else
echo "MIGRATION_SAFETY: ✅ PASS"
fi
fi
```
- Any blocking operation → **FAIL** (hard block, return to Gate 0 with `migration-safety.md` reference)
- Missing/empty DOWN migration → **FAIL**
- Non-idempotent DDL → **WARNING** (flags but does not block)
- No migration files in branch → **SKIP** (check does not apply)
#### E. Dependency Vulnerability Scanning
**Reference:** [core.md § Dependency Management](../../docs/standards/golang/core.md)
This check runs on every cycle to detect known vulnerabilities in dependencies.
```bash
# Step E.1: Detect project language
if [ -f "go.mod" ]; then
lang="go"
elif [ -f "package.json" ]; then
lang="typescript"
else
echo "VULN_SCAN: ⚠️ SKIP — no go.mod or package.json found"
lang="unknown"
fi
# Step E.2: Run vulnerability scanner
if [ "$lang" = "go" ]; then
# govulncheck scans for known CVEs in Go dependencies
if command -v govulncheck &>/dev/null; then
vuln_output=$(govulncheck ./... 2>&1)
vuln_exit=$?
if [ $vuln_exit -ne 0 ]; then
echo "$vuln_output"
# Parse severity — govulncheck reports all as actionable
echo "VULN_SCAN: ⛔ FAIL — govulncheck found vulnerabilities"
else
echo "VULN_SCAN: ✅ PASS — no known vulnerabilities"
fi
else
echo "VULN_SCAN: ⚠️ WARNING — govulncheck not installed (go install golang.org/x/vuln/cmd/govulncheck@latest)"
fi
# Also verify module integrity
go_verify=$(go mod verify 2>&1)
if [ $? -ne 0 ]; then
echo "⛔ BLOCKING: go mod verify failed — module integrity compromised"
echo "$go_verify"
fi
elif [ "$lang" = "typescript" ]; then
# npm audit for Node.js projects
if [ -f "package-lock.json" ]; then
audit_output=$(npm audit --audit-level=high 2>&1)
audit_exit=$?
if [ $audit_exit -ne 0 ]; then
echo "$audit_output"
echo "VULN_SCAN: ⛔ FAIL — npm audit found high/critical vulnerabilities"
else
echo "VULN_SCAN: ✅ PASS — no high/critical vulnerabilities"
fi
elif [ -f "yarn.lock" ]; then
audit_output=$(yarn audit --level high 2>&1)
audit_exit=$?
if [ $audit_exit -ne 0 ]; then
echo "$audit_output"
echo "VULN_SCAN: ⛔ FAIL — yarn audit found high/critical vulnerabilities"
else
echo "VULN_SCAN: ✅ PASS"
fi
fi
fi
```
- Go: `govulncheck` finds vulnerability → **FAIL** (return to Gate 0: "Update vulnerable dependency or find alternative")
- Go: `go mod verify` fails → **FAIL** (module tampered)
- TypeScript: `npm audit --audit-level=high` finds high/critical → **FAIL**
- Scanner not installed → **WARNING** (does not block, but flags for team to install)
#### F. API Backward Compatibility (oasdiff + swaggo)
**Reference:** [api-patterns.md § OpenAPI Documentation](../../docs/standards/golang/api-patterns.md#openapi-documentation-swaggo-mandatory)
This check only applies to services that have swaggo-generated OpenAPI specs (api/swagger.yaml or api/swagger.json).
```bash
# Step F.1: Check if this is an API service with OpenAPI spec
spec_file=""
if [ -f "api/swagger.yaml" ]; then
spec_file="api/swagger.yaml"
elif [ -f "api/swagger.json" ]; then
spec_file="api/swagger.json"
fi
if [ -z "$spec_file" ]; then
echo "API_COMPAT: ⚠️ SKIP — no OpenAPI spec found (not an API service or swaggo not configured)"
else
# Step F.2: Regenerate spec from current annotations
if command -v swag &>/dev/null; then
swag init -g cmd/api/main.go -o api/ --parseDependency --parseInternal 2>/dev/null
fi
# Step F.3: Get the spec from main branch for comparison
main_spec=$(git show origin/main:$spec_file 2>/dev/null)
if [ -z "$main_spec" ]; then
echo "API_COMPAT: ⚠️ SKIP — no spec on main branch (new service, nothing to compare)"
else
# Step F.4: Run oasdiff breaking change detection
if command -v oasdiff &>/dev/null; then
# Write main spec to temp file for comparison
tmp_main=$(mktemp)
echo "$main_spec" > "$tmp_main"
breaking_output=$(oasdiff breaking "$tmp_main" "$spec_file" 2>&1)
breaking_exit=$?
rm -f "$tmp_main"
if [ $breaking_exit -ne 0 ] || [ -n "$breaking_output" ]; then
echo "$breaking_output"
echo ""
echo "API_COMPAT: ⛔ FAIL — breaking changes detected in API spec"
echo "Review each change above. If intentional (new API version), document in PR description."
else
echo "API_COMPAT: ✅ PASS — no breaking changes in API spec"
fi
else
echo "API_COMPAT: ⚠️ WARNING — oasdiff not installed (go install github.com/tufin/oasdiff@latest)"
fi
fi
fi
```
- Breaking change detected → **FAIL** (return to Gate 0: "Breaking API change — use additive-only changes or document version bump")
- No spec file → **SKIP** (not an API service)
- No spec on main → **SKIP** (new service, no baseline)
- oasdiff not installed → **WARNING** (does not block, flags for installation)
- Non-breaking changes (new endpoints, new optional fields) → **PASS**
#### G. Multi-Tenant Dual-Mode Verification (Go backend only)
**Reference:** [multi-tenant.md](../../docs/standards/golang/multi-tenant.md), [dev-multi-tenant SKILL.md § Sub-Package Import Reference](../dev-multi-tenant/SKILL.md)
This check only applies to Go backend services. It verifies that all resource access uses lib-commons v4 resolvers (which work transparently in both single-tenant and multi-tenant mode).
```bash
# Step G.1: Detect if this is a Go project
if [ ! -f "go.mod" ]; then
echo "MT_DUALMODE: ⚠️ SKIP — not a Go project"
exit 0
fi
blocking=0
# Step G.2: Check PostgreSQL — must use resolvers, not direct GetDB()
pg_direct=$(grep -rn "\.GetDB()" internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "// deprecated\|// legacy\|// TODO" \
| grep -v "core\.Resolve\|tmpostgres\|Manager")
if [ -n "$pg_direct" ]; then
echo "⛔ BLOCKING: Direct .GetDB() calls found — must use core.ResolvePostgres(ctx, r.connection)"
echo "$pg_direct"
blocking=1
fi
# Step G.3: Check MongoDB — must use resolvers
mongo_direct=$(grep -rn "\.GetDatabase()\|\.Database()" internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "core\.Resolve\|tmmongo\|Manager")
if [ -n "$mongo_direct" ]; then
echo "⛔ BLOCKING: Direct MongoDB access found — must use core.ResolveMongo(ctx, r.mongoConn)"
echo "$mongo_direct"
blocking=1
fi
# Step G.4: Check Redis/Valkey — keys must use GetKeyFromContext
redis_hardcoded=$(grep -rn '\.Set\(\s*"[^"]*"\s*,' internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "GetKeyFromContext\|valkey\.")
redis_hardcoded2=$(grep -rn '\.Get\(\s*"[^"]*"\s*[,)]' internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "GetKeyFromContext\|valkey\.\|Getenv\|flag\.")
if [ -n "$redis_hardcoded" ] || [ -n "$redis_hardcoded2" ]; then
echo "⚠️ WARNING: Possible hardcoded Redis keys — verify valkey.GetKeyFromContext is used"
echo "$redis_hardcoded"
echo "$redis_hardcoded2"
fi
# Step G.5: Check S3 — keys must use GetObjectStorageKeyForTenant
s3_hardcoded=$(grep -rn 'PutObject\|GetObject\|DeleteObject' internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "GetObjectStorageKeyForTenant\|s3\.")
if [ -n "$s3_hardcoded" ]; then
echo "⚠️ WARNING: Possible hardcoded S3 keys — verify s3.GetObjectStorageKeyForTenant is used"
echo "$s3_hardcoded"
fi
# Step G.6: Check RabbitMQ — must use tmrabbitmq.Manager
if grep -rq "rabbitmq\|amqp" go.mod 2>/dev/null; then
rmq_direct=$(grep -rn "amqp\.Dial\|channel\.Publish\|channel\.Consume" internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "tmrabbitmq\|Manager")
if [ -n "$rmq_direct" ]; then
echo "⛔ BLOCKING: Direct RabbitMQ access found — must use tmrabbitmq.Manager"
echo "$rmq_direct"
blocking=1
fi
fi
# Step G.7: Check route registration — tenant middleware must use WhenEnabled
if grep -rq "TenantMiddleware\|MultiPoolMiddleware" internal/ pkg/ --include="*.go" 2>/dev/null; then
global_use=$(grep -rn "app\.Use.*[Tt]enant\|app\.Use.*[Mm]ulti[Pp]ool" internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go")
if [ -n "$global_use" ]; then
echo "⛔ BLOCKING: Tenant middleware registered globally (app.Use) — must use per-route WhenEnabled()"
echo "$global_use"
blocking=1
fi
fi
# Step G.8: Check context propagation — all exported methods must accept ctx
no_ctx=$(grep -rn "^func (r \*.*) [A-Z].*(" internal/adapters/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go" \
| grep -v "ctx context\.Context\|Close()\|String()\|Error()")
if [ -n "$no_ctx" ]; then
echo "⚠️ WARNING: Exported methods without ctx parameter found — needed for MT resolution"
echo "$no_ctx" | head -10
fi
# Step G.9: Check global DB singletons
global_db=$(grep -rn "^var.*sql\.DB\|^var.*pgx\.Pool\|^var.*mongo\.Client\|^var.*redis\.Client" internal/ pkg/ --include="*.go" 2>/dev/null \
| grep -v "_test.go")
if [ -n "$global_db" ]; then
echo "⛔ BLOCKING: Global database singletons found — must use struct fields with constructor injection"
echo "$global_db"
blocking=1
fi
if [ "$blocking" -eq 1 ]; then
echo "MT_DUALMODE: ⛔ FAIL — return to Gate 0 with multi-tenant.md"
else
echo "MT_DUALMODE: ✅ PASS — all resources use dual-mode resolvers"
fi
```
- Direct `.GetDB()` or `.GetDatabase()` calls → **FAIL** (must use resolvers)
- Direct RabbitMQ channel operations → **FAIL** (must use tmrabbitmq.Manager)
- Global tenant middleware (app.Use) → **FAIL** (must use per-route WhenEnabled)
- Global DB singletons → **FAIL** (must use struct fields)
- Hardcoded Redis/S3 keys → **WARNING** (may be false positive, verify manually)
- Missing ctx on exported methods → **WARNING** (informational)
- Not a Go project → **SKIP**
**Verdict integration:** ALL seven checks (A, B, C, D, E, F, G) must pass for overall PASS. Any FAIL in checks D, E, F, or G → overall verdict is **FAIL** (hard block, return to Gate 0). Any FAIL in checks A, B, C → overall verdict is **PARTIAL** (return to Gate 0 with fix instructions, max 2 retries). SKIP checks do not affect the verdict. WARNING checks are informational.
### Step 4: Dead Code Detection
Identify any code created by Gate 0 that is not reachable:
#### Go
```bash
# Use files_changed from Gate 0 handoff (NOT git diff — avoids drift on stacked/squashed commits)
# files_changed is provided as input to this gate
if [ -z "$files_changed" ]; then
echo "ERROR: files_changed not provided. Cannot verify delivery."
exit 1
fi
changed_files=$(echo "$files_changed" | tr ',' '\n' | grep "\.go$" | grep -v "_test.go")
for f in $changed_files; do
# Extract exported function/method names
grep -oP 'func (\(.*?\) )?(\K[A-Z]\w+)' "$f" | while read func_name; do
# Count non-test references across the repo
refs=$(grep -rn "$func_name" --include="*.go" . | grep -v "_test.go" | grep -v "^$f:" | wc -l)
if [ "$refs" -eq 0 ]; then
echo "DEAD: $func_name in $f (defined but never referenced outside tests)"
fi
done
done
```
```bash
# For each changed file, check if its package is imported by bootstrap/wire/main
for f in $changed_files; do
pkg_path=$(dirname "$f")
importers=$(grep -rn "\".*$pkg_path\"" internal/bootstrap/ cmd/ --include="*.go" | grep -v "_test.go")
if [ -z "$importers" ]; then
echo "WARNING: Package $pkg_path not imported by bootstrap/cmd — code may be unreachable"
fi
done
```
#### TypeScript
```bash
# Use files_changed from Gate 0 handoff
changed_ts_files=$(echo "$files_changed" | tr ',' '\n' | grep "\.ts$" | grep -v "\.test\.\|\.spec\.")
for f in $changed_ts_files; do
grep -oP 'export (function|class|const|interface) \K\w+' "$f" | while read name; do
refs=$(grep -rn "import.*$name\|require.*$name" --include="*.ts" src/ | grep -v "$f" | wc -l)
if [ "$refs" -eq 0 ]; then
echo "DEAD: $name in $f (exported but never imported)"
fi
done
done
```
### Step 5: Requirement Coverage Matrix
Build and output the full matrix — one row per requirement extracted in Step 1:
```markdown
## Requirement Coverage Matrix
| # | Requirement | Created | Connected | Reachable | Status |
|---|-------------|---------|-----------|-----------|--------|
| R1 | [requirement text] | ✅ file:line | ✅ called by X | ✅ main→...→here | ✅ DELIVERED |
| R2 | [requirement text] | ✅ file:line | ❌ never called | ❌ dead code | ❌ NOT DELIVERED |
| R3 | [requirement text] | ❌ not found | — | — | ❌ NOT CREATED |
```
**Every requirement from Step 1 MUST appear in this matrix.** No filtering, no "this one is obvious." If it was requested, it gets a row.
### Step 6: Verdict
```
PASS: ALL requirements have status DELIVERED
AND dead code count = 0
AND all integration checks pass
AND no modified file exceeds 300 lines (file-size-enforcement.md)
AND all modified source files have license headers (core.md)
AND linting passes (quality.md)
PARTIAL: Some requirements DELIVERED, some NOT DELIVERED
→ List specific gaps with fix instructions
→ Agent MUST fix before advancing to Gate 1
FAIL: Critical requirements NOT DELIVERED
OR majority of requirements NOT DELIVERED
OR significant dead code introduced
→ Return to Gate 0 with explicit instructions
```
**Verdict is based on the original task requirements, not on code quality.** Quality is Gate 8's job. This gate only answers: "Was the requested work actually delivered?"
## Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "The struct is there, wiring is a separate task" | Unwired struct = dead code. Each task must deliver complete functionality. | **Wire it NOW or mark task as incomplete** |
| "Tests prove it works" | Tests on isolated code prove it works in isolation. Not that it works in the app. | **Verify reachability from main()** |
| "It compiles, so it's integrated" | Compilation doesn't prove integration. Unused code compiles. | **Trace call path from main()** |
| "The next PR will wire it" | The next PR is not guaranteed. Deliver complete or don't deliver. | **Complete integration in this task** |
| "It's just scaffolding" | Scaffolding without integration is dead code with extra steps. | **Either wire it or don't create it** |
| "Review will catch it" | Review catches quality issues, not delivery completeness. Different concerns. | **Verify delivery explicitly** |
| "Go vet would catch unused code" | Go vet catches unused variables, not unused exported types/functions. | **Run integration verification** |
| "The feature works, just missing one small piece" | Partial delivery ≠ delivery. If R4 of 7 requirements is missing, task is incomplete. | **Deliver ALL requirements** |
| "That requirement was implied, not explicit" | If the task says it, verify it. Ambiguity → ask, don't skip. | **Verify or clarify with requester** |
| "I did the hard part, the wiring is trivial" | Trivial ≠ done. If it's trivial, do it now. | **Complete the trivial wiring** |
## Integration with dev-cycle
This gate runs as **Gate 0.5** — after implementation (Gate 0), before DevOps (Gate 1):
```
Gate 0: Implementation (write code)
Gate 0.5: Delivery Verification (verify ALL requested work is delivered) ← THIS GATE
Gate 1: DevOps (infrastructure)
Gate 2: SRE (reliability)
Gate 3: Unit Testing
...
```
If Gate 0.5 returns PARTIAL or FAIL:
1. List ALL undelivered requirements with specific evidence
2. Return to Gate 0 with explicit instructions: "Deliver the following: [list with file:line references]"
3. Re-run Gate 0.5 after fixes
4. Only advance to Gate 1 when Gate 0.5 returns PASS
**Gate 0.5 is language-agnostic and task-type-agnostic.** It works the same whether the task is a Go API endpoint, a TypeScript frontend component, a database migration, an infrastructure change, or a business logic refactor. The process is always: extract requirements → verify each is delivered → report gaps.
---
## Referenced Files
> The following files are referenced in this skill and included for context.
### ../shared-patterns/file-size-enforcement.md
```markdown
# File Size Enforcement (MANDATORY)
## Standard Reference
**Source:** `golang/domain.md` → File Organization (MANDATORY), `typescript.md` → File Organization (MANDATORY)
**Rule:** Max 200-300 lines per file. If longer, split by responsibility boundaries.
This is a **HARD GATE** — not a suggestion.
---
## Thresholds
| Lines | Action |
|-------|--------|
| ≤ 300 | ✅ Compliant |
| 301-500 | ⚠️ WARNING — Must split before proceeding (301+ triggers gate loop-back) |
| > 500 | ❌ BLOCKING — Must split before gate can pass |
**These thresholds apply to ALL source files** (`*.go`, `*.ts`, `*.tsx`) **including test files**. Auto-generated files (swagger, protobuf, mocks, `*.pb.go`, `*.gen.ts`, `*.generated.ts`, `*.d.ts`) are exempt.
**Gate 0 enforcement:** Any non-exempt file > 300 lines after implementation = loop back to agent with split instructions. This is not just a 500-line hard block — the 300-line cap is enforced.
---
## When to Check
| Context | Check Point |
|---------|-------------|
| **ring:dev-cycle Gate 0** | After implementation agent completes — verify no file exceeds 300 lines; >300 = loop back; >500 = hard block |
| **ring:dev-cycle Gate 0.5** | Delivery verification MUST run file-size verification command and fail if any file > 300 lines |
| **ring:dev-cycle Gate 8** | Code reviewers MUST flag any file > 300 lines as a MEDIUM+ issue |
| **ring:dev-refactor Step 4** | Agents MUST flag files > 300 lines as ISSUE-XXX |
| **ring:dev-implementation** | Agent MUST NOT create files > 300 lines. If a task would make a file exceed 300 lines, agent MUST split proactively |
---
## Verification Commands
```bash
# Go projects — excludes tests, docs, mocks, protobuf, generated files
# Note: awk filters out the "total" row emitted by wc when multiple files are counted
find . -name "*.go" \
! -name "*_test.go" \
! -path "*/docs/*" \
! -path "*/mocks*" \
! -path "*/generated/*" \
! -path "*/gen/*" \
! -name "*.pb.go" \
! -name "*.gen.go" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
# Go test files (checked separately — same 300-line threshold)
find . -name "*_test.go" \
! -path "*/mocks*" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
# TypeScript projects — excludes node_modules, dist, build, generated, declaration files, mocks
find . \( -name "*.ts" -o -name "*.tsx" \) \
! -path "*/node_modules/*" \
! -path "*/dist/*" \
! -path "*/build/*" \
! -path "*/out/*" \
! -path "*/.next/*" \
! -path "*/generated/*" \
! -path "*/__generated__/*" \
! -path "*/__mocks__/*" \
! -path "*/mocks/*" \
! -name "*.d.ts" \
! -name "*.gen.ts" \
! -name "*.generated.ts" \
! -name "*.mock.ts" \
-exec wc -l {} + | awk '$1 > 300 && $NF != "total" {print}' | sort -rn
```
---
## Split Strategy
When a file exceeds the threshold, split by **responsibility boundaries** (not arbitrary line counts):
### Go
| Pattern | Split Into |
|---------|-----------|
| CRUD + validation + business logic | `*_command.go`, `*_query.go`, `*_validator.go` |
| Provisioning + deprovisioning | `*_provision.go`, `*_deprovision.go` |
| Handler with settings + cache + CRUD | `*_handler.go`, `*_handler_settings.go`, `*_handler_cache.go` |
| Service with lifecycle + helpers | `*_lifecycle.go`, `*_helpers.go` |
| Large test file | Split test file to mirror source file split |
**Go rules:**
1. All split files stay in the **same package** — zero breaking changes
2. All methods remain on the **same receiver** (if applicable)
3. Test files split to match: `foo.go` → `foo_test.go`
4. Run `go build ./...` and `go test ./...` after each split to verify
### TypeScript
| Pattern | Split Into |
|---------|-----------|
| Service with CRUD + validation + helpers | `*.service.ts`, `*.validator.ts`, `*.helpers.ts` |
| Controller with routes + middleware + handlers | `*.controller.ts`, `*.middleware.ts` |
| Large module barrel | Split into sub-modules by domain concept |
| Large test file | Split test file to mirror source file split |
**TypeScript rules:**
1. Split files stay in the **same module/directory** — update barrel exports (`index.ts`) if needed
2. Split by logical responsibility (class methods can be extracted to separate service/helper files)
3. Test files split to match: `foo.service.ts` → `foo.service.spec.ts`
4. Run `tsc --noEmit && npm test` after each split to verify
---
## Agent Instructions
### For ring:dev-implementation (Gate 0) — Go
Include in Go implementation agent prompts:
```
⛔ FILE SIZE ENFORCEMENT (MANDATORY):
- You MUST NOT create or modify files to exceed 300 lines (including test files)
- If implementing a feature would push a file past 300 lines, you MUST split it proactively
- Split by responsibility boundaries (not arbitrary line counts)
- Each split file stays in the same package
- All methods remain on the same receiver
- Test files MUST be split to match source files
- After splitting, verify: go build ./... && go test ./...
- Files > 300 lines = loop back for split. Files > 500 lines = HARD BLOCK.
Reference: golang/domain.md → File Organization (MANDATORY)
```
### For ring:dev-implementation (Gate 0) — TypeScript
Include in TypeScript implementation agent prompts:
```
⛔ FILE SIZE ENFORCEMENT (MANDATORY):
- You MUST NOT create or modify files to exceed 300 lines (including test files)
- If implementing a feature would push a file past 300 lines, you MUST split it proactively
- Split by logical responsibility (not arbitrary line counts)
- Update barrel exports (index.ts) if needed after splitting
- Test files MUST be split to match source files
- After splitting, verify: tsc --noEmit && npm test
- Files > 300 lines = loop back for split. Files > 500 lines = HARD BLOCK.
Reference: typescript.md → File Organization (MANDATORY)
```
### For ring:dev-refactor (Step 4 agents)
Include in ALL analysis agent prompts:
```
⛔ FILE SIZE ENFORCEMENT (MANDATORY):
- Any source file > 300 lines (including test files) MUST be flagged as ISSUE-XXX
- Files 301-500 lines: severity HIGH
- Files > 500 lines: severity CRITICAL
- Files > 1000 lines: severity CRITICAL with explicit decomposition plan
- Include line count and proposed split in the finding
- Each file split = one ISSUE-XXX (not grouped)
```
---
## Anti-Rationalization
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "It's all one struct, can't split" | Methods on the same struct can live in different files (same package) | **Split by method responsibility** |
| "File will be split later" | Later = never. Split NOW during implementation. | **Split before gate passes** |
| "It's only 350 lines" | 350 > 300 = non-compliant. Standards are not negotiable. | **Split before proceeding** |
| "Splitting adds complexity" | Large files ARE complexity. Small focused files reduce cognitive load. | **Split by responsibility** |
| "Tests will break" | Split test files to match. Same package = same access. | **Split tests alongside source** |
| "Auto-generated code is large" | Auto-generated files (swagger, protobuf, mocks) are exempt. | **Check if truly auto-generated** |
| "This is a temporary file" | Temporary becomes permanent. Standards apply to all files. | **Split or delete** |
| "Test files don't count" | Large test files are equally hard to maintain. Same threshold applies. | **Split test files to match source** |
```
### ../../docs/standards/golang/migration-safety.md
```markdown
# Go Standards - Migration Safety
> **Module:** migration-safety.md | **Sections:** §1-§5 | **Parent:** [index.md](index.md)
This module covers database migration safety patterns to prevent production incidents caused by schema changes. All migrations MUST be backward-compatible and safe for zero-downtime deployments.
---
## Table of Contents
| # | Section | Description |
|---|---------|-------------|
| 1 | [Principles](#principles) | Core migration safety principles |
| 2 | [Dangerous Operations](#dangerous-operations-detection) | Operations that require special handling |
| 3 | [Expand-Contract Pattern](#expand-contract-pattern-mandatory) | Safe schema evolution strategy |
| 4 | [Multi-Tenant Considerations](#multi-tenant-considerations) | Migration safety in multi-tenant context |
| 5 | [Verification Commands](#verification-commands) | Automated checks for Gate 0.5 |
**Meta-sections:**
- [Anti-Rationalization Table](#anti-rationalization-table) - Common excuses and required actions
- [Checklist](#checklist) - Pre-submission verification
---
## Principles
Database migrations in production fintech systems carry disproportionate risk. A bad migration can:
- Lock tables for minutes, blocking all transactions
- Cause data loss if a column is dropped prematurely
- Break other services that depend on the schema
- Multiply impact in multi-tenant mode (runs per tenant)
### Core Rules
1. **All migrations MUST be backward-compatible.** The previous version of the application must continue working after the migration runs. This enables zero-downtime rolling deployments.
2. **All migrations MUST be idempotent.** Running the same migration twice must produce the same result (critical for multi-tenant where it runs N times).
3. **All migrations MUST have a tested DOWN path.** Rollback is not optional.
4. **One feature = one migration file.** See [core.md § Database Migrations](core.md#database-migrations-mandatory).
---
## Dangerous Operations Detection
The following SQL operations are dangerous in production and require specific handling:
### ⛔ BLOCKING Operations (Gate 0.5 MUST reject)
| Operation | Risk | Safe Alternative |
|-----------|------|------------------|
| `ALTER TABLE ... ADD COLUMN ... NOT NULL` (without DEFAULT) | Rewrites entire table, acquires ACCESS EXCLUSIVE lock | Add as nullable first, backfill, then add constraint |
| `ALTER TABLE ... DROP COLUMN` | Breaks any service still reading that column | Use expand-contract: stop reading → deploy → drop in next release |
| `CREATE INDEX` (without CONCURRENTLY) | Acquires SHARE lock, blocks writes on large tables | `CREATE INDEX CONCURRENTLY` (PostgreSQL) |
| `ALTER TABLE ... ALTER COLUMN TYPE` | Rewrites entire table | Add new column, migrate data, drop old column |
| `LOCK TABLE` | Explicit lock blocks all concurrent access | Never use explicit locks in migrations |
| `DROP TABLE` | Data loss, breaks dependent services | Rename first (`_deprecated_YYYYMMDD`), drop in next release |
| `TRUNCATE TABLE` | Data loss | Never in production migrations |
### ⚠️ WARNING Operations (Gate 0.5 flags, does not block)
| Operation | Risk | Recommendation |
|-----------|------|----------------|
| `ALTER TABLE ... ADD COLUMN ... DEFAULT` | Safe in PostgreSQL 11+ (metadata-only), but verify version | Confirm target PostgreSQL version supports metadata-only ADD COLUMN |
| `ALTER TABLE ... RENAME COLUMN` | May break queries using old name | Prefer add-new + migrate + drop-old |
| `CREATE UNIQUE INDEX CONCURRENTLY` | Can fail and leave invalid index | Check for invalid indexes after migration |
| Large `UPDATE` in migration | May lock rows for extended time | Batch updates (1000-5000 rows per batch with commit) |
### Detection Patterns
```bash
# Detect dangerous operations in new migration files
# Run against files added in the current branch vs main
DANGEROUS_PATTERNS="NOT NULL(?!.*DEFAULT)|DROP COLUMN|DROP TABLE|TRUNCATE|LOCK TABLE|ALTER COLUMN TYPE"
WARNING_PATTERNS="RENAME COLUMN|CREATE UNIQUE INDEX(?!.*CONCURRENTLY)"
INDEX_PATTERN="CREATE INDEX(?!.*CONCURRENTLY)"
# Get new/modified migration files
migration_files=$(git diff --name-only origin/main -- '**/migrations/*.sql' '**/*.sql')
if [ -n "$migration_files" ]; then
for f in $migration_files; do
# BLOCKING checks
if grep -Pn "$DANGEROUS_PATTERNS" "$f" 2>/dev/null; then
echo "⛔ BLOCKING: Dangerous operation in $f"
fi
# Non-concurrent index
if grep -Pn "$INDEX_PATTERN" "$f" 2>/dev/null | grep -v "CONCURRENTLY"; then
echo "⛔ BLOCKING: CREATE INDEX without CONCURRENTLY in $f"
fi
# WARNING checks
if grep -Pn "$WARNING_PATTERNS" "$f" 2>/dev/null; then
echo "⚠️ WARNING: Review required for $f"
fi
done
fi
```
---
## Expand-Contract Pattern (MANDATORY)
All schema changes that modify existing columns or tables MUST follow the expand-contract pattern:
### Phase 1: Expand (Migration N)
- Add new column/table alongside old one
- New column is nullable (no NOT NULL constraint yet)
- Application writes to BOTH old and new
- Application reads from old (backward compatible)
### Phase 2: Migrate (Migration N+1, separate deploy)
- Backfill new column from old column data
- Application reads from new, writes to both
- Verify data consistency
### Phase 3: Contract (Migration N+2, separate deploy)
- Stop writing to old column
- Add NOT NULL constraint if needed (now safe — all rows populated)
- Drop old column in a FUTURE migration (not this one)
### Example: Renaming a Column
```sql
-- Migration 1 (PR #1): EXPAND — add new column, backfill
ALTER TABLE accounts ADD COLUMN account_name VARCHAR(255);
UPDATE accounts SET account_name = name WHERE account_name IS NULL;
-- Application now writes to BOTH name and account_name, reads from account_name
-- Migration 2 (PR #2, separate deploy): CONTRACT — drop old column
-- This PR is allowed to use DROP COLUMN because:
-- 1. The previous deploy confirmed no service reads 'name' anymore
-- 2. The PR description documents the expand-contract sequence
-- 3. Gate 0.5D will flag it — the agent must acknowledge it as intentional
ALTER TABLE accounts DROP COLUMN IF EXISTS name;
```
**Never combine expand and contract in the same migration or same PR.** Each phase must be a separate deployment to allow rollback.
**Note on Gate 0.5D interaction:** The contract phase (DROP COLUMN) will trigger a Gate 0.5D BLOCKING flag. This is by design — the agent must verify the expand phase was already deployed and documented in the PR. The check forces explicit acknowledgment, not blind automation.
---
## Multi-Tenant Considerations
In multi-tenant mode, migrations run once per tenant database (or per tenant schema). This amplifies risk:
| Single-Tenant Risk | Multi-Tenant Risk |
|--------------------|--------------------|
| Table lock for 30s | Table lock for 30s × N tenants |
| Failed migration → rollback 1 DB | Failed migration → partial state across N DBs |
| Index creation blocks writes | Index creation blocks writes for ALL tenants sequentially |
### Multi-Tenant Migration Rules
1. **Migrations MUST be idempotent.** If migration fails on tenant 5 of 20, re-running must skip tenants 1-4 safely.
2. **Use IF NOT EXISTS / IF EXISTS.** All DDL must be conditional.
3. **Timeout per tenant.** Set statement timeout to prevent one tenant's large table from blocking all others.
4. **Log per-tenant progress.** The migration runner must log which tenant is being migrated for debugging partial failures.
```sql
-- ✅ Idempotent migration (safe for re-run)
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_transactions_tenant_date
ON transactions (tenant_id, created_at);
-- ❌ Non-idempotent (fails on re-run)
CREATE INDEX CONCURRENTLY idx_transactions_tenant_date
ON transactions (tenant_id, created_at);
```
---
## Verification Commands
These commands are used by Gate 0.5D (Migration Safety) in the dev-delivery-verification skill.
### Step 1: Detect migration files
```bash
migration_files=$(git diff --name-only origin/main -- '**/migrations/*.sql' '**/*.sql' 2>/dev/null)
if [ -z "$migration_files" ]; then
echo "NO_MIGRATIONS — skip migration safety checks"
exit 0
fi
echo "Found migration files: $migration_files"
```
### Step 2: Check for blocking operations
```bash
blocking=0
for f in $migration_files; do
# NOT NULL without DEFAULT
if grep -Pin "NOT\s+NULL" "$f" | grep -Piv "DEFAULT|CONSTRAINT|CHECK|WHERE|AND|OR"; then
echo "⛔ $f: ADD COLUMN NOT NULL without DEFAULT"
blocking=1
fi
# DROP COLUMN
if grep -Pin "DROP\s+COLUMN" "$f"; then
echo "⛔ $f: DROP COLUMN (use expand-contract pattern)"
blocking=1
fi
# DROP TABLE (not IF EXISTS rename pattern)
if grep -Pin "DROP\s+TABLE" "$f" | grep -Piv "IF EXISTS.*deprecated"; then
echo "⛔ $f: DROP TABLE"
blocking=1
fi
# TRUNCATE
if grep -Pin "TRUNCATE" "$f"; then
echo "⛔ $f: TRUNCATE TABLE"
blocking=1
fi
# CREATE INDEX without CONCURRENTLY
if grep -Pin "CREATE\s+(UNIQUE\s+)?INDEX\b" "$f" | grep -Piv "CONCURRENTLY"; then
echo "⛔ $f: CREATE INDEX without CONCURRENTLY"
blocking=1
fi
# ALTER COLUMN TYPE
if grep -Pin "ALTER\s+COLUMN.*TYPE" "$f"; then
echo "⛔ $f: ALTER COLUMN TYPE (rewrite risk)"
blocking=1
fi
done
if [ "$blocking" -eq 1 ]; then
echo "MIGRATION_SAFETY: ⛔ BLOCKED — fix dangerous operations above"
exit 1
fi
```
### Step 3: Check for DOWN migration
```bash
for f in $migration_files; do
dir=$(dirname "$f")
base=$(basename "$f")
# golang-migrate convention: NNNN_name.up.sql must have NNNN_name.down.sql
if echo "$base" | grep -q "\.up\.sql$"; then
down_file="${base/.up.sql/.down.sql}"
if [ ! -f "$dir/$down_file" ]; then
echo "⛔ $f: Missing DOWN migration ($down_file)"
blocking=1
elif [ ! -s "$dir/$down_file" ]; then
echo "⛔ $f: DOWN migration is empty ($down_file)"
blocking=1
fi
fi
done
```
### Step 4: Check idempotency (for multi-tenant)
```bash
for f in $migration_files; do
# Check DDL statements use IF NOT EXISTS / IF EXISTS
if grep -Pin "CREATE\s+(TABLE|INDEX)" "$f" | grep -Piv "IF NOT EXISTS|CONCURRENTLY"; then
echo "⚠️ $f: DDL without IF NOT EXISTS (not idempotent for multi-tenant re-runs)"
fi
if grep -Pin "DROP\s+(TABLE|INDEX|COLUMN)" "$f" | grep -Piv "IF EXISTS"; then
echo "⚠️ $f: DROP without IF EXISTS (not idempotent)"
fi
done
```
---
## Anti-Rationalization Table
| Excuse | Why It's Wrong | Required Action |
|--------|---------------|-----------------|
| "The table is small" | Tables grow. What's small today may be huge in 6 months. | **Apply safe patterns regardless of current size** |
| "We can just do a maintenance window" | Zero-downtime is the standard. Maintenance windows are not acceptable for SaaS. | **Use expand-contract pattern** |
| "The DOWN migration is trivial, we don't need to test it" | Untested rollbacks fail when you need them most. | **Write and verify DOWN migration** |
| "CONCURRENTLY is slower" | A 2x slower index creation that doesn't block writes is always better than a fast one that does. | **Always use CONCURRENTLY** |
| "Nobody else reads this column" | You don't know that. Other services may query it directly or via views. | **Follow expand-contract: deprecate first, drop later** |
| "We'll fix it in the next migration" | Broken state between migrations can cascade. | **Fix in this migration or don't merge** |
| "Multi-tenant just runs it N times, same risk" | N × risk ≠ same risk. Partial failure across tenants is a nightmare to debug. | **Ensure idempotency with IF NOT EXISTS/IF EXISTS** |
---
## Checklist
Before submitting a migration, verify:
- [ ] No `ALTER TABLE ... ADD COLUMN ... NOT NULL` without `DEFAULT`
- [ ] No `DROP COLUMN` without prior deprecation period
- [ ] No `CREATE INDEX` without `CONCURRENTLY`
- [ ] No `ALTER COLUMN TYPE` (use add-new-column pattern instead)
- [ ] No `TRUNCATE TABLE` or `DROP TABLE` without safety check
- [ ] DOWN migration exists and is non-empty
- [ ] All DDL uses `IF NOT EXISTS` / `IF EXISTS` (idempotent for multi-tenant)
- [ ] Large data migrations use batched updates (not single UPDATE)
- [ ] Expand-contract pattern followed for column modifications
- [ ] Migration tested locally with `migrate up` then `migrate down`
```
### ../../docs/standards/golang/core.md
```markdown
# Go Standards - Core Foundation
> **Module:** core.md | **Sections:** §1-9 | **Parent:** [index.md](index.md)
This module covers the foundational requirements for all Go projects.
---
## Table of Contents
| # | Section | Description |
| --- | ------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| 1 | [Version](#version) | Go version requirements |
| 2 | [Core Dependency: lib-commons](#core-dependency-lib-commons-mandatory) | Required lib-commons v2 integration |
| 3 | [Frameworks & Libraries](#frameworks--libraries) | Required versions, validator v10 migration |
| 4 | [Configuration](#configuration) | Environment variable handling |
| 5 | [Database Naming Convention (snake_case)](#database-naming-convention-snake-case-mandatory) | Table and column naming |
| 6 | [Database Migrations](#database-migrations-mandatory) | golang-migrate requirement |
| 7 | [License Headers](#license-headers-mandatory) | Copyright headers in source files |
| 8 | [MongoDB Patterns](#mongodb-patterns-mandatory) | Injection prevention, pooling, index management |
| 9 | [Dependency Management](#dependency-management-mandatory) | Go modules, version pinning, security updates |
---
## Version
- **Minimum**: Go 1.24
- **Recommended**: Latest stable release
---
## Core Dependency: lib-commons (MANDATORY)
All Lerian Studio Go projects **MUST** use `lib-commons/v2` as the foundation library. This ensures consistency across all services.
### Required Import (lib-commons v2)
```go
import (
libCommons "github.com/LerianStudio/lib-commons/v2/commons"
libZap "github.com/LerianStudio/lib-commons/v2/commons/zap" // Logger initialization (config/bootstrap only)
libLog "github.com/LerianStudio/lib-commons/v2/commons/log" // Logger interface (services, routes, consumers)
libOpentelemetry "github.com/LerianStudio/lib-commons/v2/commons/opentelemetry"
libServer "github.com/LerianStudio/lib-commons/v2/commons/server"
libHTTP "github.com/LerianStudio/lib-commons/v2/commons/net/http"
libPostgres "github.com/LerianStudio/lib-commons/v2/commons/postgres"
libMongo "github.com/LerianStudio/lib-commons/v2/commons/mongo"
libRedis "github.com/LerianStudio/lib-commons/v2/commons/redis"
)
```
> **Note:** v2 uses `lib` prefix aliases (e.g., `libCommons`, `libZap`, `libLog`) to distinguish lib-commons packages from standard library and other imports.
### What lib-commons Provides
| Package | Purpose | Where Used |
| ----------------------- | ------------------------------------------------------ | ------------------------------------- |
| `commons` | Core utilities, config loading, tracking context | Everywhere |
| `commons/zap` | Logger initialization/configuration | **Config/bootstrap files only** |
| `commons/log` | Logger interface (`log.Logger`) for logging operations | Services, routes, consumers, handlers |
| `commons/postgres` | PostgreSQL connection management, pagination | Bootstrap, repositories |
| `commons/mongo` | MongoDB connection management | Bootstrap, repositories |
| `commons/redis` | Redis connection management | Bootstrap, repositories |
| `commons/opentelemetry` | OpenTelemetry initialization and helpers | Bootstrap, middleware |
| `commons/net/http` | HTTP utilities, telemetry middleware, pagination | Routes, handlers |
| `commons/server` | Server lifecycle with graceful shutdown | Bootstrap |
### ⛔ FORBIDDEN: Custom Utilities That Duplicate lib-commons (HARD GATE)
**HARD GATE:** You CANNOT create custom helpers, utilities, or wrappers that duplicate functionality already provided by lib-commons. This is NON-NEGOTIABLE.
#### What lib-commons Already Provides (DO NOT RECREATE)
| Category | lib-commons Provides | FORBIDDEN to Create |
| -------------- | ---------------------------------------------------- | -------------------------------------- |
| **Logging** | `libLog.Logger`, `libZap.NewLogger()` | Custom logger, log wrapper, log helper |
| **Telemetry** | `libOpentelemetry.NewTracerProvider()`, span helpers | Custom tracer, telemetry wrapper |
| **HTTP** | `libHTTP.NewRouter()`, middleware, response helpers | Custom HTTP utils, response formatters |
| **Config** | `libCommons.SetConfigFromEnvVars()` | Custom config loader, env parser |
| **Server** | `libServer.NewServer()`, graceful shutdown | Custom server lifecycle |
| **PostgreSQL** | `libPostgres.Connect()`, pagination, query builders | Custom DB helpers, pagination utils |
| **MongoDB** | `libMongo.Connect()` | Custom Mongo wrapper |
| **Redis** | `libRedis.Connect()` | Custom Redis wrapper |
| **Context** | `libCommons.TrackingContext` | Custom context propagation |
| **Errors** | Error wrapping utilities | Custom error helpers |
#### Detection Commands (Run Before Creating Any Utility)
```bash
# BEFORE creating any utility, search lib-commons first
# Clone or browse: https://github.com/LerianStudio/lib-commons
# Search for existing functionality
grep -rn "func.*Logger" ./vendor/github.com/LerianStudio/lib-commons/
grep -rn "func.*Trace" ./vendor/github.com/LerianStudio/lib-commons/
grep -rn "func.*Config" ./vendor/github.com/LerianStudio/lib-commons/
```
#### FORBIDDEN Patterns
```go
// ❌ FORBIDDEN: Custom logger wrapper
package utils
func NewLogger() *zap.Logger {
// DON'T DO THIS - use libZap.NewLogger()
}
// ❌ FORBIDDEN: Custom telemetry helper
package helpers
func StartSpan(ctx context.Context, name string) (context.Context, trace.Span) {
// DON'T DO THIS - use libOpentelemetry helpers
}
// ❌ FORBIDDEN: Custom config loader
package config
func LoadFromEnv(cfg interface{}) error {
// DON'T DO THIS - use libCommons.SetConfigFromEnvVars()
}
// ❌ FORBIDDEN: Custom HTTP response helper
package utils
func JSONResponse(c *fiber.Ctx, status int, data interface{}) error {
// DON'T DO THIS - use libHTTP response helpers
}
// ❌ FORBIDDEN: Custom pagination utility
package helpers
func Paginate(page, pageSize int) (offset, limit int) {
// DON'T DO THIS - use libPostgres or libHTTP pagination
}
```
#### When Custom Utilities ARE Allowed
| Scenario | Allowed? | Condition |
| ----------------------------------- | -------- | --------------------------------------------------- |
| Functionality exists in lib-commons | ❌ NO | Use lib-commons instead |
| Domain-specific business logic | ✅ YES | Not infrastructure-level |
| lib-commons lacks the feature | ✅ YES | Document why, consider contributing to lib-commons |
| Thin wrapper for testing | ⚠️ MAYBE | Only if it improves testability without duplicating |
#### Verification Checklist (MANDATORY Before Creating Any Utility)
```text
Before creating any file in utils/, helpers/, pkg/common/, or similar:
[ ] 1. Did I search lib-commons for this functionality?
[ ] 2. Does lib-commons have a package that does this?
[ ] 3. If lib-commons has it → USE IT, do not create custom
[ ] 4. If lib-commons lacks it → Is this infrastructure or domain logic?
[ ] 5. If infrastructure → Consider contributing to lib-commons instead
If you checked YES to #2 or #3 → STOP. Use lib-commons.
```
#### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| ----------------------------------------- | --------------------------------------------------------------- | ---------------------------- |
| "My wrapper is simpler" | Simpler ≠ better. Consistency > convenience. | **Use lib-commons** |
| "lib-commons is too complex for this" | Complexity exists for good reasons (telemetry, error handling). | **Use lib-commons** |
| "I need a slightly different interface" | Adapt your code to lib-commons, not the other way around. | **Use lib-commons** |
| "It's just a small helper" | Small helpers grow. Today's helper is tomorrow's tech debt. | **Use lib-commons** |
| "I'll migrate to lib-commons later" | Later = never. Start with lib-commons. | **Use lib-commons now** |
| "The project doesn't use lib-commons yet" | That's the first problem to fix. Add lib-commons dependency. | **Add lib-commons first** |
| "I didn't know lib-commons had this" | Ignorance ≠ excuse. Always search lib-commons before creating. | **Search lib-commons first** |
| "lib-commons version is outdated" | Update lib-commons, don't fork functionality. | **Update dependency** |
---
## Frameworks & Libraries
### Required Versions (Minimum)
| Library | Minimum Version | Purpose |
| -------------------------- | --------------- | ------------------------------------------------ |
| `lib-commons` | v2.0.0 | Core infrastructure |
| `fiber/v2` | v2.52.0 | HTTP framework |
| `pgx/v5` | v5.7.0 | PostgreSQL driver |
| `go.opentelemetry.io/otel` | v1.38.0 | Telemetry |
| `zap` | v1.27.0 | Logging implementation (internal to lib-commons) |
| `testify` | v1.10.0 | Testing |
| `gomock` | v0.5.0 | Mock generation |
| `mongo-driver` | v1.17.0 | MongoDB driver |
| `go-redis/v9` | v9.7.0 | Redis client |
| `validator/v10` | v10.26.0 | Input validation |
### Validator Migration: v9 to v10 (MANDATORY)
Projects using `go-playground/validator/v9` have unmaintained dependencies with known security issues.
**⛔ HARD GATE:** All projects MUST use `validator/v10`. Version v9 is FORBIDDEN and MUST be migrated.
#### Why v10 Is MANDATORY
| Issue | v9 | v10 |
| -------------------- | -------------------------------- | -------------------------------- |
| **Maintenance** | ❌ Unmaintained since 2020 | ✅ Actively maintained |
| **Security** | ❌ Known CVEs unpatched | ✅ Security patches applied |
| **Features** | ❌ Missing modern validations | ✅ New validators, better errors |
| **Go compatibility** | ❌ Issues with Go 1.18+ generics | ✅ Full Go 1.24 support |
#### Detection Commands (MANDATORY)
```bash
# MUST: Check for v9 usage (should return 0 matches)
grep -rn "go-playground/validator/v9" go.mod go.sum
# If found: BLOCKER - Migrate to v10 before proceeding
# Check current validator version
grep "go-playground/validator" go.mod
# Expected: github.com/go-playground/validator/v10 v10.x.x
```
#### Migration Steps
**1. Update go.mod:**
```bash
# Remove v9
go mod edit -droprequire github.com/go-playground/validator/v9
# Add v10
go get github.com/go-playground/validator/v10@latest
```
**2. Update imports in code:**
```go
// ❌ BEFORE: v9 import
import "github.com/go-playground/validator/v9"
// ✅ AFTER: v10 import
import "github.com/go-playground/validator/v10"
```
**3. Handle API changes:**
```go
// ❌ v9: validator.New()
v := validator.New()
// ✅ v10: Same API, new features available
v := validator.New(validator.WithRequiredStructEnabled())
```
**4. Update custom validators:**
```go
// ❌ v9: Old registration pattern
v.RegisterValidation("custom", customValidator)
// ✅ v10: Same pattern, use new error types
v.RegisterValidation("custom", customValidator)
// Access improved error details via v10.ValidationErrors
```
#### Common Migration Issues
| Issue | Solution |
| ------------------------- | ----------------------------------------------- |
| `FieldError` type changed | Use `validator.ValidationErrors` type assertion |
| `StructLevel` changes | Update to `validator.StructLevel` interface |
| Tag format changes | Some tags renamed (check release notes) |
| Custom validators | Re-register with v10 API |
#### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| --------------------------- | ---------------------------------------------------------- | --------------------------- |
| "v9 still works" | Works ≠ maintained. Security vulnerabilities accumulate. | **Migrate to v10** |
| "Migration is risky" | Risk of not migrating is higher (security, compatibility). | **Migrate to v10** |
| "We have custom validators" | Custom validators work with v10. API is compatible. | **Migrate to v10** |
| "Dependencies use v9" | Update dependencies too. Transitive v9 is also vulnerable. | **Update all dependencies** |
| "We'll migrate later" | Later = never. Migrate now while context is fresh. | **Migrate NOW** |
---
### HTTP Framework
| Library | Use Case |
| ------------ | ------------------------------------------ |
| **Fiber v2** | **Primary choice** - High-performance APIs |
| gRPC-Go | Service-to-service communication |
### Database
| Library | Use Case |
| ------------------- | ------------------------ |
| **pgx/v5** | PostgreSQL (recommended) |
| sqlc | Type-safe SQL queries |
| GORM | ORM (when needed) |
| **go-redis/v9** | Redis client |
| **mongo-go-driver** | MongoDB |
### Testing
| Library | Use Case |
| ----------------- | ------------------------------------------- |
| testify | Assertions |
| GoMock | Interface mocking (MANDATORY for all mocks) |
| SQLMock | Database mocking |
| testcontainers-go | Integration tests |
---
## Configuration
All services **MUST** use `libCommons.SetConfigFromEnvVars` for configuration loading.
### 1. Define Configuration Struct
```go
// bootstrap/config.go
package bootstrap
const ApplicationName = "your-service-name"
// Config is the top level configuration struct for the entire application.
type Config struct {
// Application
EnvName string `env:"ENV_NAME"`
LogLevel string `env:"LOG_LEVEL"`
ServerAddress string `env:"SERVER_ADDRESS"`
// Database - Primary
PrimaryDBHost string `env:"DB_HOST"`
PrimaryDBUser string `env:"DB_USER"`
PrimaryDBPassword string `env:"DB_PASSWORD"`
PrimaryDBName string `env:"DB_NAME"`
PrimaryDBPort string `env:"DB_PORT"`
PrimaryDBSSLMode string `env:"DB_SSLMODE"`
// Database - Replica (for read scaling)
ReplicaDBHost string `env:"DB_REPLICA_HOST"`
ReplicaDBUser string `env:"DB_REPLICA_USER"`
ReplicaDBPassword string `env:"DB_REPLICA_PASSWORD"`
ReplicaDBName string `env:"DB_REPLICA_NAME"`
ReplicaDBPort string `env:"DB_REPLICA_PORT"`
ReplicaDBSSLMode string `env:"DB_REPLICA_SSLMODE"`
// Database - Connection Pool
MaxOpenConnections int `env:"DB_MAX_OPEN_CONNS"`
MaxIdleConnections int `env:"DB_MAX_IDLE_CONNS"`
// MongoDB (if needed)
MongoDBHost string `env:"MONGO_HOST"`
MongoDBName string `env:"MONGO_NAME"`
MongoDBUser string `env:"MONGO_USER"`
MongoDBPassword string `env:"MONGO_PASSWORD"`
MongoDBPort string `env:"MONGO_PORT"`
MongoDBParameters string `env:"MONGO_PARAMETERS"`
MaxPoolSize int `env:"MONGO_MAX_POOL_SIZE"`
// Redis
RedisHost string `env:"REDIS_HOST"`
RedisPassword string `env:"REDIS_PASSWORD"`
RedisDB int `env:"REDIS_DB"`
RedisPoolSize int `env:"REDIS_POOL_SIZE"`
// OpenTelemetry
OtelServiceName string `env:"OTEL_RESOURCE_SERVICE_NAME"`
OtelLibraryName string `env:"OTEL_LIBRARY_NAME"`
OtelServiceVersion string `env:"OTEL_RESOURCE_SERVICE_VERSION"`
OtelDeploymentEnv string `env:"OTEL_RESOURCE_DEPLOYMENT_ENVIRONMENT"`
OtelColExporterEndpoint string `env:"OTEL_EXPORTER_OTLP_ENDPOINT"`
EnableTelemetry bool `env:"ENABLE_TELEMETRY"`
// Auth
AuthEnabled bool `env:"PLUGIN_AUTH_ENABLED"`
AuthHost string `env:"PLUGIN_AUTH_HOST"`
// External Services (gRPC)
ExternalServiceAddress string `env:"EXTERNAL_SERVICE_GRPC_ADDRESS"`
ExternalServicePort string `env:"EXTERNAL_SERVICE_GRPC_PORT"`
}
```
### 2. Load Configuration
```go
// bootstrap/config.go
func InitServers() *Service {
cfg := &Config{}
// Load all environment variables into config struct
if err := libCommons.SetConfigFromEnvVars(cfg); err != nil {
// bootstrap-only: panic is acceptable in main/init; NEVER use panic in business logic
panic(err)
}
// Validate required fields
if cfg.PrimaryDBHost == "" || cfg.PrimaryDBName == "" {
// bootstrap-only: panic is acceptable in main/init; NEVER use panic in business logic
panic("DB_HOST and DB_NAME must be configured")
}
// Continue with initialization...
}
```
### Supported Types
| Go Type | Default Value | Example |
| ---------------------------------------- | ------------- | ------------------------------------------------- |
| `string` | `""` | `ServerAddress string \`env:"SERVER_ADDRESS"\`` |
| `bool` | `false` | `EnableTelemetry bool \`env:"ENABLE_TELEMETRY"\`` |
| `int`, `int8`, `int16`, `int32`, `int64` | `0` | `MaxPoolSize int \`env:"MONGO_MAX_POOL_SIZE"\`` |
### Environment Variable Naming Convention
| Category | Prefix | Example |
| ------------------ | ----------------- | ----------------------------------------- |
| Application | None | `ENV_NAME`, `LOG_LEVEL`, `SERVER_ADDRESS` |
| PostgreSQL | `DB_` | `DB_HOST`, `DB_USER`, `DB_PASSWORD` |
| PostgreSQL Replica | `DB_REPLICA_` | `DB_REPLICA_HOST`, `DB_REPLICA_USER` |
| MongoDB | `MONGO_` | `MONGO_HOST`, `MONGO_NAME` |
| Redis | `REDIS_` | `REDIS_HOST`, `REDIS_PASSWORD` |
| OpenTelemetry | `OTEL_` | `OTEL_RESOURCE_SERVICE_NAME` |
| Auth Plugin | `PLUGIN_AUTH_` | `PLUGIN_AUTH_ENABLED`, `PLUGIN_AUTH_HOST` |
| Idempotency | `IDEMPOTENCY_` | `IDEMPOTENCY_ENABLED`, `IDEMPOTENCY_DEFAULT_TTL_SEC` |
| gRPC Services | `{SERVICE}_GRPC_` | `TRANSACTION_GRPC_ADDRESS` |
### What not to Do
```go
// FORBIDDEN: Manual os.Getenv calls scattered across code
host := os.Getenv("DB_HOST") // DON'T do this
// FORBIDDEN: Configuration outside bootstrap
func NewService() *Service {
dbHost := os.Getenv("DB_HOST") // DON'T do this
}
// CORRECT: All configuration in Config struct, loaded once in bootstrap
type Config struct {
PrimaryDBHost string `env:"DB_HOST"` // Centralized
}
// Load with: libCommons.SetConfigFromEnvVars(&cfg)
```
---
## Database Naming Convention (snake_case) (MANDATORY)
**HARD GATE:** All database tables and columns MUST use `snake_case` naming. This is NON-NEGOTIABLE.
### Naming Rules
| Element | Convention | Example |
| ---------------------- | -------------------------------- | ---------------------------------------------- |
| **Tables** | `snake_case`, plural | `users`, `user_preferences`, `order_items` |
| **Columns** | `snake_case` | `user_id`, `created_at`, `email_address` |
| **Primary keys** | `id` | `id UUID PRIMARY KEY` |
| **Foreign keys** | `{referenced_table_singular}_id` | `user_id`, `organization_id` |
| **Indexes** | `idx_{table}_{column(s)}` | `idx_users_email`, `idx_orders_user_id_status` |
| **Unique constraints** | `uq_{table}_{column(s)}` | `uq_users_email`, `uq_preferences_user` |
| **Check constraints** | `chk_{table}_{description}` | `chk_orders_positive_amount` |
### Layer Separation
**CRITICAL:** Different naming conventions apply at different layers:
| Layer | Convention | Example |
| --------------- | ------------ | ---------------------------------------- |
| **Database** | `snake_case` | `user_id`, `created_at`, `email_address` |
| **Go structs** | `PascalCase` | `UserID`, `CreatedAt`, `EmailAddress` |
| **JSON output** | `camelCase` | `userId`, `createdAt`, `emailAddress` |
### Correct Examples
#### SQL Migration
```sql
-- ✅ CORRECT: All identifiers use snake_case
CREATE TABLE user_preferences (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id),
theme_name VARCHAR(50) DEFAULT 'light',
notification_enabled BOOLEAN DEFAULT true,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_user_preferences_user_id ON user_preferences(user_id);
ALTER TABLE user_preferences ADD CONSTRAINT uq_user_preferences_user UNIQUE (user_id);
```
#### Go Model with Database Tags
```go
// ✅ CORRECT: Go uses PascalCase, db tags use snake_case, json tags use camelCase
type UserPreference struct {
ID string `json:"id" db:"id"`
UserID string `json:"userId" db:"user_id"`
ThemeName string `json:"themeName" db:"theme_name"`
NotificationEnabled bool `json:"notificationEnabled" db:"notification_enabled"`
CreatedAt time.Time `json:"createdAt" db:"created_at"`
UpdatedAt time.Time `json:"updatedAt" db:"updated_at"`
}
```
### FORBIDDEN Patterns
```sql
-- ❌ FORBIDDEN: camelCase in database
CREATE TABLE userPreferences (
id UUID PRIMARY KEY,
userId UUID NOT NULL, -- WRONG: should be user_id
themeName VARCHAR(50), -- WRONG: should be theme_name
createdAt TIMESTAMP -- WRONG: should be created_at
);
-- ❌ FORBIDDEN: PascalCase in database
CREATE TABLE UserPreferences (
ID UUID PRIMARY KEY,
UserID UUID NOT NULL, -- WRONG: should be user_id
ThemeName VARCHAR(50) -- WRONG: should be theme_name
);
-- ❌ FORBIDDEN: Mixed conventions
CREATE TABLE user_preferences (
id UUID PRIMARY KEY,
userId UUID NOT NULL, -- WRONG: inconsistent with table naming
theme_name VARCHAR(50), -- OK
CreatedAt TIMESTAMP -- WRONG: PascalCase
);
```
```go
// ❌ FORBIDDEN: Database tags not using snake_case
type UserPreference struct {
UserID string `json:"userId" db:"userId"` // WRONG: db tag should be "user_id"
ThemeName string `json:"themeName" db:"themeName"` // WRONG: db tag should be "theme_name"
}
```
### Detection Commands
```bash
# Detect camelCase in SQL migrations (should return 0 matches for compliant code)
grep -rn "[a-z][A-Z]" --include="*.sql" ./migrations | grep -v "^--"
# Detect PascalCase column definitions (should return 0 matches)
grep -rn "^\s*[A-Z][a-z]*[A-Z]" --include="*.sql" ./migrations
# Detect incorrect db tags in Go files (should return 0 matches)
grep -rn 'db:"[a-z]*[A-Z]' --include="*.go" ./internal
```
### Why snake_case for Databases
| Reason | Explanation |
| ----------------------- | ----------------------------------------------------------------------------- |
| **PostgreSQL standard** | PostgreSQL folds unquoted identifiers to lowercase; snake_case avoids quoting |
| **Readability** | `user_id` is clearer than `userid` or `UserID` in SQL queries |
| **SQL convention** | Industry standard for relational databases |
| **Tool compatibility** | Most DB tools expect snake_case |
| **Cross-platform** | Works consistently across PostgreSQL, MySQL, SQLite |
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| ---------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------- |
| "camelCase matches our Go code" | DB layer ≠ Go layer. Different conventions for different contexts. | **Use snake_case in DB** |
| "PostgreSQL accepts camelCase in quotes" | Requiring quotes everywhere is error-prone and non-standard. | **Use snake_case without quotes** |
| "ORM handles the mapping" | Explicit > implicit. Clear db tags prevent surprises. | **Use explicit db tags with snake_case** |
| "It's just an internal database" | Internal ≠ exempt from standards. Consistency matters everywhere. | **Use snake_case** |
| "The existing table uses camelCase" | Legacy debt must be migrated. New code cannot perpetuate mistakes. | **Create migration to fix naming** |
---
## Database Migrations (MANDATORY)
**HARD GATE:** All database migrations MUST use `golang-migrate`. Creating custom migration runners is FORBIDDEN.
### Required Tool
| Tool | Version | Purpose |
| ------------------------ | ------- | -------------------------- |
| `golang-migrate/migrate` | v4.x | Database schema migrations |
```bash
# Installation
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
```
### Why golang-migrate Is Mandatory
| Benefit | Description |
| ------------------ | --------------------------------------- |
| **Battle-tested** | Used by thousands of production systems |
| **Atomic** | Migrations run in transactions |
| **Bi-directional** | Supports up/down migrations |
| **Driver support** | PostgreSQL, MySQL, MongoDB, etc. |
| **CI/CD friendly** | Easy to integrate with pipelines |
### FORBIDDEN: Custom Migration Systems
```go
// ❌ FORBIDDEN: Creating custom version tracking table
func initMigrations(db *sql.DB) {
db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version INT PRIMARY KEY,
applied_at TIMESTAMP
)`)
}
// ❌ FORBIDDEN: Manual version checking
func runMigrations(db *sql.DB) {
var currentVersion int
db.QueryRow("SELECT MAX(version) FROM schema_migrations").Scan(¤tVersion)
// ... apply migrations manually
}
// ❌ FORBIDDEN: Embedding migrations in application code
func applyMigration001(db *sql.DB) error {
return db.Exec("CREATE TABLE users (...)")
}
```
**Why this is wrong:**
- Reinvents what golang-migrate already does
- Lacks transaction safety
- No rollback support
- Inconsistent across projects
- Harder to debug and maintain
### Correct Pattern: golang-migrate
#### Migration File Structure
```text
/migrations
000001_create_users_table.up.sql
000001_create_users_table.down.sql
000002_add_email_column.up.sql
000002_add_email_column.down.sql
000003_create_orders_table.up.sql
000003_create_orders_table.down.sql
```
#### Naming Convention
```text
{version}_{description}.{direction}.sql
version: 6-digit zero-padded number (000001, 000002, ...)
description: snake_case description of the change
direction: up (apply) or down (rollback)
```
#### Migration Granularity (MANDATORY)
**RULE: One migration per feature/release. NOT one migration per alteration.**
| Approach | Atomicity | Rollback | Status |
| ------------------------------- | -------------------------- | ------------------------------- | ------------- |
| One migration per feature | ✅ Atomic (all-or-nothing) | `migrate down 1` | **CORRECT** |
| Multiple migrations per feature | ❌ Non-atomic | `migrate down N` (manual count) | **FORBIDDEN** |
**Why this matters:**
- **Atomicity:** A single migration runs in a transaction - it either fully succeeds or fully rolls back
- **Simple rollback:** One feature = one migration = `migrate down 1` to undo
- **Release alignment:** Migrations map 1:1 to features/releases for traceability
**FORBIDDEN: Multiple migrations for one feature**
```text
# ❌ WRONG: 5 migrations for "add user preferences" feature
/migrations
000005_create_preferences_table.up.sql
000006_add_theme_column.up.sql
000007_add_language_column.up.sql
000008_add_timezone_column.up.sql
000009_add_preferences_index.up.sql
# Problem: To rollback this feature, you need "migrate down 5"
# If you forget and do "migrate down 1", feature is partially rolled back
```
**CORRECT: One migration for one feature**
```text
# ✅ CORRECT: 1 migration for "add user preferences" feature
/migrations
000005_add_user_preferences.up.sql
000005_add_user_preferences.down.sql
# Rollback: "migrate down 1" undoes the entire feature
```
**What goes in a single migration:**
```sql
-- 000005_add_user_preferences.up.sql
-- All changes for "user preferences" feature in ONE file
-- 1. Create table
CREATE TABLE user_preferences (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id),
theme VARCHAR(50) DEFAULT 'light',
language VARCHAR(10) DEFAULT 'en',
timezone VARCHAR(50) DEFAULT 'UTC',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- 2. Add index
CREATE INDEX idx_user_preferences_user_id ON user_preferences(user_id);
-- 3. Add constraint
ALTER TABLE user_preferences ADD CONSTRAINT uq_user_preferences_user UNIQUE (user_id);
```
```sql
-- 000005_add_user_preferences.down.sql
-- Reverse ALL changes in ONE file (reverse order)
ALTER TABLE user_preferences DROP CONSTRAINT IF EXISTS uq_user_preferences_user;
DROP INDEX IF EXISTS idx_user_preferences_user_id;
DROP TABLE IF EXISTS user_preferences;
```
**Migration Granularity Decision Table:**
| Scenario | Migrations | Example |
| ------------------------------------ | ------------------- | --------------------------------- |
| New feature with table + indexes | 1 migration | `000005_add_user_preferences.sql` |
| Bug fix requiring schema change | 1 migration | `000006_fix_email_constraint.sql` |
| Refactor with multiple table changes | 1 migration | `000007_normalize_addresses.sql` |
| Unrelated changes in same release | Separate migrations | Each gets own migration |
**Anti-Rationalization:**
| Rationalization | Why It's WRONG | Required Action |
| ---------------------------------------------- | ------------------------------------------------------------------- | ------------------------------ |
| "Smaller migrations are safer" | Atomicity makes single migration safer. Partial state is dangerous. | **Combine into one migration** |
| "I want to track each change separately" | Use comments inside the migration file. Git tracks file history. | **Combine into one migration** |
| "Rollback granularity is better with multiple" | Partial rollback = broken state. All-or-nothing is correct. | **Combine into one migration** |
| "The migration file would be too long" | Long but atomic > short but fragmented. Use comments for sections. | **Combine into one migration** |
#### Migration File Examples
```sql
-- 000001_create_users_table.up.sql
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255) NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_users_email ON users(email);
```
```sql
-- 000001_create_users_table.down.sql
DROP INDEX IF EXISTS idx_users_email;
DROP TABLE IF EXISTS users;
```
### Makefile Commands (REQUIRED)
**See [devops.md - Database Migration Commands](../devops.md#database-migration-commands-mandatory)** for the complete Makefile implementation.
**Quick reference:**
| Command | Purpose |
| ------------------------------ | -------------------------------------- |
| `make migrate-up` | Apply all pending migrations |
| `make migrate-down` | Rollback last migration |
| `make migrate-create NAME=xxx` | Create new migration |
| `make migrate-version` | Show current version |
| `make dev-setup` | Install golang-migrate and other tools |
### Docker Compose Integration
```yaml
# docker-compose.yml
services:
migrate:
image: migrate/migrate:v4.17.0
volumes:
- ./migrations:/migrations
command:
[
"-path",
"/migrations",
"-database",
"postgres://user:pass@db:5432/dbname?sslmode=disable",
"up",
]
depends_on:
db:
condition: service_healthy
```
### Anti-Patterns (Detection Commands)
```bash
# Detect custom migration tables (should return 0 matches)
grep -rn "schema_migrations\|migration_version\|db_version" --include="*.go" ./internal
# Detect manual migration tracking (should return 0 matches)
grep -rn "CREATE TABLE.*migration" --include="*.go" ./internal
# Detect embedded SQL DDL in Go code (review each match)
grep -rn "CREATE TABLE\|ALTER TABLE\|DROP TABLE" --include="*.go" ./internal
```
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| --------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------- |
| "golang-migrate is overkill for this project" | Consistency > simplicity. All projects use the same tool. | **Use golang-migrate** |
| "I need custom logic before/after migrations" | Use golang-migrate hooks or run scripts separately. | **Use golang-migrate** |
| "Embedding migrations is more portable" | SQL files are portable. Custom Go code is not. | **Use golang-migrate with SQL files** |
| "My migration table is simpler" | Simpler ≠ better. golang-migrate handles edge cases you haven't thought of. | **Use golang-migrate** |
| "This is just a small schema change" | Small changes grow. Start with the right tool. | **Use golang-migrate** |
---
## License Headers (MANDATORY)
**⛔ HARD GATE:** All `.go` source files MUST include a license header. Missing license headers indicate incomplete compliance and must be fixed before production deployment.
### Why License Headers Are MANDATORY
| Without Headers | With Headers |
| ------------------------- | ----------------------------- |
| IP ownership unclear | Clear copyright attribution |
| Legal exposure in copies | Protected when code is shared |
| Compliance audit failures | Audit-ready codebase |
| Inconsistent attribution | Uniform legal protection |
### Required Format (Elastic License 2.0)
```go
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License 2.0;
// you may not use this file except in compliance with the Elastic License 2.0.
package yourpackage
```
### Header Components
| Component | Value | Notes |
| ----------------- | --------------------- | ----------------------------------------- |
| Copyright holder | `Elasticsearch B.V.` | Fixed for all projects |
| License reference | `Elastic License 2.0` | Or as specified in LICENSE file |
| LICENSE location | Inline in header | No separate LICENSE file reference needed |
### Files That MUST Have Headers
| File Type | Required | Notes |
| ------------------------ | -------- | -------------------------- |
| `*.go` (source files) | ✅ YES | All source code |
| `*_test.go` (test files) | ✅ YES | Tests are also source code |
| `cmd/**/*.go` | ✅ YES | Entry points |
| `internal/**/*.go` | ✅ YES | Internal packages |
| `pkg/**/*.go` | ✅ YES | Public packages |
### Files That MAY Skip Headers
| File Type | Required | Reason |
| --------------------------- | ----------- | ------------------------- |
| Generated files (`*.pb.go`) | ⚠️ OPTIONAL | Auto-generated by protoc |
| Mock files (`mock_*.go`) | ⚠️ OPTIONAL | Auto-generated by mockgen |
| Vendor files (`vendor/**`) | ❌ NO | Third-party code |
### Correct Examples
```go
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License 2.0;
// you may not use this file except in compliance with the Elastic License 2.0.
package bootstrap
import (
"context"
"fmt"
)
```
```go
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License 2.0;
// you may not use this file except in compliance with the Elastic License 2.0.
package bootstrap_test
import (
"testing"
)
```
### FORBIDDEN Patterns
```go
// ❌ FORBIDDEN: Missing header entirely
package model
import "time"
// ❌ FORBIDDEN: Wrong format (missing full license text)
// Copyright Elasticsearch B.V.
// Licensed under Elastic License 2.0
package model
// ❌ FORBIDDEN: Header after package declaration
package model
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License 2.0;
// you may not use this file except in compliance with the Elastic License 2.0.
import "time"
```
### Verification Commands
```bash
# Find .go files without license header (should return 0 for compliant projects)
find . -name "*.go" -not -path "./vendor/*" -not -name "*.pb.go" -not -name "mock_*.go" \
-exec sh -c 'head -1 "$1" | grep -q "^// Copyright" || echo "$1"' _ {} \;
# Count files with correct header
grep -rl "Copyright (c).*Lerian Studio" --include="*.go" . | wc -l
# Count total .go files (excluding vendor/generated)
find . -name "*.go" -not -path "./vendor/*" -not -name "*.pb.go" -not -name "mock_*.go" | wc -l
```
### Adding Headers to Existing Files
For projects adopting this standard, use this script to add headers:
```bash
#!/bin/bash
# add-license-headers.sh
HEADER='// Copyright (c) 2024 Lerian Studio. All rights reserved.
// Use of this source code is governed by the Elastic License 2.0
// that can be found in the LICENSE file.
'
find . -name "*.go" -not -path "./vendor/*" -not -name "*.pb.go" -not -name "mock_*.go" | while read file; do
if ! head -1 "$file" | grep -q "^// Copyright"; then
echo "Adding header to: $file"
echo "$HEADER$(cat "$file")" > "$file"
fi
done
```
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| ------------------------------ | ---------------------------------------------------------------------------- | ---------------------------- |
| "It's just internal code" | Internal code is still copyrighted. Headers protect IP. | **Add header to all files** |
| "Tests don't need headers" | Tests are source code. Same rules apply. | **Add header to test files** |
| "I'll add them later" | Later = never. Add headers when creating files. | **Add header immediately** |
| "The LICENSE file is enough" | Per-file headers provide clear attribution in copies. | **Add header to all files** |
| "Generated files are excluded" | Only truly auto-generated (protobuf, mocks). Hand-written = header required. | **Check if truly generated** |
---
## MongoDB Patterns (MANDATORY)
Common MongoDB issues include $regex injection vectors, unconfigured MaxPoolSize, blocking index creation, and deprecated SetBackground calls.
**⛔ HARD GATE:** All MongoDB operations MUST follow these patterns to prevent injection, ensure performance, and avoid deprecated APIs.
### Injection Prevention (CRITICAL)
Using `$regex` operators with unvalidated user input allows NoSQL injection attacks.
**⛔ FORBIDDEN: Unescaped $regex with User Input**
```go
// ❌ FORBIDDEN: User input directly in $regex
filter := bson.M{
"name": bson.M{"$regex": userInput}, // INJECTION VECTOR
}
cursor, _ := collection.Find(ctx, filter)
// Attack example: userInput = ".*" returns all documents
// Attack example: userInput = "admin|" matches "admin" or empty
```
**✅ CORRECT: Use $eq or Escape Special Characters**
```go
// ✅ CORRECT: Use $eq for exact matches (preferred)
filter := bson.M{
"name": bson.M{"$eq": userInput},
}
// ✅ CORRECT: Use $text search (requires text index)
filter := bson.M{
"$text": bson.M{"$search": userInput},
}
// ✅ CORRECT: Escape regex special characters if $regex is required
import "regexp"
func escapeRegex(s string) string {
return regexp.QuoteMeta(s)
}
filter := bson.M{
"name": bson.M{
"$regex": "^" + escapeRegex(userInput), // Escaped
"$options": "i",
},
}
```
**Detection Commands:**
```bash
# MANDATORY: Run before every PR that touches MongoDB code
grep -rn '\$regex' internal/adapters/mongodb --include="*.go"
# Review each match - if userInput is used without escaping: VIOLATION
# Expected: All $regex uses have escapeRegex() or validated input
```
### Connection Pooling (MANDATORY)
MongoDB connections without MaxPoolSize configuration cause connection exhaustion under load.
**⛔ FORBIDDEN: Default Pool Configuration**
```go
// ❌ FORBIDDEN: No MaxPoolSize configured
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
// ❌ FORBIDDEN: MaxPoolSize too high or too low
opts := options.Client().SetMaxPoolSize(1000) // Too high - memory issues
opts := options.Client().SetMaxPoolSize(1) // Too low - contention
```
**✅ CORRECT: Configure Pool Size Based on Load**
```go
// ✅ CORRECT: Configure MaxPoolSize in connection options
clientOpts := options.Client().
ApplyURI(mongoURI).
SetMaxPoolSize(uint64(cfg.MongoMaxPoolSize)). // From environment
SetMinPoolSize(10). // Maintain baseline
SetMaxConnIdleTime(30 * time.Second) // Release idle connections
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
return nil, fmt.Errorf("failed to connect to MongoDB: %w", err)
}
// ✅ CORRECT: Verify connection
if err := client.Ping(ctx, nil); err != nil {
return nil, fmt.Errorf("failed to ping MongoDB: %w", err)
}
```
**Pool Size Guidelines:**
| Workload | MaxPoolSize | MinPoolSize | Rationale |
| -------------------- | ----------- | ----------- | ------------------------ |
| Low (< 100 RPS) | 50 | 5 | Conservative, low memory |
| Medium (100-500 RPS) | 100 | 10 | Balanced |
| High (> 500 RPS) | 200 | 20 | High throughput |
**Detection Commands:**
```bash
# Find MongoDB connection setup
grep -rn "mongo.Connect\|SetMaxPoolSize\|MaxPoolSize" internal/bootstrap --include="*.go"
# Expected: MaxPoolSize is set via configuration, not hardcoded or missing
```
### Index Management (MANDATORY)
Blocking index creation and deprecated SetBackground calls cause production issues.
**⛔ FORBIDDEN: Blocking Index Creation and Deprecated APIs**
```go
// ❌ FORBIDDEN: SetBackground (deprecated in MongoDB 4.2+)
indexModel := mongo.IndexModel{
Keys: bson.D{{Key: "email", Value: 1}},
}
opts := options.CreateIndexes().SetBackground(true) // DEPRECATED
collection.Indexes().CreateOne(ctx, indexModel, opts)
// ❌ FORBIDDEN: Blocking index creation on large collections
// (No background/non-blocking option specified)
collection.Indexes().CreateOne(ctx, indexModel) // BLOCKS WRITES
```
**✅ CORRECT: Use CreateIndexes with Batch Operations**
```go
// ✅ CORRECT: Use CreateIndexes (plural) for batch index creation
// MongoDB 4.2+ creates indexes in background automatically for replica sets
indexModels := []mongo.IndexModel{
{
Keys: bson.D{{Key: "email", Value: 1}},
Options: options.Index().SetUnique(true),
},
{
Keys: bson.D{{Key: "created_at", Value: -1}},
Options: options.Index().SetName("idx_created_at_desc"),
},
}
// CreateIndexes (not CreateIndex) is non-blocking on replica sets
names, err := collection.Indexes().CreateMany(ctx, indexModels)
if err != nil {
return fmt.Errorf("failed to create indexes: %w", err)
}
logger.Infof("Created indexes: %v", names)
```
**Index Creation Best Practices:**
| Method | Blocking? | Use Case |
| ------------------- | ----------------- | ----------------------------- |
| `CreateMany()` | No (replica sets) | Production - multiple indexes |
| `CreateOne()` | No (replica sets) | Production - single index |
| SetBackground(true) | N/A - DEPRECATED | **NEVER USE** |
**Detection Commands:**
```bash
# Find deprecated SetBackground usage
grep -rn "SetBackground" internal/adapters/mongodb --include="*.go"
# Expected: 0 matches (SetBackground is deprecated)
# Find index creation patterns
grep -rn "CreateIndex\|CreateMany\|CreateOne" internal/adapters/mongodb --include="*.go"
# Review: Ensure no blocking operations on large collections
```
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| --------------------------------------- | -------------------------------------------------------- | --------------------------- |
| "$regex is convenient for search" | $regex with user input = injection. Use $text or escape. | **Use $eq or escape input** |
| "Default pool size works fine" | Works until load spikes. Then connections exhaust. | **Configure MaxPoolSize** |
| "We have few documents, blocking is OK" | Few now = many later. Non-blocking is always safer. | **Use CreateMany** |
| "SetBackground still works" | Deprecated = will be removed. Code breaks on upgrade. | **Remove SetBackground** |
| "MongoDB handles injection" | MongoDB executes operators. $regex is an operator. | **Escape or avoid $regex** |
| "Connection pool is internal detail" | Internal detail that causes production outages. | **Configure explicitly** |
### Complete MongoDB Connection Example
```go
// internal/bootstrap/config.go
type Config struct {
// MongoDB
MongoURI string `env:"MONGO_URI" default:"mongodb"`
MongoDBHost string `env:"MONGO_HOST"`
MongoDBName string `env:"MONGO_NAME"`
MongoDBUser string `env:"MONGO_USER"`
MongoDBPassword string `env:"MONGO_PASSWORD"`
MongoDBPort string `env:"MONGO_PORT"`
MongoDBParameters string `env:"MONGO_PARAMETERS"`
MongoMaxPoolSize int `env:"MONGO_MAX_POOL_SIZE" default:"100"`
MongoMinPoolSize int `env:"MONGO_MIN_POOL_SIZE" default:"10"`
}
func connectMongoDB(cfg *Config, logger libLog.Logger) (*mongo.Client, error) {
// Build connection string
mongoSource := fmt.Sprintf("%s://%s:%s@%s:%s/",
cfg.MongoURI, cfg.MongoDBUser, cfg.MongoDBPassword,
cfg.MongoDBHost, cfg.MongoDBPort)
if cfg.MongoDBParameters != "" {
mongoSource += "?" + cfg.MongoDBParameters
}
// Configure client options
clientOpts := options.Client().
ApplyURI(mongoSource).
SetMaxPoolSize(uint64(cfg.MongoMaxPoolSize)).
SetMinPoolSize(uint64(cfg.MongoMinPoolSize)).
SetMaxConnIdleTime(30 * time.Second)
// Connect
client, err := mongo.Connect(context.Background(), clientOpts)
if err != nil {
return nil, fmt.Errorf("failed to connect to MongoDB: %w", err)
}
// Verify connection
if err := client.Ping(context.Background(), nil); err != nil {
return nil, fmt.Errorf("failed to ping MongoDB: %w", err)
}
logger.Infof("Connected to MongoDB at %s:%s", cfg.MongoDBHost, cfg.MongoDBPort)
return client, nil
}
```
---
## Dependency Management (MANDATORY)
**⛔ HARD GATE:** All Go projects MUST use Go modules with explicit version pinning. Floating versions and vendoring without go.mod are FORBIDDEN.
### go.mod Requirements (MANDATORY)
```go
// ✅ CORRECT: Explicit Go version and module path
module github.com/LerianStudio/your-service
go 1.24
require (
github.com/LerianStudio/lib-commons/v2 v2.4.0
github.com/gofiber/fiber/v2 v2.52.0
github.com/jackc/pgx/v5 v5.5.0
)
```
### Version Pinning Rules
| Type | Pattern | Example | Required? |
| --------------------- | ----------------- | ------------------------ | ------------ |
| Direct dependencies | Exact version | `v2.4.0` | ✅ MANDATORY |
| Indirect dependencies | Managed by go mod | `// indirect` | Auto-managed |
| Pre-release | Explicit commit | `v0.0.0-20240101-abc123` | When needed |
### FORBIDDEN Patterns
```go
// ❌ FORBIDDEN: Latest/floating versions
require (
github.com/some/package v0.0.0 // WRONG: Not a real version
)
// ❌ FORBIDDEN: Missing go.sum
// go.sum MUST be committed to version control
// ❌ FORBIDDEN: Replacing with local paths in committed go.mod
replace github.com/LerianStudio/lib-commons => ../lib-commons // Development only
```
### Security Updates (MANDATORY)
**Run weekly or before each release:**
```bash
# Check for vulnerabilities
go list -m -json all | go run golang.org/x/vuln/cmd/govulncheck@latest
# Update dependencies (review changes before committing)
go get -u ./...
go mod tidy
# Verify no breaking changes
go build ./...
go test ./...
```
### Dependency Review Checklist
```text
Before adding a new dependency:
[ ] Is it actively maintained? (commits within last 6 months)
[ ] Does it have a license compatible with Apache 2.0?
[ ] Is the version stable (not v0.x.x for production)?
[ ] Does it duplicate functionality already in lib-commons?
[ ] Is the transitive dependency count acceptable?
If any checkbox fails → Reconsider or document exception.
```
### Private Modules (GOPRIVATE)
```bash
# For Lerian private repos
export GOPRIVATE=github.com/LerianStudio/*
# In ~/.gitconfig
[url "ssh://[email protected]/"]
insteadOf = https://github.com/
```
### Detection Commands (MANDATORY)
```bash
# MANDATORY: Run before every PR with dependency changes
# Check for missing go.sum entries
go mod verify
# Check for unused dependencies
go mod tidy -v
# List outdated dependencies
go list -m -u all
# Scan for vulnerabilities
govulncheck ./...
# Expected: go.sum is complete, no vulnerabilities, no unused deps
```
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
| ---------------------------------- | ------------------------------------------------- | ------------------------- |
| "Latest is always best" | Latest can have breaking changes or new bugs. | **Pin explicit versions** |
| "go.sum is auto-generated" | go.sum is a security artifact. Must be committed. | **Commit go.sum** |
| "I'll update deps later" | Later = security vulnerabilities accumulate. | **Update regularly** |
| "Small package, no license needed" | All OSS has licenses. Verify compatibility. | **Check license** |
| "Vendor folder is safer" | Vendor without go.mod is unmaintainable. | **Use go.mod + go.sum** |
| "Replace directive for debugging" | Replace directives break reproducible builds. | **Remove before commit** |
---
```
### ../../docs/standards/golang/multi-tenant.md
```markdown
# Go Standards - Multi-Tenant
> **Module:** multi-tenant.md | **Sections:** §27 | **Parent:** [index.md](index.md)
This module covers multi-tenant patterns with Tenant Manager.
---
## Table of Contents
| # | Section | Description |
|---|---------|-------------|
| 1 | [Multi-Tenant Patterns (CONDITIONAL)](#multi-tenant-patterns-conditional) | Configuration, Tenant Manager API, middleware, context injection, repository adaptation |
| 1a | [Generic TenantMiddleware (Standard Pattern)](#generic-tenantmiddleware-standard-pattern) | Single-module services (CRM, plugins, reporter) |
| 1b | [Multi-module middleware (MultiPoolMiddleware)](#multi-module-middleware-multipoolmiddleware) | Multi-module unified services (midaz ledger) |
| 2 | [Tenant Isolation Verification (⚠️ CONDITIONAL)](#tenant-isolation-verification-conditional) | Database-per-tenant verification, context-based connection checks |
| 3 | [Route-Level Auth-Before-Tenant Ordering (MANDATORY)](#route-level-auth-before-tenant-ordering-mandatory) | Auth MUST validate JWT before tenant middleware calls Tenant Manager API |
| 24 | [Multi-Tenant Message Queue Consumers (Lazy Mode)](#multi-tenant-message-queue-consumers-lazy-mode) | Lazy consumer initialization, on-demand connection, exponential backoff |
| 25 | [M2M Credentials via Secret Manager (Plugin-Only)](#m2m-credentials-via-secret-manager-plugin-only) | AWS Secrets Manager integration for plugin-to-product authentication per tenant |
| 26 | [Service Authentication (MANDATORY)](#service-authentication-mandatory) | API key authentication for tenant-manager /settings endpoint via X-API-Key header |
---
## Multi-Tenant Patterns (CONDITIONAL)
**CONDITIONAL:** Only implement if `MULTI_TENANT_ENABLED=true` is required for your service.
### HARD GATE: Canonical Model Compliance
**Existence ≠ Compliance.** A service that has "some multi-tenant code" is NOT considered multi-tenant unless every component matches the canonical patterns defined in this document exactly.
MUST replace multi-tenant implementations that use custom middleware, manual DB switching, non-standard env var names, or any mechanism other than the lib-commons v4 tenant-manager sub-packages — they are **non-compliant**. Not patched, not adapted, **replaced**.
The only valid multi-tenant implementation uses:
- `tenantId` from JWT via `TenantMiddleware` or `MultiPoolMiddleware` (from `lib-commons/v4/commons/tenant-manager/middleware`), registered per-route using a local `WhenEnabled` helper
- `core.ResolvePostgres` / `core.ResolveMongo` / `core.ResolveModuleDB` for database resolution (from `lib-commons/v4/commons/tenant-manager/core`)
- `valkey.GetKeyFromContext` for Redis key prefixing (from `lib-commons/v4/commons/tenant-manager/valkey`)
- `s3.GetObjectStorageKeyForTenant` for S3 key prefixing (from `lib-commons/v4/commons/tenant-manager/s3`)
- `tmrabbitmq.Manager` for RabbitMQ vhost isolation (from `lib-commons/v4/commons/tenant-manager/rabbitmq`)
- The 8 canonical `MULTI_TENANT_*` environment variables with correct names and defaults
- `client.WithCircuitBreaker` on the Tenant Manager HTTP client
- `client.WithServiceAPIKey` on the Tenant Manager HTTP client for `/settings` endpoint authentication
MUST correct any deviation from these patterns before the service can be considered multi-tenant.
**Any file outside this canonical set that claims to handle multi-tenant logic (custom tenant resolvers, manual pool managers, wrapper middleware, etc.) is non-compliant and MUST NOT be considered part of the multi-tenant implementation.** Only files following the patterns below are valid.
### Canonical File Map
These are the only files that require multi-tenant changes. The exact paths follow the standard Go project layout used across Lerian services. Files not listed here MUST NOT contain multi-tenant logic.
**Always modified (every service):**
| File | Gate | What Changes |
|------|------|-------------|
| `go.mod` | 2 | lib-commons v4, lib-auth v2 |
| `internal/bootstrap/config.go` | 3 | 8 canonical `MULTI_TENANT_*` env vars in Config struct |
| `internal/bootstrap/service.go` (or equivalent init file) | 4 | Conditional initialization: Tenant Manager client, connection managers, middleware creation. Branch on `cfg.MultiTenantEnabled` |
| `internal/bootstrap/routes.go` (or equivalent router file) | 4 | Per-route composition via `WhenEnabled(ttHandler)` — auth validates JWT before tenant resolves DB. Each project implements the `WhenEnabled` helper locally. See [Route-Level Auth-Before-Tenant Ordering](#route-level-auth-before-tenant-ordering-mandatory) |
**Per detected database/storage (Gate 5):**
| File Pattern | Stack | What Changes |
|-------------|-------|-------------|
| `internal/adapters/postgres/**/*.postgresql.go` | PostgreSQL | `r.connection.GetDB()` → `core.ResolvePostgres(ctx, r.connection)` or `core.ResolveModuleDB(ctx, module, r.connection)` (multi-module) |
| `internal/adapters/mongodb/**/*.mongodb.go` | MongoDB | Static mongo connection → `core.ResolveMongo(ctx, r.connection, r.dbName)` |
| `internal/adapters/redis/**/*.redis.go` | Redis | Every key operation → `valkey.GetKeyFromContext(ctx, key)` (including Lua script `KEYS[]` and `ARGV[]`) |
| `internal/adapters/storage/**/*.go` (or S3 adapter) | S3 | Every object key → `s3.GetObjectStorageKeyForTenant(ctx, key)` |
**Conditional — plugin only (Gate 5.5):**
| File | Condition | What Changes |
|------|-----------|-------------|
| `internal/adapters/product/client.go` (or equivalent product API client) | Plugin that calls product APIs | M2M authenticator with per-tenant credential caching via `secretsmanager.GetM2MCredentials` |
| `internal/bootstrap/service.go` | Plugin | Conditional M2M wiring: `if cfg.MultiTenantEnabled` → AWS Secrets Manager client + M2M provider |
**Conditional — RabbitMQ only (Gate 6):**
| File Pattern | What Changes |
|-------------|-------------|
| `internal/adapters/rabbitmq/producer*.go` | Dual constructor: single-tenant (direct connection) + multi-tenant (`tmrabbitmq.Manager.GetChannel`). `X-Tenant-ID` header injection |
| `internal/adapters/rabbitmq/consumer*.go` (or `internal/bootstrap/`) | `tmconsumer.MultiTenantConsumer` with lazy initialization. `X-Tenant-ID` header extraction |
**Tests (Gate 7-8):**
| File Pattern | What Tests |
|-------------|------------|
| `internal/bootstrap/*_test.go` | `TestMultiTenant_BackwardCompatibility` — validates single-tenant mode works unchanged |
| `internal/adapters/**/*_test.go` | Unit tests with mock tenant context, tenant isolation tests (two tenants, data separation) |
| `internal/service/*_test.go` (or integration test dir) | Integration tests with two distinct tenants verifying cross-tenant isolation |
**Output artifacts (Gate 11):**
| File | What |
|------|------|
| `docs/multi-tenant-guide.md` | Activation guide: env vars, how to enable/disable, verification steps |
| `docs/multi-tenant-preview.html` | Visual implementation preview (generated at Gate 1.5, kept for reference) |
**HARD GATE: Files outside this map that contain multi-tenant logic are non-compliant.** If a service has custom files like `internal/tenant/resolver.go`, `internal/middleware/tenant_middleware.go`, `pkg/multitenancy/pool.go` or similar — these MUST be removed and replaced with the canonical lib-commons v4 tenant-manager sub-packages wired through the files listed above.
### Required lib-commons Version
Multi-tenant support requires **lib-commons v4** (`github.com/LerianStudio/lib-commons/v4`). The `tenant-manager` package does not exist in v2.
| lib-commons version | Multi-tenant support | Package path |
|--------------------|-----------------------|-------------|
| **v2** (`lib-commons/v2`) | Not available | N/A — no `tenant-manager` package |
| **v3** (`lib-commons/v3`) | Legacy | Same sub-packages as v4 but without `tenant-manager/cache`. Upgrade to v4. |
| **v4** (`lib-commons/v4`) | Full support (current) | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/...` (sub-packages: `core`, `client`, `cache`, `postgres`, `mongo`, `middleware`, `rabbitmq`, `consumer`, `valkey`, `s3`). The `middleware` sub-package contains both `TenantMiddleware` (single-module) and `MultiPoolMiddleware` (multi-module). Route-level composition uses a local `WhenEnabled` helper (not from lib-commons). |
**Migration to v4:**
Services on lib-commons v2 or v3 MUST upgrade to v4 before implementing multi-tenant. The upgrade involves:
1. Update `go.mod` to the latest v4 tag (currently beta — use latest beta until stable is released)
2. Update all import paths to v4
3. Add the `tenant-manager` package imports where needed
```bash
# Check latest v4 tag
git ls-remote --tags https://github.com/LerianStudio/lib-commons.git | grep "v4" | sort -V | tail -1
# Update go.mod (use latest beta until stable is released)
go get github.com/LerianStudio/lib-commons/[email protected]
# Update import paths across the codebase (portable — works on macOS and Linux)
# From v2:
find . -name "*.go" -exec perl -pi -e 's|lib-commons/v2|lib-commons/v4|g' {} +
# From v3:
find . -name "*.go" -exec perl -pi -e 's|lib-commons/v3|lib-commons/v4|g' {} +
# Verify build
go build ./...
```
### When to Use Multi-Tenant Mode
| Scenario | Mode | Configuration |
|----------|------|---------------|
| Single customer deployment | Single-tenant | `MULTI_TENANT_ENABLED=false` (default) |
| SaaS with shared infrastructure | Multi-tenant | `MULTI_TENANT_ENABLED=true` |
| Multiple isolated databases per customer | Multi-tenant | Requires Tenant Manager |
### Environment Variables
| Env Var | Description | Default | Required |
|---------|-------------|---------|----------|
| `APPLICATION_NAME` | Service name for Tenant Manager API (`/tenants/{id}/services/{service}/settings`) | - | Yes |
| `MULTI_TENANT_ENABLED` | Enable multi-tenant mode | `false` | Yes |
| `MULTI_TENANT_URL` | Tenant Manager service URL | - | If multi-tenant |
| `MULTI_TENANT_ENVIRONMENT` | Deployment environment for cache key segmentation (lazy consumer tenant discovery) | `staging` | Only if RabbitMQ |
| `MULTI_TENANT_MAX_TENANT_POOLS` | Soft limit for tenant connection pools (LRU eviction) | `100` | No |
| `MULTI_TENANT_IDLE_TIMEOUT_SEC` | Seconds before idle tenant connection is eviction-eligible | `300` | No |
| `MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD` | Consecutive failures before circuit breaker opens | `5` | Yes |
| `MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC` | Seconds before circuit breaker resets (half-open) | `30` | Yes |
| `MULTI_TENANT_SERVICE_API_KEY` | API key for authenticating with tenant-manager `/settings` endpoint. Generated via service catalog. | - | If multi-tenant |
**Example `.env` for multi-tenant:**
```bash
MULTI_TENANT_ENABLED=true
MULTI_TENANT_URL=http://tenant-manager:4003
MULTI_TENANT_ENVIRONMENT=production
MULTI_TENANT_MAX_TENANT_POOLS=100
MULTI_TENANT_IDLE_TIMEOUT_SEC=300
MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD=5
MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC=30
MULTI_TENANT_SERVICE_API_KEY=your-service-api-key-here
```
### Configuration
```go
// internal/bootstrap/config.go
type Config struct {
ApplicationName string `env:"APPLICATION_NAME"`
// Multi-Tenant Configuration
MultiTenantEnabled bool `env:"MULTI_TENANT_ENABLED" default:"false"`
MultiTenantURL string `env:"MULTI_TENANT_URL"`
MultiTenantEnvironment string `env:"MULTI_TENANT_ENVIRONMENT" default:"staging"`
MultiTenantMaxTenantPools int `env:"MULTI_TENANT_MAX_TENANT_POOLS" default:"100"`
MultiTenantIdleTimeoutSec int `env:"MULTI_TENANT_IDLE_TIMEOUT_SEC" default:"300"`
MultiTenantCircuitBreakerThreshold int `env:"MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD" default:"5"`
MultiTenantCircuitBreakerTimeoutSec int `env:"MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC" default:"30"`
MultiTenantServiceAPIKey string `env:"MULTI_TENANT_SERVICE_API_KEY"`
// PostgreSQL Primary (used as default connection in single-tenant mode)
PrimaryDBHost string `env:"DB_HOST"`
PrimaryDBUser string `env:"DB_USER"`
PrimaryDBPassword string `env:"DB_PASSWORD"`
PrimaryDBName string `env:"DB_NAME"`
PrimaryDBPort string `env:"DB_PORT"`
PrimaryDBSSLMode string `env:"DB_SSLMODE"`
}
```
### Service Name Resolution
The `service` parameter in `NewManager` maps to the Tenant Manager API path: `/tenants/{id}/services/{service}/settings`. Use `cfg.ApplicationName` (env `APPLICATION_NAME`):
```go
pgMgr := tmpostgres.NewManager(tmClient, cfg.ApplicationName,
tmpostgres.WithModule(ApplicationName), // module = component name constant
tmpostgres.WithLogger(logger),
)
```
| Parameter | Source | Purpose | Example |
|-----------|--------|---------|---------|
| `service` (2nd arg) | `cfg.ApplicationName` (env `APPLICATION_NAME`) | Tenant Manager API path | `"ledger"`, `"reporter"` |
| `module` (WithModule) | Component constant `ApplicationName` | Key in `TenantConfig.Databases[module]` | `"onboarding"`, `"transaction"`, `"manager"` |
### Manager Wiring
**TenantMiddleware (single-module):** Managers are passed directly to the middleware:
```go
mongoManager := tmmongo.NewManager(tmClient, cfg.ApplicationName, ...)
ttMid := tmmiddleware.NewTenantMiddleware(
tmmiddleware.WithMongoManager(mongoManager),
)
```
**MultiPoolMiddleware (multi-module):** Managers MUST be assigned to the Service struct and exposed via getters:
```go
type Service struct {
pgManager interface{}
mongoManager interface{}
}
func (s *Service) GetPGManager() interface{} { return s.pgManager }
func (s *Service) GetMongoManager() interface{} { return s.mongoManager }
// At construction, MUST assign managers
service := &Service{
pgManager: pg.pgManager,
mongoManager: mgo.mongoManager,
}
```
### Tenant Manager Service API
The Tenant Manager is an external service that stores database credentials per tenant. All connection managers in lib-commons call this API to resolve tenant-specific connections.
**Endpoints:**
| Method | Path | Returns | Purpose |
|--------|------|---------|---------|
| `GET` | `/tenants/{tenantID}/services/{service}/settings` | `TenantConfig` | Full tenant configuration with DB credentials |
| `GET` | `/tenants/active?service={service}` | `[]*TenantSummary` | List of active tenants (fallback for discovery) |
**Tenant Discovery (for lazy consumer mode):**
1. Primary: Redis `SMEMBERS "tenant-manager:tenants:active"` (fast, <1ms)
2. Fallback: HTTP `GET /tenants/active?service={service}` (slower, network call)
### TenantConfig Data Model
The Tenant Manager returns this structure for each tenant. The `Databases` map is keyed by **module name** (e.g., `"onboarding"`, `"transaction"`).
```go
type TenantConfig struct {
ID string // Tenant UUID
TenantSlug string // Human-readable slug
IsolationMode string // "isolated" (default) or "schema"
Databases map[string]DatabaseConfig // module -> config
Messaging *MessagingConfig // RabbitMQ config (optional)
}
type DatabaseConfig struct {
PostgreSQL *PostgreSQLConfig
PostgreSQLReplica *PostgreSQLConfig // Read replica (optional)
MongoDB *MongoDBConfig
ConnectionSettings *ConnectionSettings // Per-tenant pool overrides (optional)
}
// ConnectionSettings holds per-tenant database connection pool settings.
// When present, these values override the global defaults on PostgresManager/MongoManager.
// If nil (e.g., older tenant associations), global defaults apply.
type ConnectionSettings struct {
MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"`
}
```
**Isolation Modes:**
| Mode | Database | Schema | Connection String | When to Use |
|------|----------|--------|-------------------|-------------|
| `isolated` (default) | Separate database per tenant | Default `public` schema | Standard connection | Strong isolation, recommended |
| `schema` | Shared database | Schema per tenant | Adds `options=-csearch_path="{schema}"` | Cost optimization, weaker isolation |
### Connection Pool Management
All connection managers (PostgreSQL, MongoDB, RabbitMQ) use **LRU eviction with soft limits**:
- **Soft limit** (`WithMaxTenantPools`): When the pool reaches this size and a new tenant needs a connection, only connections idle longer than the timeout are evicted. If all connections are active, the pool grows beyond the limit.
- **Idle timeout** (`WithIdleTimeout`): Connections not accessed within this window become eligible for eviction. Default: 5 minutes.
- **Connection health**: Cached connections are pinged before reuse (3s timeout). Stale connections are recreated transparently.
The Tenant Manager HTTP client MUST enable the **circuit breaker** (`WithCircuitBreaker`):
- After N consecutive failures, the circuit opens and requests fail fast with `ErrCircuitBreakerOpen`
- After the timeout, the circuit enters half-open state and allows one request through
- On success, the circuit resets to closed
### JWT Tenant Extraction
**Claim key:** `tenantId` (camelCase, hardcoded)
<cannot_skip>
**⛔ CRITICAL: `tenantId` from JWT is the ONLY multi-tenant mechanism.**
The `tenantId` identifies the client/customer. The lib-commons `TenantMiddleware` extracts it from the JWT, resolves the tenant-specific database connection via Tenant Manager API, and stores it in context. Each tenant has its own database — tenant A cannot query tenant B's database.
**`organization_id` is NOT part of multi-tenant isolation.** It is a separate concern (entity within a domain). Adding `organization_id` filters to queries does NOT provide tenant isolation. Multi-tenant isolation comes exclusively from `tenantId` → `TenantConnectionManager` → database-per-tenant.
**Anti-Rationalization:**
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Adding organization_id filters = multi-tenant" | organization_id does NOT route to different databases. All data still in ONE database. | **MUST implement tenantId → TenantConnectionManager** |
| "The codebase already has organization_id wiring" | organization_id is irrelevant for multi-tenant. tenantId from JWT is the mechanism. | **Implement TenantMiddleware with JWT tenantId extraction** |
| "Midaz uses organization_id for tenant isolation" | WRONG. Midaz has ZERO organization_id in WHERE clauses. It uses tenantId → core.ResolveModuleDB(ctx, module, fallback). | **Follow the actual pattern: tenantId → context → database routing** |
</cannot_skip>
```go
// internal/bootstrap/middleware.go
func extractTenantIDFromToken(c *fiber.Ctx) (string, error) {
// Use lib-commons helper for token extraction
accessToken := libHTTP.ExtractTokenFromHeader(c)
if accessToken == "" {
return "", errors.New("no authorization token provided")
}
// Parse without validation (validation done by auth middleware)
token, _, err := new(jwt.Parser).ParseUnverified(accessToken, jwt.MapClaims{})
if err != nil {
return "", err
}
claims, ok := token.Claims.(jwt.MapClaims)
if !ok {
return "", errors.New("invalid token claims format")
}
// Extract tenantId (camelCase only - no fallbacks)
tenantID, ok := claims["tenantId"].(string)
if !ok || tenantID == "" {
return "", errors.New("tenantId claim not found in token")
}
return tenantID, nil
}
```
### Generic TenantMiddleware (Standard Pattern)
**This is the standard pattern for all services.** The lib-commons `TenantMiddleware` handles JWT extraction, tenant resolution, and context injection automatically.
```go
// internal/bootstrap/config.go
import (
"github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/client"
tmpostgres "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/postgres"
tmmongo "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/mongo"
"github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware"
)
func initService(cfg *Config) {
// 1. Create Tenant Manager HTTP client (with circuit breaker — MANDATORY)
var clientOpts []client.ClientOption
if cfg.MultiTenantCircuitBreakerThreshold > 0 {
clientOpts = append(clientOpts,
client.WithCircuitBreaker(
cfg.MultiTenantCircuitBreakerThreshold,
time.Duration(cfg.MultiTenantCircuitBreakerTimeoutSec)*time.Second,
),
)
}
clientOpts = append(clientOpts,
client.WithServiceAPIKey(cfg.MultiTenantServiceAPIKey),
)
tmClient, err := client.NewClient(cfg.MultiTenantURL, logger, clientOpts...)
if err != nil {
return fmt.Errorf("creating tenant-manager client: %w", err)
}
idleTimeout := time.Duration(cfg.MultiTenantIdleTimeoutSec) * time.Second
// 2. Create PostgreSQL manager (one per service or per module)
pgManager := tmpostgres.NewManager(tmClient, "my-service",
tmpostgres.WithModule("my-module"),
tmpostgres.WithLogger(logger),
tmpostgres.WithMaxTenantPools(cfg.MultiTenantMaxTenantPools),
tmpostgres.WithIdleTimeout(idleTimeout),
)
// 3. Create MongoDB manager (optional)
mongoManager := tmmongo.NewManager(tmClient, "my-service",
tmmongo.WithModule("my-module"),
tmmongo.WithLogger(logger),
tmmongo.WithMaxTenantPools(cfg.MultiTenantMaxTenantPools),
tmmongo.WithIdleTimeout(idleTimeout),
)
// 4. Create middleware (do NOT register globally — use per-route with WhenEnabled)
ttMid := middleware.NewTenantMiddleware(
middleware.WithPostgresManager(pgManager),
middleware.WithMongoManager(mongoManager), // optional
)
// Pass ttMid.WithTenantDB as the ttHandler to routes.go.
// In routes.go, register per-route using WhenEnabled(ttHandler).
// When MULTI_TENANT_ENABLED=false, pass nil instead — WhenEnabled handles it.
// See "Route-Level Auth-Before-Tenant Ordering" section
}
```
**What the middleware does internally:**
1. Extracts `Authorization: Bearer {token}` header
2. Parses JWT (unverified — auth middleware already validated it)
3. Extracts `tenantId` claim
4. Calls `PostgresManager.GetConnection(ctx, tenantID)` to resolve tenant-specific DB
5. Stores tenant ID and DB connection in context
6. If MongoDB manager is set, resolves and stores MongoDB connection
7. Calls `c.Next()`
**In repositories, use context-based getters:**
```go
import (
"github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/core"
"github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/valkey"
)
// Single-module service: use generic getter
db, err := core.ResolvePostgres(ctx, r.connection)
// MongoDB
mongoDB, err := core.ResolveMongo(ctx, r.connection, r.dbName)
// Redis key prefixing
key := valkey.GetKeyFromContext(ctx, "cache-key")
// -> "tenant:{tenantId}:cache-key"
// Get tenant ID directly
tenantID := core.GetTenantID(ctx)
```
### Multi-module middleware (MultiPoolMiddleware)
**When to use:** Services that serve multiple modules on a single port with different databases per module. For example, midaz ledger serves onboarding and transaction modules in a single process, each with its own PostgreSQL and MongoDB pools.
**Most services do NOT need this.** If your service has a single database (CRM, plugin-auth, reporter, etc.), use the standard `TenantMiddleware` above. Only reach for `MultiPoolMiddleware` when you have path-based routing to separate database pools.
**Import:**
```go
import (
tmmiddleware "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware"
)
```
**Key types:**
| Type | Purpose |
|------|---------|
| `MultiPoolMiddleware` | Routes requests to module-specific tenant pools based on URL path matching |
| `MultiPoolOption` | Functional option for configuring `MultiPoolMiddleware` |
| `ConsumerTrigger` | Interface for lazy consumer spawning (defined in middleware package) |
| `ErrorMapper` | Function type for custom HTTP error responses |
**Available options:**
| Option | Purpose | Required |
|--------|---------|----------|
| `WithRoute(paths, module, pgPool, mongoPool)` | Map URL path prefixes to a module's database pools | At least one route or default |
| `WithDefaultRoute(module, pgPool, mongoPool)` | Fallback route when no path-based route matches | At least one route or default |
| `WithPublicPaths(paths...)` | URL prefixes that bypass tenant resolution entirely | No |
| `WithConsumerTrigger(ct)` | Enable lazy consumer spawning after tenant ID extraction | No (only if using lazy RabbitMQ mode) |
| `WithCrossModuleInjection()` | Resolve PG connections for all registered modules, not just the matched one | No (only if handlers need cross-module DB access) |
| `WithErrorMapper(fn)` | Custom function to convert tenant-manager errors into HTTP responses | No (built-in mapper is used by default) |
| `WithMultiPoolLogger(logger)` | Set logger for the middleware (otherwise extracted from context) | No |
#### Multi-module service example
```go
// config.go - Multi-module service (e.g., unified ledger with onboarding + transaction)
import (
tmmiddleware "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware"
)
transactionPaths := []string{"/transactions", "/operations", "/balances", "/asset-rates"}
multiMid := tmmiddleware.NewMultiPoolMiddleware(
tmmiddleware.WithRoute(transactionPaths, "transaction", txPGPool, txMongoPool),
tmmiddleware.WithDefaultRoute("onboarding", onbPGPool, onbMongoPool),
tmmiddleware.WithPublicPaths("/health", "/version", "/swagger"),
tmmiddleware.WithConsumerTrigger(consumerTrigger), // optional, for lazy RabbitMQ mode
tmmiddleware.WithCrossModuleInjection(), // enables cross-module PG connections
tmmiddleware.WithErrorMapper(customErrorMapper), // optional, for custom HTTP error responses
tmmiddleware.WithMultiPoolLogger(logger),
)
// Pass multiMid.Handle to routes.go — register per-route using WhenEnabled:
// See "Route-Level Auth-Before-Tenant Ordering" section
```
**What the middleware does internally:**
1. Checks if the request path matches a public path — if so, calls `c.Next()` immediately
2. Matches the request path against registered routes (first match wins, falls back to default route)
3. Checks if the matched route's PG pool is multi-tenant — if not, calls `c.Next()`
4. Extracts `Authorization: Bearer {token}` header and parses JWT
5. Extracts `tenantId` claim from JWT
6. Calls `ConsumerTrigger.EnsureConsumerStarted(ctx, tenantID)` if configured
7. Resolves PG connection via `route.pgPool.GetConnection(ctx, tenantID)` and stores it using module-scoped context keys
8. If `WithCrossModuleInjection()` is enabled, resolves PG connections for all other routes too
9. Resolves MongoDB connection if the route has a mongo pool
10. Calls `c.Next()`
**In repositories for multi-module services, use module-scoped getters:**
```go
import "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/core"
// Multi-module: use module-specific getter
db, err := core.ResolveModuleDB(ctx, "transaction", r.connection)
db, err := core.ResolveModuleDB(ctx, "onboarding", r.connection)
```
#### Simple single-module service example
```go
// config.go - Single-module service (e.g., CRM, plugin, reporter)
// Just use TenantMiddleware directly — no need for MultiPoolMiddleware
import (
tmmiddleware "github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware"
)
ttMid := tmmiddleware.NewTenantMiddleware(
tmmiddleware.WithPostgresManager(pgManager),
tmmiddleware.WithMongoManager(mongoManager), // optional
)
// Pass ttMid.WithTenantDB to routes.go — register per-route using WhenEnabled:
// See "Route-Level Auth-Before-Tenant Ordering" section
```
#### Choosing between TenantMiddleware and MultiPoolMiddleware
| Feature | TenantMiddleware | MultiPoolMiddleware |
|---------|-----------------|-------------------|
| Single PG pool | Yes | Yes (via `WithDefaultRoute`) |
| Multiple PG pools (path-based) | No | Yes (via `WithRoute`) |
| MongoDB support | Yes | Yes |
| Cross-module injection | No | Yes (via `WithCrossModuleInjection`) |
| Consumer trigger (lazy mode) | No | Yes (via `WithConsumerTrigger`) |
| Custom error mapping | No | Yes (via `WithErrorMapper`) |
| Public path bypass | No (handled externally) | Yes (via `WithPublicPaths`) |
| When to use | Single-module services | Multi-module unified services |
**Rule of thumb:** If your service has one database module, use `TenantMiddleware`. If your service combines multiple modules with different databases behind one HTTP port, use `MultiPoolMiddleware`.
#### ConsumerTrigger interface
The `ConsumerTrigger` interface is defined in the lib-commons middleware package. Services that process messages in multi-tenant mode implement this interface and pass it to the middleware for lazy consumer activation.
```go
// Defined in github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware
type ConsumerTrigger interface {
EnsureConsumerStarted(ctx context.Context, tenantID string)
}
```
The middleware calls `EnsureConsumerStarted` after extracting the tenant ID. The first request per tenant spawns the consumer (~500ms). Subsequent requests return immediately (<1ms). Import this interface from `tmmiddleware`, not from service-specific packages.
#### ErrorMapper
The `ErrorMapper` type lets you customize how tenant-manager errors are converted into HTTP responses. When not set, the built-in default mapper handles standard cases (401, 403, 404, 503).
```go
// Defined in github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware
type ErrorMapper func(c *fiber.Ctx, err error, tenantID string) error
```
Example custom mapper:
```go
customErrorMapper := func(c *fiber.Ctx, err error, tenantID string) error {
if errors.Is(err, core.ErrTenantNotFound) {
return c.Status(404).JSON(fiber.Map{
"code": "TENANT_NOT_FOUND",
"message": fmt.Sprintf("tenant %s not found", tenantID),
})
}
// Fall through to default behavior for other errors
return c.Status(500).JSON(fiber.Map{
"code": "INTERNAL_ERROR",
"message": err.Error(),
})
}
```
### Database Connection in Repositories
Repositories use context-based getters to retrieve tenant connections:
```go
// internal/adapters/postgres/entity/entity.postgresql.go
func (r *EntityPostgreSQLRepository) Create(ctx context.Context, entity *mmodel.Entity) (*mmodel.Entity, error) {
logger, tracer, _, _ := libCommons.NewTrackingFromContext(ctx)
ctx, span := tracer.Start(ctx, "postgres.create_entity")
defer span.End()
// Get tenant-specific connection from context
db, err := core.ResolvePostgres(ctx, r.connection)
if err != nil {
libOpentelemetry.HandleSpanError(&span, "Failed to get database connection", err)
logger.Errorf("Failed to get database connection: %v", err)
return nil, err
}
record := &EntityPostgreSQLModel{}
record.FromEntity(entity)
// Use db for queries - automatically scoped to tenant's database
// ...
}
```
### Redis Key Prefixing
```go
// internal/adapters/redis/repository.go
func (r *RedisRepository) Set(ctx context.Context, key, value string, ttl time.Duration) error {
logger, tracer, _, _ := libCommons.NewTrackingFromContext(ctx)
ctx, span := tracer.Start(ctx, "redis.set")
defer span.End()
// Tenant-aware key prefixing (adds tenant:{tenantId}: prefix if multi-tenant)
key = valkey.GetKeyFromContext(ctx, key)
rds, err := r.conn.GetConnection(ctx)
if err != nil {
return err
}
return rds.Set(ctx, key, value, ttl).Err()
}
```
### S3/Object Storage Key Prefixing
Services that store files in S3 MUST prefix object keys with the tenant ID for tenant isolation. The bucket is configured per service via environment variable. Tenant separation is by directory within the bucket.
```go
// In any service/adapter that uploads, downloads, or deletes files from S3:
func (r *StorageRepository) Upload(ctx context.Context, originalKey, contentType string, data io.Reader) error {
// Tenant-aware key prefixing: {tenantId}/{originalKey} in multi-tenant, {originalKey} in single-tenant
key := s3.GetObjectStorageKeyForTenant(ctx, originalKey)
return r.s3Client.Upload(ctx, key, data, contentType)
}
func (r *StorageRepository) Download(ctx context.Context, originalKey string) (io.ReadCloser, error) {
// MUST use the same prefixed key for reads and writes
key := s3.GetObjectStorageKeyForTenant(ctx, originalKey)
return r.s3Client.Download(ctx, key)
}
```
**Storage structure:**
```
Bucket: {service-name} (env var: OBJECT_STORAGE_BUCKET)
└── {tenantId}/
└── {resource}/{path}
```
**Backward compatibility:** When no tenant is in context (single-tenant mode), the key is returned unchanged — no prefix added.
### RabbitMQ Multi-Tenant: Two-Layer Isolation Model
RabbitMQ multi-tenant requires **two complementary layers** — both are mandatory:
| Layer | Mechanism | Purpose |
|-------|-----------|---------|
| **1. Vhost Isolation** | `tmrabbitmq.Manager` → `GetChannel(ctx, tenantID)` | **Isolation.** Each tenant gets its own RabbitMQ vhost. Queues, exchanges, and connections are fully separated. |
| **2. X-Tenant-ID Header** | `headers["X-Tenant-ID"] = tenantID` | **Audit + context propagation.** Enables distributed tracing, log correlation, and downstream tenant resolution. Does NOT provide isolation. |
**⛔ Layer 2 alone is NOT multi-tenant compliant.** A shared connection with `X-Tenant-ID` headers provides traceability but zero isolation — a poison message or traffic spike from one tenant affects all tenants.
**⛔ Layer 1 alone is incomplete.** Vhosts isolate but the `X-Tenant-ID` header is needed for log correlation, distributed tracing, and downstream context propagation across services.
### RabbitMQ Multi-Tenant Producer
```go
// internal/adapters/rabbitmq/producer.go
type ProducerRepository struct {
conn *libRabbitmq.RabbitMQConnection
rabbitMQManager *tmrabbitmq.Manager
multiTenantMode bool
}
// Single-tenant constructor
func NewProducer(conn *libRabbitmq.RabbitMQConnection) *ProducerRepository {
return &ProducerRepository{
conn: conn,
multiTenantMode: false,
}
}
// Multi-tenant constructor
func NewProducerMultiTenant(pool *tmrabbitmq.Manager) *ProducerRepository {
return &ProducerRepository{
rabbitMQManager: pool,
multiTenantMode: true,
}
}
func (p *ProducerRepository) Publish(ctx context.Context, exchange, key string, message []byte) error {
// Inject tenant ID header
tenantID := core.GetTenantID(ctx)
headers := amqp.Table{}
if tenantID != "" {
headers["X-Tenant-ID"] = tenantID
}
if p.multiTenantMode {
if tenantID == "" {
return fmt.Errorf("tenant ID is required in multi-tenant mode")
}
// Get tenant-specific channel from pool
channel, err := p.rabbitMQManager.GetChannel(ctx, tenantID)
if err != nil {
return err
}
return channel.PublishWithContext(ctx, exchange, key, false, false,
amqp.Publishing{
ContentType: "application/json",
DeliveryMode: amqp.Persistent,
Headers: headers,
Body: message,
})
}
// Single-tenant: use static connection
return p.conn.Channel.Publish(exchange, key, false, false,
amqp.Publishing{Body: message, Headers: headers})
}
```
### MongoDB Multi-Tenant Repository
```go
// internal/adapters/mongodb/metadata.go
type MetadataMongoDBRepository struct {
connection *libMongo.MongoConnection
dbName string
}
func NewMetadataMongoDBRepository(conn *libMongo.MongoConnection, dbName string) *MetadataMongoDBRepository {
return &MetadataMongoDBRepository{connection: conn, dbName: dbName}
}
func (r *MetadataMongoDBRepository) Create(ctx context.Context, collection string, metadata *Metadata) error {
logger, tracer, _, _ := libCommons.NewTrackingFromContext(ctx)
ctx, span := tracer.Start(ctx, "mongodb.create_metadata")
defer span.End()
// Get tenant-specific database from context
tenantDB, err := core.ResolveMongo(ctx, r.connection, r.dbName)
if err != nil {
libOpentelemetry.HandleSpanError(&span, "Failed to get database connection", err)
return err
}
// Use tenant's database for operations
coll := tenantDB.Collection(strings.ToLower(collection))
record := &MetadataMongoDBModel{}
if err := record.FromEntity(metadata); err != nil {
return err
}
_, err = coll.InsertOne(ctx, record)
if err != nil {
libOpentelemetry.HandleSpanError(&span, "Failed to insert metadata", err)
return err
}
return nil
}
```
### Conditional Initialization
The initialization path depends on whether the service runs a single module or combines multiple modules:
```go
// internal/bootstrap/service.go
func InitService(cfg *Config) (*Service, error) {
// ttHandler starts as nil — WhenEnabled(nil) is a no-op (single-tenant passthrough)
var ttHandler fiber.Handler
if cfg.MultiTenantEnabled && cfg.MultiTenantURL != "" {
if isUnifiedService {
// Multi-module: use MultiPoolMiddleware
// See "Multi-module middleware (MultiPoolMiddleware)" section above
multiMid := initMultiTenantMiddleware(cfg, logger, consumerTrigger)
ttHandler = multiMid.Handle
} else {
// Single-module: use TenantMiddleware
// See "Generic TenantMiddleware (Standard Pattern)" section above
ttMid := tmmiddleware.NewTenantMiddleware(
tmmiddleware.WithPostgresManager(pgManager),
)
ttHandler = ttMid.WithTenantDB
}
// Do NOT register globally with app.Use() — register per-route in routes.go
// using WhenEnabled(ttHandler). See "Route-Level Auth-Before-Tenant Ordering" section.
logger.Infof("Multi-tenant mode enabled with Tenant Manager URL: %s", cfg.MultiTenantURL)
} else {
logger.Info("Running in SINGLE-TENANT MODE")
// ttHandler remains nil — WhenEnabled(nil) calls c.Next() immediately
}
// Pass ttHandler to NewRoutes — routes use WhenEnabled(ttHandler) per-route
// ...
}
```
**Most services follow the single-module path.** Only unified services like midaz ledger need the multi-module path with `MultiPoolMiddleware`.
### Testing Multi-Tenant Code
#### Unit Tests with Mock Tenant Context
```go
// internal/service/user_service_test.go
func TestUserService_Create_WithTenantContext(t *testing.T) {
// Setup tenant context
tenantID := "tenant-123"
ctx := core.ContextWithTenantID(context.Background(), tenantID)
// Mock database connection
mockDB := setupMockDB(t)
ctx = core.ContextWithTenantPGConnection(ctx, mockDB)
// Create service with mock dependencies
repo := repository.NewUserRepository()
service := service.NewUserService(repo, logger)
// Execute
input := &CreateUserInput{Name: "John", Email: "[email protected]"}
result, err := service.Create(ctx, input)
// Assert
require.NoError(t, err)
assert.Equal(t, "John", result.Name)
}
```
#### Testing Tenant Isolation
```go
func TestRepository_Create_TenantIsolation(t *testing.T) {
tests := []struct {
name string
tenantID string
input *Entity
wantErr bool
}{
{
name: "tenant-1 creates entity",
tenantID: "tenant-1",
input: &Entity{Name: "Entity A"},
wantErr: false,
},
{
name: "tenant-2 creates same entity (isolated)",
tenantID: "tenant-2",
input: &Entity{Name: "Entity A"},
wantErr: false, // Different tenant = different database = allowed
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Inject tenant-specific context
ctx := core.ContextWithTenantID(context.Background(), tt.tenantID)
ctx = core.ContextWithTenantPGConnection(ctx, mockDB)
_, err := repo.Create(ctx, tt.input)
if tt.wantErr {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}
```
#### Integration Tests with Tenant Isolation
```go
// tests/integration/multi_tenant_test.go
func TestMultiTenant_TenantIsolation(t *testing.T) {
if !h.IsMultiTenantEnabled() {
t.Skip("Multi-tenant mode is not enabled")
}
env := h.LoadEnvironment()
ctx := context.Background()
client := h.NewHTTPClient(env.ServerURL, env.HTTPTimeout)
// Define two distinct tenants
tenantA := "tenant-a-" + h.RandString(6)
tenantB := "tenant-b-" + h.RandString(6)
headersTenantA := h.TenantAuthHeaders(h.RandHex(8), tenantA)
headersTenantB := h.TenantAuthHeaders(h.RandHex(8), tenantB)
// Step 1: Tenant A creates organization
codeA, bodyA, _ := client.Request(ctx, "POST", "/v1/organizations", headersTenantA, orgPayload)
require.Equal(t, 201, codeA)
var orgA struct{ ID string `json:"id"` }
json.Unmarshal(bodyA, &orgA)
// Step 2: Tenant B creates organization
codeB, bodyB, _ := client.Request(ctx, "POST", "/v1/organizations", headersTenantB, orgPayload)
require.Equal(t, 201, codeB)
var orgB struct{ ID string `json:"id"` }
json.Unmarshal(bodyB, &orgB)
// Step 3: Verify Tenant A cannot see Tenant B's data
code, body, _ := client.Request(ctx, "GET", "/v1/organizations", headersTenantA, nil)
require.Equal(t, 200, code)
var list struct{ Items []struct{ ID string `json:"id"` } `json:"items"` }
json.Unmarshal(body, &list)
for _, item := range list.Items {
assert.NotEqual(t, orgB.ID, item.ID, "ISOLATION VIOLATION: Tenant A can see Tenant B's data")
}
}
```
#### Testing Error Cases
```go
func TestMiddleware_WithTenantDB_ErrorCases(t *testing.T) {
tests := []struct {
name string
setupContext func(*fiber.Ctx)
expectedStatus int
expectedCode string
}{
{
name: "missing JWT token",
setupContext: func(c *fiber.Ctx) {
// No Authorization header
},
expectedStatus: 401,
expectedCode: "TENANT_ID_REQUIRED",
},
{
name: "JWT without tenantId claim",
setupContext: func(c *fiber.Ctx) {
token := createJWTWithoutTenantClaim()
c.Request().Header.Set("Authorization", "Bearer "+token)
},
expectedStatus: 401,
expectedCode: "TENANT_ID_REQUIRED",
},
{
name: "tenant not found in Tenant Manager",
setupContext: func(c *fiber.Ctx) {
token := createJWT(map[string]interface{}{"tenantId": "unknown-tenant"})
c.Request().Header.Set("Authorization", "Bearer "+token)
},
expectedStatus: 404,
expectedCode: "TENANT_NOT_FOUND",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
app := fiber.New()
ctx := app.AcquireCtx(&fasthttp.RequestCtx{})
defer app.ReleaseCtx(ctx)
tt.setupContext(ctx)
err := middleware.WithTenantDB(ctx)
require.Error(t, err)
fiberErr, ok := err.(*fiber.Error)
require.True(t, ok)
assert.Equal(t, tt.expectedStatus, fiberErr.Code)
})
}
}
```
#### Testing RabbitMQ Multi-Tenant Consumer
```go
func TestRabbitMQConsumer_MultiTenant(t *testing.T) {
t.Run("injects tenant context from X-Tenant-ID header", func(t *testing.T) {
tenantID := "tenant-456"
// Create message with tenant header
message := amqp.Delivery{
Headers: amqp.Table{
"X-Tenant-ID": tenantID,
},
Body: []byte(`{"action": "create"}`),
}
// Setup consumer
consumer := NewMultiTenantConsumer(pool, logger)
// Process message
ctx, err := consumer.injectTenantDBConnections(
context.Background(),
tenantID,
logger,
)
// Assert tenant context injected
require.NoError(t, err)
extractedTenant := core.GetTenantID(ctx)
assert.Equal(t, tenantID, extractedTenant)
})
}
```
#### Testing Redis Key Prefixing
```go
func TestRedisRepository_MultiTenant_KeyPrefixing(t *testing.T) {
t.Run("prefixes keys with tenant ID", func(t *testing.T) {
tenantID := "tenant-789"
ctx := core.ContextWithTenantID(context.Background(), tenantID)
repo := NewRedisRepository(redisConn)
err := repo.Set(ctx, "user:session", "value123", 3600)
require.NoError(t, err)
// Verify key was prefixed
key := valkey.GetKeyFromContext(ctx, "user:session")
assert.Equal(t, "tenant:tenant-789:user:session", key)
})
t.Run("single-tenant mode does not prefix keys", func(t *testing.T) {
ctx := context.Background()
key := valkey.GetKeyFromContext(ctx, "user:session")
assert.Equal(t, "user:session", key)
})
}
```
### Error Handling
| Error | HTTP Status | Code | When |
|-------|-------------|------|------|
| Missing tenantId claim | 401 | `TENANT_ID_REQUIRED` | JWT doesn't have tenantId |
| Tenant not found | 404 | `TENANT_NOT_FOUND` | Tenant not registered in Tenant Manager |
| Tenant not provisioned | 422 | `TENANT_NOT_PROVISIONED` | Database schema not initialized (SQLSTATE 42P01) |
| Tenant suspended | 403 | service-specific | Tenant status is suspended or purged (use `errors.As(err, &core.TenantSuspendedError{})`) |
| Service not configured | 503 | service-specific | Tenant exists but has no config for this service/module (`core.ErrServiceNotConfigured`) |
| Schema mode error | 422 | service-specific | Invalid schema configuration for tenant database |
| Connection error | 503 | service-specific | Failed to get or establish tenant connection |
| Manager closed | 503 | service-specific | Connection manager has been shut down (`core.ErrManagerClosed`) |
| Circuit breaker open | 503 | service-specific | Tenant Manager client tripped after consecutive failures (`core.ErrCircuitBreakerOpen`) |
| Tenant config rate limited | 503 | service-specific | Too many concurrent requests for the same tenant config — retry after brief delay |
### Tenant Isolation Verification (⚠️ CONDITIONAL)
Multi-tenant applications MUST verify tenant isolation to prevent data leakage between tenants.
**⛔ CONDITIONAL:** This section applies ONLY if `MULTI_TENANT_ENABLED=true`. If single-tenant, mark as N/A.
**Detection Question:** Is this a multi-tenant service?
```bash
# Check if multi-tenant mode is enabled
grep -rn "MULTI_TENANT_ENABLED\|MultiTenantEnabled" internal/ --include="*.go"
# If 0 matches OR always set to false: Mark N/A
# If found AND can be true: Apply this section
```
#### Isolation Architecture
Multi-tenant isolation uses a **database-per-tenant** model. The `tenantId` from JWT determines which database the request connects to. Each tenant has its own database — tenant A cannot query tenant B's database.
| Mechanism | How It Works | Protection |
|-----------|-------------|------------|
| **JWT `tenantId` extraction** | `TenantMiddleware` extracts `tenantId` claim from JWT | Identifies the tenant |
| **Database routing** | `TenantConnectionManager` resolves tenant-specific DB connection | Tenant A → Database A, Tenant B → Database B |
| **Context injection** | Connection stored in request context | Repositories use `core.ResolvePostgres(ctx, fallback)` / `core.ResolveMongo(ctx, fallback, dbName)` |
| **Single-tenant passthrough** | `IsMultiTenant() == false` → `c.Next()` immediately | Backward compatibility |
#### Why Tenant Isolation Verification Is MANDATORY
| Attack | Without Verification | With Verification |
|--------|----------------------|-------------------|
| Cross-tenant data access | Tenant A accesses Tenant B's database | Connection-level isolation prevents it |
| Data exfiltration | Cross-tenant data leakage | Separate databases per tenant |
#### Detection Commands (MANDATORY)
```bash
# MANDATORY: Run before every PR in multi-tenant services
# Verify all repositories use context-based connections (not static)
grep -rn "ResolvePostgres\|ResolveModuleDB\|ResolveMongo" internal/adapters/ --include="*.go"
# Verify no repositories use static/hardcoded connections when multi-tenant is enabled
# Excludes tenant-aware variables (tenantDB, tenantmanager) to avoid false positives
grep -rn "\.DB\.\|\.Database\." internal/adapters/ --include="*.go" | grep -v "_test.go" | grep -v "tenantmanager\|tenantDB"
# Expected: All repositories should use tenant-manager context getters (core, valkey, s3 packages)
```
#### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Static connection works fine" | Static connection goes to ONE database. All tenants share it. No isolation. | **Use core.ResolvePostgres(ctx, fallback) / core.ResolveMongo(ctx, fallback, dbName)** |
| "We only have one customer" | Requirements change. Multi-tenant is easy to add now, hard later. | **Design for multi-tenant, deploy as single** |
| "organization_id filtering = tenant isolation" | organization_id does NOT route to different databases. It is NOT multi-tenant. | **Use tenantId from JWT → TenantConnectionManager** |
### Context Functions
lib-commons provides two sets of context functions. They use **separate, isolated context keys** with no fallback between them.
| Function | Context Key | Use When |
|----------|-------------|----------|
| `core.ResolvePostgres(ctx, fallback)` | `tenantPGConnection` | Standard — one database per tenant |
| `core.ResolveModuleDB(ctx, module, fallback)` | `tenantPGConnection:{module}` | When service has multiple database modules |
```go
// Standard usage
db, err := core.ResolvePostgres(ctx, r.connection)
if err != nil {
return err
}
// If service has multiple database modules
db, err := core.ResolveModuleDB(ctx, "my-module", r.connection)
```
**Context setters (used by middleware, not by service code):**
- `core.ContextWithTenantID(ctx, tenantID)` — stores tenant ID
- `core.ContextWithTenantPGConnection(ctx, db)` — generic PG connection (set by TenantMiddleware)
- `core.ContextWithModulePGConnection(ctx, module, db)` — module-specific PG (when service has multiple DB modules)
- `core.ContextWithTenantMongo(ctx, mongoDB)` — MongoDB connection
### Tenant ID Validation
lib-commons validates tenant IDs to prevent path traversal and injection:
```go
const maxTenantIDLength = 256
var validTenantIDPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_-]*$`)
```
Rules:
- Must start with alphanumeric character
- Only alphanumeric, underscore, and hyphen allowed
- Maximum 256 characters
- Empty strings rejected
### Redis Key Prefixing for Lua Scripts
Beyond simple `Set/Get` operations, Redis Lua scripts require special attention. ALL keys passed as `KEYS[]` and `ARGV[]` to Lua scripts MUST be pre-prefixed in Go **before** the script execution:
```go
// ✅ CORRECT: Prefix keys in Go before Lua execution
prefixedBackupQueue := valkey.GetKeyFromContext(ctx, TransactionBackupQueue)
prefixedTransactionKey := valkey.GetKeyFromContext(ctx, transactionKey)
prefixedBalanceSyncKey := valkey.GetKeyFromContext(ctx, utils.BalanceSyncScheduleKey)
// Also prefix ARGV values that are used as keys inside the Lua script
prefixedInternalKey := valkey.GetKeyFromContext(ctx, blcs.InternalKey)
result, err := script.Run(ctx, rds,
[]string{prefixedBackupQueue, prefixedTransactionKey, prefixedBalanceSyncKey},
finalArgs...).Result()
```
```go
// ❌ FORBIDDEN: Hardcoded keys inside Lua script
-- Lua script must NEVER reference keys by name
local key = "balance:lock" -- WRONG: not tenant-prefixed
// ✅ CORRECT: Lua script uses only KEYS[] and ARGV[]
local backupQueue = KEYS[1] -- Already prefixed by Go caller
local txKey = KEYS[2] -- Already prefixed by Go caller
```
This pattern also ensures Redis Cluster compatibility (all keys in `KEYS[]` must be in the same hash slot for atomic operations).
### Multi-Tenant Metrics
Services implementing multi-tenant MUST expose these metrics:
| Metric | Type | Description |
|--------|------|-------------|
| `tenant_connections_total` | Counter | Total tenant connections created |
| `tenant_connection_errors_total` | Counter | Connection failures per tenant |
| `tenant_consumers_active` | Gauge | Active message consumers |
| `tenant_messages_processed_total` | Counter | Messages processed per tenant |
### Responsibility Split: lib-commons vs Service
| Responsibility | lib-commons handles | Service MUST implement |
|---------------|--------------------|-----------------------|
| **Connection pooling** | Cache per tenant, double-check locking | - |
| **Credential fetching** | HTTP call to Tenant Manager API | - |
| **JWT parsing** | Extract `tenantId` from token (both middlewares) | - |
| **Tenant discovery** | Redis -> API fallback, sync loop | - |
| **Lazy consumer lifecycle** | On-demand spawn, backoff, degraded tracking | - |
| **Path-based pool routing** | `MultiPoolMiddleware` matches URL to module pools | - |
| **Cross-module connection injection** | `MultiPoolMiddleware` resolves PG for all modules when enabled | - |
| **Error mapping** | Default error mapper in both middlewares; customizable via `ErrorMapper` in `MultiPoolMiddleware` | - |
| **Middleware registration** | - | Register `TenantMiddleware` or `MultiPoolMiddleware` on routes |
| **Repository adaptation** | - | Use `core.ResolvePostgres(ctx, fallback)` or `core.ResolveModuleDB(ctx, module, fallback)` instead of global DB |
| **Redis key prefixing** | - | Call `valkey.GetKeyFromContext(ctx, key)` for every Redis operation |
| **S3 key prefixing** | Tenant-aware key prefix (`s3.GetObjectStorageKeyForTenant`) | Call `s3.GetObjectStorageKeyForTenant(ctx, key)` for every S3 operation |
| **Consumer setup** | - | Register handlers, call `consumer.Run(ctx)` at startup |
| **Consumer trigger** | - | Implement `ConsumerTrigger` interface (imported from `tmmiddleware`) and wire to middleware |
| **Error handling** | Return sentinel errors | Map errors to HTTP status codes (or provide custom `ErrorMapper`) |
### ConsumerTrigger Wiring
For `ConsumerTrigger` interface definition and usage, see [ConsumerTrigger interface](#consumertrigger-interface) in the Multi-module middleware section above. Import from `tmmiddleware`, not from service-specific packages. For `TenantMiddleware`, wire it externally. For `MultiPoolMiddleware`, pass it via `WithConsumerTrigger(ct)`.
### Anti-Rationalization Table (General)
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Service already has multi-tenant code" | Existence ≠ compliance. Code that doesn't match the Ring canonical model (lib-commons v4 tenant-manager sub-packages) is non-compliant and MUST be replaced entirely. | **STOP. Run compliance audit against this document. Replace every non-compliant component.** |
| "Our custom multi-tenant approach works" | Working ≠ compliant. Custom implementations create drift, block lib-commons upgrades, prevent standardized tooling, and cannot be validated by automated compliance checks. | **STOP. Replace with canonical lib-commons v4 implementation.** |
| "Just need to adapt/patch the existing code" | Non-standard implementations cannot be patched into compliance. The patterns are structurally different (context-based resolution vs static connections, lib-commons middleware vs custom middleware). | **STOP. Replace, do not patch.** |
| "We only have one customer" | Requirements change. Multi-tenant is easy to add now, hard later. | **Design for multi-tenant, deploy as single** |
| "Tenant Manager adds complexity" | Complexity is in connection management anyway. Tenant Manager standardizes it. | **Use Tenant Manager for multi-tenant** |
| "JWT parsing is expensive" | Parse once in middleware, use from context everywhere. | **Extract tenant once, propagate via context** |
| "We'll add tenant isolation later" | Retrofitting tenant isolation is a rewrite. | **Build tenant-aware from the start** |
### Final Step: Single-Tenant Backward Compatibility Validation (MANDATORY)
**HARD GATE: This is the LAST step of every multi-tenant implementation.** CANNOT merge or deploy without completing this validation.
If the service was already running as single-tenant before multi-tenant was added, it MUST continue working unchanged with `MULTI_TENANT_ENABLED=false` (the default). Multi-tenant is opt-in. Single-tenant is the baseline. Breaking it is a production incident for all existing deployments.
#### How the Backward Compatibility Mechanism Works
The middleware checks `pool.IsMultiTenant()` at the start of every request. When `MULTI_TENANT_ENABLED=false`:
- `IsMultiTenant()` returns `false`
- Middleware returns `c.Next()` immediately — zero tenant logic applied
- No JWT parsing, no Tenant Manager calls, no pool routing
- The service uses its default database connection as before
```go
// Single-tenant mode: pass through if pool is not multi-tenant
if !pool.IsMultiTenant() {
return c.Next() // No tenant logic applied — service works as before
}
```
#### Validation Steps (execute in order)
**Step 1 — Remove all multi-tenant env vars and start the service:**
```bash
# Unset ALL multi-tenant variables
unset MULTI_TENANT_ENABLED MULTI_TENANT_URL MULTI_TENANT_ENVIRONMENT
unset MULTI_TENANT_MAX_TENANT_POOLS MULTI_TENANT_IDLE_TIMEOUT_SEC
unset MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC
# Start the service — MUST start without errors, without Tenant Manager running
go run cmd/app/main.go
# Expected log: "Running in SINGLE-TENANT MODE"
```
**Step 2 — Run the full existing test suite (single-tenant):**
```bash
# ALL pre-existing tests MUST pass with multi-tenant disabled
MULTI_TENANT_ENABLED=false go test ./...
```
**Step 3 — Run backward compatibility integration test:**
```go
func TestMultiTenant_BackwardCompatibility(t *testing.T) {
// MUST skip when multi-tenant is enabled — this test validates single-tenant only
if h.IsMultiTenantEnabled() {
t.Skip("Skipping backward compatibility test - multi-tenant mode is enabled")
}
// Create resources WITHOUT tenant context — MUST work in single-tenant mode
code, body, err := client.Request(ctx, "POST", "/v1/organizations", headers, orgPayload)
require.Equal(t, 201, code, "single-tenant CRUD must work without tenant context")
// List resources — MUST return data normally
code, body, err = client.Request(ctx, "GET", "/v1/organizations", headers, nil)
require.Equal(t, 200, code, "single-tenant list must work without tenant context")
// Health endpoints — MUST work without any auth or tenant context
code, _, _ = client.Request(ctx, "GET", "/health", nil, nil)
require.Equal(t, 200, code, "health endpoint must work in single-tenant mode")
}
```
**Step 4 — Validate against this checklist:**
| # | Check | How to Verify | Pass Criteria |
|---|-------|--------------|---------------|
| 1 | Service starts without `MULTI_TENANT_*` vars | Remove all vars, start service | Starts normally, logs "SINGLE-TENANT MODE" |
| 2 | Service starts without Tenant Manager | Don't run Tenant Manager, start service | No connection errors, no panics |
| 3 | All existing CRUD operations work | Run pre-existing integration tests | All pass with same behavior as before |
| 4 | Health/version/swagger endpoints work | `GET /health`, `GET /version` | 200 OK without any auth headers |
| 5 | Default DB connection is used | Check DB queries go to the configured `DB_HOST` | Queries hit single-tenant database |
| 6 | No new required env vars break startup | Start with only the env vars the service had before | Service starts without errors |
**Step 5 — Run multi-tenant test suite (both modes work):**
```bash
# Confirm multi-tenant mode also works
MULTI_TENANT_ENABLED=true MULTI_TENANT_URL=http://tenant-manager:4003 go test ./... -run "MultiTenant"
```
#### Anti-Rationalization
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Nobody uses single-tenant anymore" | Existing deployments depend on it. Breaking them is a production incident for every customer running self-hosted. | **STOP. Run the validation steps.** |
| "We tested multi-tenant, that's enough" | Multi-tenant tests exercise a DIFFERENT code path (`IsMultiTenant()=true`). Single-tenant (`IsMultiTenant()=false`) is a separate path that needs separate verification. | **STOP. Run both test suites.** |
| "The passthrough is trivial, it can't break" | Config struct changes, new required env vars, middleware ordering changes, import side effects — all of these can silently break the passthrough path. | **STOP. Verify with actual requests.** |
| "I'll test single-tenant later" | Later never comes. Once merged, the damage is done. Existing CI may not catch it if tests assume multi-tenant. | **STOP. Test now, before merge.** |
### Checklist
**Environment Variables:**
- [ ] `MULTI_TENANT_ENABLED` in config struct (default: `false`)
- [ ] `MULTI_TENANT_URL` in config struct (required if multi-tenant)
- [ ] `MULTI_TENANT_ENVIRONMENT` in config struct (default: `staging`, only if RabbitMQ)
- [ ] `MULTI_TENANT_MAX_TENANT_POOLS` in config struct (default: `100`)
- [ ] `MULTI_TENANT_IDLE_TIMEOUT_SEC` in config struct (default: `300`)
- [ ] `MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD` in config struct (default: `5`)
- [ ] `MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC` in config struct (default: `30`)
- [ ] `MULTI_TENANT_SERVICE_API_KEY` in config struct (required)
**Architecture:**
- [ ] `client.NewClient(url, logger, opts...)` returns `(*Client, error)` — handle error for fail-fast
- [ ] `client.WithServiceAPIKey(cfg.MultiTenantServiceAPIKey)` always passed (lib-commons validates internally)
- [ ] `tmpostgres.NewManager(client, service, WithModule(...), WithLogger(...), WithMaxTenantPools(...), WithIdleTimeout(...))` for PostgreSQL pool
- [ ] Each manager has `Stats()`, `IsMultiTenant()`, and `ApplyConnectionSettings()` methods
**Middleware — choose one:**
- [ ] For single-module services: `tmmiddleware.NewTenantMiddleware(WithPostgresManager(...))` registered in routes via `WhenEnabled`
- [ ] For multi-module services: `tmmiddleware.NewMultiPoolMiddleware(WithRoute(...), WithDefaultRoute(...))` registered in routes via `WhenEnabled`
- [ ] `WhenEnabled` helper implemented locally in the routes file (nil check → `c.Next()`)
- [ ] Tenant middleware passed as nil when `MULTI_TENANT_ENABLED=false` (single-tenant passthrough via `WhenEnabled` nil check)
**Middleware & Context:**
- [ ] JWT tenant extraction (claim key: `tenantId`)
- [ ] `core.ContextWithTenantID()` in middleware
- [ ] Public endpoints (`/health`, `/version`, `/swagger`) bypass tenant middleware
- [ ] `core.ErrTenantNotFound` → 404, `core.ErrManagerClosed` → 503, `core.ErrServiceNotConfigured` → 503
- [ ] `core.IsTenantNotProvisionedError()` in error handler → 422
- [ ] `core.ErrTenantContextRequired` handled in repositories
- [ ] `ConsumerTrigger` imported from `tmmiddleware` (not from midaz pkg or service-specific packages)
- [ ] `ConsumerTrigger.EnsureConsumerStarted()` called after tenant ID extraction (if using lazy mode)
**Repositories:**
- [ ] `core.ResolvePostgres(ctx, fallback)` in PostgreSQL repositories (single-module services)
- [ ] `core.ResolveModuleDB(ctx, module, fallback)` in PostgreSQL repositories (multi-module services)
- [ ] `valkey.GetKeyFromContext(ctx, key)` for ALL Redis keys (including Lua script KEYS[] and ARGV[])
- [ ] `core.ResolveMongo(ctx, fallback, dbName)` in MongoDB repositories (if using MongoDB)
- [ ] `s3.GetObjectStorageKeyForTenant(ctx, key)` for ALL S3 operations (if using S3/object storage)
**Async Processing:**
- [ ] Tenant ID header (`X-Tenant-ID`) in RabbitMQ messages
- [ ] `consumer.NewMultiTenantConsumer` with `consumer.WithPostgresManager` and `consumer.WithMongoManager`
- [ ] `consumer.Register(queueName, handler)` for each queue
- [ ] `consumer.Run(ctx)` at startup (non-blocking, <1s)
- [ ] `ConsumerTrigger` interface implemented and wired to middleware
**Testing:**
- [ ] Unit tests with mock tenant context
- [ ] Tenant isolation tests (verify data separation between tenants)
- [ ] Error case tests (missing tenant, invalid tenant, tenant not found)
- [ ] Integration tests with two distinct tenants verifying cross-tenant isolation
- [ ] RabbitMQ consumer tests (X-Tenant-ID header extraction)
- [ ] Redis key prefixing tests (verify tenant prefix applied, including Lua scripts)
**Single-Tenant Backward Compatibility (MANDATORY):**
- [ ] All existing tests pass with `MULTI_TENANT_ENABLED=false` (default)
- [ ] Service starts without any `MULTI_TENANT_*` environment variables
- [ ] Service starts without Tenant Manager running
- [ ] All CRUD operations work in single-tenant mode
- [ ] Backward compatibility integration test exists (`TestMultiTenant_BackwardCompatibility`)
- [ ] Health/version endpoints work without tenant context
---
## Route-Level Auth-Before-Tenant Ordering (MANDATORY)
**MANDATORY:** When using multi-tenant middleware, auth MUST validate the JWT **before** tenant middleware resolves the database connection. This ordering is a security requirement, not a performance optimization.
### Why This Matters
| Concern | Impact Without Auth-Before-Tenant |
|---------|-----------------------------------|
| **SECURITY** | Forged or expired JWTs trigger Tenant Manager API calls before token signature validation. Any request with a `tenantId` claim — valid or not — causes a network round-trip to resolve tenant DB credentials. |
| **PERFORMANCE** | Unauthenticated requests trigger unnecessary Tenant Manager API round-trips (~50ms+ each). At scale, this adds significant latency and load to the Tenant Manager service. |
| **DoS VECTOR** | Attackers can flood the Tenant Manager API with crafted tokens containing valid-looking `tenantId` claims. Since tenant resolution happens before auth rejects the token, every malicious request costs a TM API call. |
### The WRONG Pattern (Anti-Pattern)
```go
// ❌ WRONG: Tenant middleware runs before auth on ALL routes
app.Use(tenantMid.WithTenantDB) // Runs first — calls TM API before auth validates JWT
app.Post("/v1/resources", auth.Authorize("app", "resource", "post"), handler.Create)
```
In this pattern, `WithTenantDB` executes for **every request** before `auth.Authorize` validates the JWT. A request with a forged JWT containing `tenantId: "victim-tenant"` triggers a full Tenant Manager resolution — fetching credentials, opening connections — before auth rejects it.
### The CORRECT Pattern: WhenEnabled
**MUST use `WhenEnabled` — a simple helper function that each project implements locally — to conditionally apply tenant middleware per-route.** Auth is listed before `WhenEnabled` in the handler chain, guaranteeing auth runs first. Tenant resolution runs only for authenticated requests.
**`WhenEnabled` implementation (each project implements this locally):**
```go
// WhenEnabled is a helper that conditionally applies a middleware if it's not nil.
// When multi-tenant is disabled, the middleware passed is nil and WhenEnabled calls c.Next().
func WhenEnabled(middleware fiber.Handler) fiber.Handler {
return func(c *fiber.Ctx) error {
if middleware == nil {
return c.Next()
}
return middleware(c)
}
}
```
**Route registration:**
```go
// ✅ CORRECT: Auth validates JWT FIRST, then tenant resolves DB
// ttHandler is nil when MULTI_TENANT_ENABLED=false (single-tenant passthrough)
f.Post("/v1/resources", auth.Authorize("app", "resource", "post"), WhenEnabled(ttHandler), handler.Create)
f.Get("/v1/resources", auth.Authorize("app", "resource", "get"), WhenEnabled(ttHandler), handler.GetAll)
f.Get("/v1/resources/:id", auth.Authorize("app", "resource", "get"), WhenEnabled(ttHandler), handler.GetByID)
```
**How it works:**
1. `auth.Authorize(...)` is the first handler — validates JWT before anything else
2. `WhenEnabled(ttHandler)` runs second — if `ttHandler` is nil (single-tenant mode), it calls `c.Next()` immediately; if non-nil, it executes the tenant middleware
3. The business handler runs last
4. If auth rejects the request, tenant middleware never runs — no TM API call
5. If `MULTI_TENANT_ENABLED=false`, `ttHandler` is nil and `WhenEnabled` is a no-op — zero overhead
### Detection Commands (MANDATORY)
```bash
# MANDATORY: Run before every PR in multi-tenant services
# Check for global tenant middleware registration (anti-pattern)
grep -rn "app\.Use(.*WithTenantDB\|app\.Use(.*tenantMid" internal/ --include="*.go"
# Expected: 0 matches. Tenant middleware MUST NOT be registered globally.
# Check for correct per-route composition: auth.Authorize BEFORE WhenEnabled on same route
grep -rnE '^\s*(app|f)\.(Get|Post|Put|Patch|Delete)\(.*auth\.Authorize\(.*WhenEnabled\(' internal/ --include="*.go"
# Expected: 1+ matches in routes.go — auth appears before WhenEnabled on protected routes.
```
### Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Auth middleware is already global, so order doesn't matter" | Global middleware ordering is implicit and fragile — a refactor can silently break it. Different routes may need different auth handlers. | **MUST use explicit per-route composition: `auth, WhenEnabled(tenant), handler`** |
| "Tenant resolution is fast, no harm running it first" | TM API calls are network round-trips (~50ms+). At scale, unauthorized traffic amplifies cost. Every unauthenticated request wastes a TM API call. | **MUST authenticate before any TM API call** |
| "We'll just register auth middleware before tenant in app.Use()" | Global ordering provides no guarantee per-route. Different routes may need different auth handlers. A single `app.Use()` reorder silently breaks security for all routes. | **MUST compose auth+tenant per-route using WhenEnabled** |
| "Only internal services call this endpoint, no DoS risk" | Internal networks are not trusted by default. Compromised services, misconfigured proxies, or lateral movement can generate unauthorized traffic. | **MUST enforce auth-before-tenant regardless of network topology** |
| "We already validate tokens at the API gateway" | Defense in depth. Gateway validation can be bypassed, misconfigured, or removed. Service-level auth is the last line of defense. | **MUST validate auth at service level before tenant resolution** |
---
## Multi-Tenant Message Queue Consumers (Lazy Mode)
### When to Use
**CONDITIONAL:** Only applies if your service:
- Processes messages from a message broker (RabbitMQ, SQS, etc.)
- Uses per-tenant message isolation (dedicated vhosts or queues per tenant)
- Has 10+ tenants where startup time is a concern
**If single-tenant or <10 tenants:** Multi-tenant mode overhead may be unnecessary. Consider single-tenant architecture.
---
### Problem: Startup Time Scales with Tenant Count
In multi-tenant services consuming messages, connecting to ALL tenant vhosts at startup causes:
```
Startup Time = N tenants × 500ms per connection
10 tenants = 5 seconds
100 tenants = 50 seconds ← Unacceptable for autoscaling/deployments
```
**Symptoms:**
- Slow deployments (rolling updates wait for each pod to connect)
- Poor autoscaling responsiveness (new pods take 30-60s to become ready)
- Wasted resources (connections to inactive tenants)
---
### Solution: Lazy Consumer Initialization
**Pattern:** Decouple tenant discovery from consumer connection.
```
Startup:
1. Discover tenant list (Redis/API) - lightweight, <1s
2. Track discovered tenants in memory (knownTenants map)
3. Do NOT start consumers yet
4. Return immediately (startup complete)
On First Request per Tenant:
1. HTTP middleware extracts tenant ID
2. Middleware calls consumer.EnsureConsumerStarted(ctx, tenantID)
3. Consumer spawns on-demand (first time: ~500ms)
4. Connection cached for reuse
5. Subsequent requests: fast path (<1ms)
```
**Result:** Startup time O(1) regardless of tenant count, resources scale with active tenants only.
---
### Architecture Components
#### 1. Tenant Discovery Service
**Responsibility:** Provide list of active tenants without connection overhead.
**Implementation:**
- **Primary:** Cache (Redis SET: `tenant-manager:tenants:active`)
- **Fallback:** HTTP API (`GET /tenants/active?service={serviceName}`)
**Response format (API):**
```json
[
{"id": "tenant-001", "name": "Tenant A", "status": "active"},
{"id": "tenant-002", "name": "Tenant B", "status": "active"}
]
```
**Endpoint characteristics:**
- Returns minimal info (id, name, status only)
- Supports optional filtering by service (query param: `?service={serviceName}`)
#### 2. Consumer Manager (lib-commons)
**Responsibility:** Manage lifecycle of message consumers across tenants with lazy initialization.
**Key methods:**
```go
type MultiTenantConsumer struct {
mu sync.RWMutex // Protects knownTenants and activeTenants maps
knownTenants map[string]bool // Discovered tenants
activeTenants map[string]CancelFunc // Running consumers
consumerLocks sync.Map // Per-tenant mutexes
retryState sync.Map // Failure tracking
}
// Discover tenants without connecting (non-blocking, <1s)
func (c *MultiTenantConsumer) Run(ctx context.Context) error
// Ensure consumer is active for tenant (idempotent, thread-safe, fire-and-forget)
func (c *MultiTenantConsumer) EnsureConsumerStarted(ctx context.Context, tenantID string)
// Check if tenant has failed repeatedly
func (c *MultiTenantConsumer) IsDegraded(tenantID string) bool
// Get runtime statistics
func (c *MultiTenantConsumer) Stats() ConsumerStats
```
#### 3. HTTP Middleware Trigger
**Responsibility:** Trigger lazy consumer spawn when requests arrive.
**Implementation pattern:**
```go
func TenantMiddleware(consumer ConsumerTrigger) fiber.Handler {
return func(c *fiber.Ctx) error {
// Extract tenant ID from header or JWT
tenantID := c.Get("x-tenant-id")
if tenantID == "" {
return fiber.NewError(400, "x-tenant-id required")
}
// Lazy mode trigger: ensure consumer is active
// First time: spawns consumer (~500ms)
// Subsequent: fast path (<1ms)
if consumer != nil { // Nil-safe for single-tenant mode
ctx := c.UserContext()
consumer.EnsureConsumerStarted(ctx, tenantID)
// Fire-and-forget: consumer retries via background sync if spawn fails
}
// Continue with request processing
return c.Next()
}
}
```
**Placement:** After tenant ID extraction, before database connection resolution.
---
### Implementation Steps
#### Step 1: Update Shared Library
Implement lazy mode in your message queue consumer library:
1. **Add state tracking:**
```go
knownTenants map[string]bool // Discovered via API/cache
activeTenants map[string]CancelFunc // Actually running
consumerLocks sync.Map // Prevent duplicate spawns
```
2. **Non-blocking discovery:**
```go
func (c *Consumer) Run(ctx context.Context) error {
// Discover tenants (timeout: 500ms, soft failure)
c.discoverTenants(ctx)
// Start background sync loop (for tenant add/remove)
go c.runSyncLoop(ctx)
return nil // Return immediately
}
```
3. **On-demand spawning with double-check locking:**
```go
func (c *Consumer) EnsureConsumerStarted(ctx context.Context, tenantID string) {
// Fast path: check if already active (read lock)
c.mu.RLock()
if _, exists := c.activeTenants[tenantID]; exists {
c.mu.RUnlock()
return // Already running
}
c.mu.RUnlock()
// Slow path: acquire per-tenant mutex (prevent thundering herd)
mutex := c.getPerTenantMutex(tenantID)
mutex.Lock()
defer mutex.Unlock()
// Double-check after acquiring lock
c.mu.RLock()
if _, exists := c.activeTenants[tenantID]; exists {
c.mu.RUnlock()
return // Another goroutine created it
}
c.mu.RUnlock()
// Spawn consumer (fire-and-forget, errors logged internally)
c.startTenantConsumer(ctx, tenantID)
}
```
#### Step 2: Add Tenant Discovery Endpoint
In your tenant management service:
```go
// GET /tenants/active?service={serviceName}
func GetActiveTenants(c *fiber.Ctx) error {
service := c.Query("service")
// Query active tenants (filter by service if provided)
tenants, err := repo.ListActiveTenants(service)
if err != nil {
return libHTTP.InternalServerError(c, "TENANT_LIST_FAILED", err)
}
// Return minimal info (no credentials)
response := []TenantSummary{}
for _, t := range tenants {
response = append(response, TenantSummary{
ID: t.ID,
Name: t.Name,
Status: t.Status,
})
}
return libHTTP.OK(c, response)
}
// Register as PUBLIC endpoint (no auth)
app.Get("/tenants/active", GetActiveTenants)
```
#### Step 3: Wire Trigger in Service Middleware
In each service consuming messages:
```go
// In bootstrap/config.go
func InitServers() {
// Create multi-tenant consumer
tmConsumer := consumer.NewMultiTenantConsumer(...)
if err := tmConsumer.Run(ctx); err != nil {
logger.Errorf("tenant manager startup failed: %v", err)
// Note: startup continues - consumer will retry tenant discovery in background
}
// Create middleware with consumer trigger
svc.ttHandler = NewTenantMiddleware(tmConsumer)
// Register per-route in routes.go using WhenEnabled
// See "Route-Level Auth-Before-Tenant Ordering" section
}
// In middleware file
type TenantMiddleware struct {
consumer ConsumerTrigger // Interface with EnsureConsumerStarted method
}
func (m *TenantMiddleware) Handle(c *fiber.Ctx) error {
tenantID := extractTenantID(c) // From header or JWT
// Lazy mode trigger (fire-and-forget, errors logged internally)
if m.consumer != nil {
ctx := c.UserContext()
m.consumer.EnsureConsumerStarted(ctx, tenantID)
}
return c.Next()
}
```
---
### Failure Resilience Pattern
**Exponential Backoff:**
```go
func backoffDelay(retryCount int) time.Duration {
delays := []time.Duration{5*time.Second, 10*time.Second, 20*time.Second, 40*time.Second}
if retryCount >= len(delays) {
return delays[len(delays)-1] // Cap at 40s
}
return delays[retryCount]
}
```
**Per-Tenant Retry State:**
```go
type retryState struct {
count int
degraded bool // True after 3 failures
}
retryState sync.Map // Key: tenantID, Value: *retryState
```
**Degraded Tenant Handling:**
```go
if consumer.IsDegraded(tenantID) {
logger.Errorf("Tenant %s is degraded (3+ connection failures)", tenantID)
return errors.New("tenant degraded")
}
```
---
### Observability Pattern
**Enhanced Stats API:**
```go
type ConsumerStats struct {
ConnectionMode string `json:"connectionMode"` // "lazy"
ActiveTenants int `json:"activeTenants"` // Connected
KnownTenants int `json:"knownTenants"` // Discovered
PendingTenants []string `json:"pendingTenants"` // Known but not active
DegradedTenants []string `json:"degradedTenants"` // Failed 3+ times
}
```
**Structured Logs:**
- `connection_mode=lazy` at startup
- `on-demand consumer start for tenant: {id}` when spawning
- `connecting to vhost: tenant={id}` when connecting
- `tenant {id} marked degraded` after max retries
---
### Testing Strategy
**Unit Tests:**
- Startup completes in <1s (0, 100, 500 tenants)
- Concurrent EnsureConsumerStarted spawns exactly 1 consumer
- Exponential backoff sequence (5s, 10s, 20s, 40s)
- Degraded tenant detection after 3 failures
**Integration Tests:**
- Discovery fallback (Redis → API)
- On-demand connection with testcontainers
- Tenant removal cleanup (<30s)
---
### When to Use Multi-Tenant Lazy Consumer
| Scenario | Recommended | Rationale |
|----------|-------------|-----------|
| **10+ tenants** | ✅ Yes | Startup time becomes significant with many tenants |
| **<10 tenants** | ⚠️ Consider | Overhead may not justify complexity |
| **Most tenants inactive** | ✅ Yes | Resources scale with active count only |
| **All tenants active** | ✅ Yes | Still faster startup, resources scale proportionally |
| **Frequent deployments** | ✅ Yes | Fast startup critical for CI/CD velocity |
| **Latency-sensitive** | ✅ Yes* | *First request per tenant: +500ms (acceptable trade-off) |
---
### Common Pitfalls
**❌ Don't:** Start consumers in discovery loop (defeats lazy purpose)
**✅ Do:** Populate knownTenants only, defer connection to trigger
**❌ Don't:** Use global mutex for all tenants (contention)
**✅ Do:** Per-tenant mutex via sync.Map (fine-grained locking)
**❌ Don't:** Fail HTTP request if consumer spawn fails
**✅ Do:** Log warning, let background sync retry
**❌ Don't:** Forget to cleanup on tenant removal
**✅ Do:** Remove from knownTenants, activeTenants, consumerLocks
**❌ Don't:** Retry indefinitely on connection failure
**✅ Do:** Mark degraded after 3 failures, stop retrying
---
### Single-Tenant vs Multi-Tenant Mode
```go
// Support both single-tenant and multi-tenant deployments
if !config.MultiTenantEnabled {
// Single-tenant: static RabbitMQ connection (no tenant isolation)
consumer = initSingleTenantConsumer(...)
} else {
// Multi-tenant: lazy mode with per-tenant vhosts
consumer = initMultiTenantConsumer(...)
}
```
**Middleware trigger (multi-tenant only):**
```go
if m.consumerTrigger != nil { // Nil in single-tenant mode
m.consumerTrigger.EnsureConsumerStarted(ctx, tenantID)
}
```
**Note:** Single-tenant uses a different consumer implementation without tenant isolation. Multi-tenant consumers ALWAYS use lazy mode for optimal startup performance.
---
## M2M Credentials via Secret Manager (Plugin-Only)
**CONDITIONAL:** Only implement if the service is a **plugin** that needs to authenticate with a **product** (e.g., ledger, midaz, CRM). Products do NOT need this — only plugins that call product APIs.
### When This Applies
| Service Type | `MULTI_TENANT_ENABLED` | Needs Secret Manager? | Reason |
|-------------|------------------------|----------------------|--------|
| **Plugin** (plugin-pix, plugin-crm, etc.) | `true` | ✅ YES | Plugin must authenticate with product APIs per tenant |
| **Plugin** (plugin-pix, plugin-crm, etc.) | `false` | ❌ NO | Single-tenant — plugin uses existing static auth, no Secrets Manager calls |
| **Product** (midaz, ledger, CRM) | any | ❌ NO | Products don't call other products via M2M |
| **Infrastructure** (tenant-manager, reporter) | any | ❌ NO | Infrastructure services use internal auth |
**⛔ Backward Compatibility:** When `MULTI_TENANT_ENABLED=false` (the default), the plugin MUST continue working with its existing authentication mechanism — no AWS Secrets Manager calls, no M2M credential fetching. The Secret Manager path is activated **only** when multi-tenant mode is enabled. This follows the same conditional pattern as all other tenant-manager resources (PostgreSQL, MongoDB, Redis, S3, RabbitMQ).
### How It Works
When `MULTI_TENANT_ENABLED=true`, each tenant has its own M2M credentials stored in **AWS Secrets Manager**. When a plugin needs to call a product API (e.g., ledger), it must:
1. Extract `tenantOrgID` from the JWT context (already available via tenant middleware)
2. Call `secretsmanager.GetM2MCredentials()` to fetch `clientId` + `clientSecret` for that tenant
3. Pass the credentials to the existing Plugin Access Manager integration (which handles JWT acquisition)
**Note:** The plugin already handles JWT token acquisition via Plugin Access Manager. This section only covers **how to retrieve M2M credentials** from AWS Secrets Manager — not how to exchange them for tokens.
### Required lib-commons Package
```go
import (
secretsmanager "github.com/LerianStudio/lib-commons/v4/commons/secretsmanager"
)
```
### Secret Path Convention
Credentials are stored in AWS Secrets Manager following this path:
```
tenants/{env}/{tenantOrgID}/{applicationName}/m2m/{targetService}/credentials
```
| Segment | Source | Example |
|---------|--------|---------|
| `env` | `MULTI_TENANT_ENVIRONMENT` env var | `staging`, `production` |
| `tenantOrgID` | JWT `owner` claim via `auth.GetTenantID(ctx)` | `org_01KHVKQQP6D2N4RDJK0ADEKQX1` |
| `applicationName` | Plugin's own service name constant | `plugin-pix`, `plugin-crm` |
| `targetService` | The product being called | `ledger`, `midaz` |
### Environment Variables (Plugin-Only)
In addition to the 8 canonical multi-tenant env vars, plugins MUST add:
| Env Var | Description | Default | Required |
|---------|-------------|---------|----------|
| `AWS_REGION` | AWS region for Secrets Manager | - | Yes (for plugins) |
| `M2M_TARGET_SERVICE` | Target product service name | - | Yes (for plugins) |
| `M2M_CREDENTIAL_CACHE_TTL_SEC` | Local cache TTL for credentials | `300` | No |
### Implementation Pattern
#### 1. Fetching M2M Credentials
```go
package m2m
import (
"context"
"fmt"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
awssm "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
secretsmanager "github.com/LerianStudio/lib-commons/v4/commons/secretsmanager"
)
// FetchCredentials retrieves M2M credentials from AWS Secrets Manager for a specific tenant.
func FetchCredentials(ctx context.Context, env, tenantOrgID, applicationName, targetService string) (*secretsmanager.M2MCredentials, error) {
cfg, err := awsconfig.LoadDefaultConfig(ctx)
if err != nil {
return nil, fmt.Errorf("loading AWS config: %w", err)
}
client := awssm.NewFromConfig(cfg)
creds, err := secretsmanager.GetM2MCredentials(ctx, client, env, tenantOrgID, applicationName, targetService)
if err != nil {
return nil, fmt.Errorf("fetching M2M credentials for tenant %s: %w", tenantOrgID, err)
}
return creds, nil
}
```
#### 2. Credential Caching (MANDATORY)
**MUST cache credentials locally.** Hitting AWS Secrets Manager on every request is expensive and adds latency.
```go
package m2m
import (
"context"
"fmt"
"sync"
"time"
secretsmanager "github.com/LerianStudio/lib-commons/v4/commons/secretsmanager"
)
type cachedCredentials struct {
creds *secretsmanager.M2MCredentials
expiresAt time.Time
}
// M2MCredentialProvider handles per-tenant M2M credential retrieval with caching.
// Token acquisition is handled by Plugin Access Manager — this only provides credentials.
type M2MCredentialProvider struct {
smClient secretsmanager.SecretsManagerClient
env string
applicationName string
targetService string
credCacheTTL time.Duration
credCache sync.Map // map[tenantOrgID]*cachedCredentials
}
// NewM2MCredentialProvider creates a credential provider with configurable cache TTL.
func NewM2MCredentialProvider(
smClient secretsmanager.SecretsManagerClient,
env, applicationName, targetService string,
credCacheTTL time.Duration,
) *M2MCredentialProvider {
return &M2MCredentialProvider{
smClient: smClient,
env: env,
applicationName: applicationName,
targetService: targetService,
credCacheTTL: credCacheTTL,
}
}
// GetCredentials returns M2M credentials for the given tenant, using cache when possible.
// The caller (Plugin Access Manager integration) handles token acquisition.
func (p *M2MCredentialProvider) GetCredentials(ctx context.Context, tenantOrgID string) (*secretsmanager.M2MCredentials, error) {
if cached, ok := p.credCache.Load(tenantOrgID); ok {
cc := cached.(*cachedCredentials)
if time.Now().Before(cc.expiresAt) {
return cc.creds, nil
}
}
creds, err := secretsmanager.GetM2MCredentials(ctx, p.smClient, p.env, tenantOrgID, p.applicationName, p.targetService)
if err != nil {
return nil, fmt.Errorf("fetching M2M credentials for tenant %s: %w", tenantOrgID, err)
}
p.credCache.Store(tenantOrgID, &cachedCredentials{
creds: creds,
expiresAt: time.Now().Add(p.credCacheTTL),
})
return creds, nil
}
```
#### 3. Single-Tenant vs Multi-Tenant: Conditional Flow
This is the core pattern. The plugin MUST work in both modes — the `M2MCredentialProvider` is **nil** in single-tenant mode.
**Few-shot 1 — Bootstrap wiring (picks the right path at startup):**
```go
// In bootstrap/config.go or bootstrap/dependencies.go
var m2mProvider *m2m.M2MCredentialProvider // nil = single-tenant mode
if cfg.MultiTenantEnabled {
// MULTI-TENANT: create credential provider that fetches from AWS Secrets Manager
awsCfg, err := awsconfig.LoadDefaultConfig(ctx)
if err != nil {
logger.Fatalf("Failed to load AWS config for M2M: %v", err)
}
smClient := awssm.NewFromConfig(awsCfg)
m2mProvider = m2m.NewM2MCredentialProvider(
smClient,
cfg.MultiTenantEnvironment,
constant.ApplicationName,
cfg.M2MTargetService,
time.Duration(cfg.M2MCredentialCacheTTLSec) * time.Second,
)
}
// SINGLE-TENANT: m2mProvider stays nil — no AWS calls, no Secret Manager
// Both modes use the same client — it checks internally if m2mProvider is nil
productClient := product.NewClient(cfg.ProductURL, m2mProvider)
```
**Few-shot 2 — Product client (handles both modes transparently):**
```go
// internal/adapters/product/client.go
type Client struct {
baseURL string
m2mProvider *m2m.M2MCredentialProvider // nil in single-tenant mode
httpClient *http.Client
}
func NewClient(baseURL string, m2mProvider *m2m.M2MCredentialProvider) *Client {
return &Client{
baseURL: baseURL,
m2mProvider: m2mProvider,
httpClient: &http.Client{Timeout: 30 * time.Second},
}
}
func (c *Client) CreateTransaction(ctx context.Context, input TransactionInput) (*TransactionOutput, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/v1/transactions", marshal(input))
if err != nil {
return nil, err
}
if c.m2mProvider != nil {
// MULTI-TENANT: fetch per-tenant credentials from Secret Manager
tenantOrgID := auth.GetTenantID(ctx)
creds, err := c.m2mProvider.GetCredentials(ctx, tenantOrgID)
if err != nil {
return nil, fmt.Errorf("fetching M2M credentials for tenant %s: %w", tenantOrgID, err)
}
req.SetBasicAuth(creds.ClientID, creds.ClientSecret)
}
// SINGLE-TENANT: no credentials injected — plugin uses existing auth
// (e.g., static token from env var, already set in headers by middleware, etc.)
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, fmt.Errorf("calling product API: %w", err)
}
defer resp.Body.Close()
// ... handle response
}
```
**Few-shot 3 — Service layer (no branching needed, client handles it):**
```go
func (uc *ProcessPaymentUseCase) Execute(ctx context.Context, input PaymentInput) error {
// Works in both modes:
// - Single-tenant: client calls product API with existing static auth
// - Multi-tenant: client fetches per-tenant creds from Secret Manager first
resp, err := uc.ledgerClient.CreateTransaction(ctx, input.Transaction)
if err != nil {
return fmt.Errorf("creating transaction in ledger: %w", err)
}
return nil
}
```
**The pattern:** The conditional logic lives in the **client/adapter layer**, not in the service/use-case layer. The service layer calls the same method regardless of mode — it doesn't know or care whether credentials came from Secret Manager or static config.
### Error Handling
The `secretsmanager` package provides sentinel errors for precise error handling:
```go
import (
"errors"
secretsmanager "github.com/LerianStudio/lib-commons/v4/commons/secretsmanager"
)
creds, err := secretsmanager.GetM2MCredentials(ctx, client, env, tenantOrgID, appName, target)
if err != nil {
switch {
case errors.Is(err, secretsmanager.ErrM2MCredentialsNotFound):
// Tenant not provisioned yet — return 503 or queue for retry
case errors.Is(err, secretsmanager.ErrM2MVaultAccessDenied):
// IAM permissions missing or token expired — alert ops
case errors.Is(err, secretsmanager.ErrM2MInvalidCredentials):
// Secret exists but clientId/clientSecret missing — alert ops
default:
// Infrastructure error — retry with backoff
}
}
```
### AWS IAM Permissions
The plugin's IAM role (or ECS task role / EKS service account) MUST have permission to read the tenant secrets. Minimal policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:*:*:secret:tenants/*/m2m/*/credentials-*"
}
]
}
```
For tighter scoping, replace wildcards with specific values:
| Wildcard | Scoped Example |
|----------|---------------|
| First `*` (env) | `production` or `staging` |
| Second `*` (target) | `ledger` or `midaz` |
| Trailing `-*` | Required — AWS appends a random suffix to secret ARNs |
**MUST NOT** grant `secretsmanager:*` — only `GetSecretValue` is needed.
### Testing Guidance
The `secretsmanager.SecretsManagerClient` is an interface, so it can be mocked in unit tests without hitting AWS.
#### Mocking the client
```go
type mockSMClient struct {
getSecretValueFunc func(ctx context.Context, input *awssm.GetSecretValueInput, opts ...func(*awssm.Options)) (*awssm.GetSecretValueOutput, error)
}
func (m *mockSMClient) GetSecretValue(ctx context.Context, input *awssm.GetSecretValueInput, opts ...func(*awssm.Options)) (*awssm.GetSecretValueOutput, error) {
return m.getSecretValueFunc(ctx, input, opts...)
}
```
#### Testing credential cache TTL
```go
func TestCredentialCacheExpiry(t *testing.T) {
callCount := 0
mock := &mockSMClient{
getSecretValueFunc: func(_ context.Context, _ *awssm.GetSecretValueInput, _ ...func(*awssm.Options)) (*awssm.GetSecretValueOutput, error) {
callCount++
secret := `{"clientId":"id","clientSecret":"secret"}`
return &awssm.GetSecretValueOutput{SecretString: &secret}, nil
},
}
provider := NewM2MCredentialProvider(mock, "test", "plugin-pix", "ledger", 1*time.Second)
// First call — fetches from AWS
_, err := provider.GetCredentials(context.Background(), "tenant-1")
require.NoError(t, err)
assert.Equal(t, 1, callCount)
// Second call — served from cache
_, err = provider.GetCredentials(context.Background(), "tenant-1")
require.NoError(t, err)
assert.Equal(t, 1, callCount) // still 1
// Wait for cache expiry
time.Sleep(1100 * time.Millisecond)
// Third call — cache expired, fetches again
_, err = provider.GetCredentials(context.Background(), "tenant-1")
require.NoError(t, err)
assert.Equal(t, 2, callCount)
}
```
#### Testing error scenarios
Test each sentinel error path (`ErrM2MCredentialsNotFound`, `ErrM2MVaultAccessDenied`, `ErrM2MInvalidCredentials`) by returning the corresponding error from the mock.
### Observability & Metrics
Instrument `M2MCredentialProvider` to track credential retrieval health. Recommended counters and histogram:
| Metric | Type | Where to Increment | Description |
|--------|------|-------------------|-------------|
| `m2m_credential_cache_hits` | Counter | `GetCredentials` — cache hit path | Credential served from local cache |
| `m2m_credential_cache_misses` | Counter | `GetCredentials` — cache miss path | Cache miss, fetching from AWS |
| `m2m_credential_fetch_errors` | Counter | `GetCredentials` — error return | AWS Secrets Manager call failed |
| `m2m_credential_fetch_duration_seconds` | Histogram | `GetCredentials` — around the `GetM2MCredentials` call | Latency of AWS Secrets Manager requests |
Labels: `tenant_org_id`, `target_service`, `environment`.
**MUST NOT** include `clientId` or `clientSecret` in metric labels or log fields.
### Security Considerations
1. **MUST NOT log credentials** — never log `clientId` or `clientSecret` values
2. **MUST NOT store credentials in environment variables** — always fetch from Secrets Manager at runtime
3. **MUST cache locally** — avoid per-request AWS API calls (latency + cost)
4. **MUST handle credential rotation** — cache TTL ensures stale credentials are refreshed automatically
### Anti-Rationalization
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Product service doesn't need M2M" | Correct. Only plugins need M2M. | **Skip this gate for products** |
| "We can hardcode credentials per tenant" | Hardcoded creds don't scale and are a security risk. | **MUST use Secrets Manager** |
| "Caching is optional, we'll add it later" | Every request hitting AWS adds ~50-100ms latency + cost. | **MUST implement caching from day one** |
| "We'll use env vars for client credentials" | Env vars are shared across tenants. M2M is per-tenant. | **MUST use Secrets Manager per tenant** |
| "Single-tenant plugins don't need this" | Correct if MULTI_TENANT_ENABLED=false. | **Skip when single-tenant** |
---
## Service Authentication (MANDATORY)
Consumer services that call the Tenant Manager `/settings` endpoint MUST authenticate using an API key sent via the `X-API-Key` HTTP header. Without this header, the Tenant Manager rejects requests to protected endpoints.
### How It Works
1. **Key generation:** API keys are generated per-service via the service catalog endpoint `POST /services/:name/api-keys`.
2. **Key limit:** Maximum 2 keys per environment per service, enabling zero-downtime rotation (create new key, roll out, revoke old key).
3. **Header injection:** The lib-commons Tenant Manager HTTP client sends the `X-API-Key` header automatically when configured with `client.WithServiceAPIKey()`.
4. **Consumer configuration:** Consumer services set the `MULTI_TENANT_SERVICE_API_KEY` environment variable. The bootstrap code passes it to the client via `WithServiceAPIKey`.
### Configuration
Add `MULTI_TENANT_SERVICE_API_KEY` to the Config struct (see [Environment Variables](#environment-variables)):
```go
MultiTenantServiceAPIKey string `env:"MULTI_TENANT_SERVICE_API_KEY"`
```
Wire it when creating the Tenant Manager HTTP client:
```go
clientOpts = append(clientOpts,
client.WithServiceAPIKey(cfg.MultiTenantServiceAPIKey),
)
tmClient, err := client.NewClient(cfg.MultiTenantURL, logg
... (truncated)
```
### ../dev-multi-tenant/SKILL.md
```markdown
---
name: ring:dev-multi-tenant
slug: dev-multi-tenant
version: 2.0.0
type: skill
description: |
Multi-tenant development cycle orchestrator following Ring Standards.
Auto-detects the service stack (PostgreSQL, MongoDB, Redis, RabbitMQ, S3)
and service type (plugin vs product),
then executes a gate-based implementation using tenantId from JWT
for database-per-tenant isolation via lib-commons v4 tenant-manager sub-packages (postgres.Manager, mongo.Manager).
For plugins: includes mandatory M2M credential retrieval from AWS Secrets Manager
via lib-commons v4 secretsmanager package (per-tenant authentication with product APIs).
MUST update lib-commons v4 first; lib-auth v2 depends on it. Both are required dependencies.
Each gate dispatches ring:backend-engineer-golang with context and section references.
The agent loads multi-tenant.md via WebFetch and has all code examples.
trigger: |
- User requests multi-tenant implementation for a Go service
- User asks to add tenant isolation to an existing service
- Task mentions "multi-tenant", "tenant isolation", "tenant-manager", "postgres.Manager", "MultiPoolMiddleware"
prerequisite: |
- Go service with existing single-tenant functionality
NOT_skip_when: |
- "organization_id already exists" → organization_id is NOT multi-tenant. tenantId via JWT is required.
- "Just need to connect the wiring" → Multi-tenant requires lib-commons v4 tenant-manager sub-packages.
- "lib-commons v4 upgrade is too risky" → REQUIRES lib-commons v4 tenant-manager sub-packages. No v3 = no multi-tenant.
- "Service already has multi-tenant" → Existence ≠ compliance. MUST replace non-standard implementations with Ring canonical model.
- "Multi-tenant is already done" → Every gate verifies compliance. MUST fix non-compliant code — it is wrong, not done.
sequence:
after: [ring:dev-devops]
related:
complementary: [ring:dev-cycle, ring:dev-implementation, ring:dev-devops, ring:dev-unit-testing, ring:requesting-code-review, ring:dev-validation]
input_schema:
description: |
When invoked from ring:dev-cycle (post-cycle step), receives structured handoff context.
When invoked standalone (direct user request), these fields are auto-detected in Gate 0.
fields:
- name: execution_mode
type: string
enum: ["FULL", "SCOPED"]
description: "FULL = complete 12-gate cycle. SCOPED = only adapt new files (when existing MT is compliant)."
required: false
default: "FULL"
- name: files_changed
type: array
items: string
description: "File paths changed during dev-cycle (only in SCOPED mode). Used to limit Gate 5 scope."
required: false
- name: multi_tenant_exists
type: boolean
description: "Whether multi-tenant code was detected by dev-cycle Step 1.5."
required: false
- name: multi_tenant_compliant
type: boolean
description: "Whether existing MT code passed compliance audit in dev-cycle Step 1.5."
required: false
- name: detected_dependencies
type: object
properties:
postgresql: boolean
mongodb: boolean
redis: boolean
rabbitmq: boolean
s3: boolean
description: "Stack detection from dev-cycle. Each key indicates whether that technology was detected."
required: false
- name: skip_gates
type: array
items: string
description: "Gate identifiers to skip (e.g., '0', '1.5', '5.5'). Set by dev-cycle based on execution_mode."
required: false
output_schema:
format: markdown
required_sections:
- name: "Multi-Tenant Cycle Summary"
pattern: "^## Multi-Tenant Cycle Summary"
required: true
- name: "Stack Detection"
pattern: "^## Stack Detection"
required: true
- name: "Gate Results"
pattern: "^## Gate Results"
required: true
- name: "Verification"
pattern: "^## Verification"
required: true
metrics:
- name: gates_passed
type: integer
- name: gates_failed
type: integer
- name: total_files_changed
type: integer
examples:
- name: "Add multi-tenant to a service"
invocation: "/ring:dev-multi-tenant"
expected_flow: |
1. Gate 0: Auto-detect stack + service type (plugin vs product)
2. Gate 1: Analyze codebase (build implementation roadmap)
3. Gate 1.5: Visual implementation preview (HTML report for developer approval)
4. Gates 2-5: Implementation (agent loads multi-tenant.md, follows roadmap)
5. Gate 5.5: M2M Secret Manager for plugin auth (if plugin)
6. Gate 6: RabbitMQ multi-tenant (if RabbitMQ detected)
7. Gate 7: Metrics & Backward compatibility
8. Gate 8: Tests
9. Gate 9: Code review
10. Gate 10: User validation
11. Gate 11: Activation guide
---
# Multi-Tenant Development Cycle
<cannot_skip>
## CRITICAL: This Skill ORCHESTRATES. Agents IMPLEMENT.
| Who | Responsibility |
|-----|----------------|
| **This Skill** | Detect stack, determine gates, pass context to agent, verify outputs, enforce order |
| **ring:backend-engineer-golang** | Load multi-tenant.md via WebFetch, implement following the standards |
| **7 reviewers** | Review at Gate 9 |
**CANNOT change scope:** the skill defines WHAT to implement. The agent implements HOW.
**FORBIDDEN: Orchestrator MUST NOT use Edit, Write, or Bash tools to modify source code files.**
All code changes MUST go through `Task(subagent_type="ring:backend-engineer-golang")`.
The orchestrator only verifies outputs (grep, go build, go test) — MUST NOT write implementation code.
**MANDATORY: TDD for all implementation gates (Gates 2-6).** MUST follow RED → GREEN → REFACTOR: write a failing test first, then implement to make it pass, then refactor for clarity/performance. MUST include in every dispatch: "Follow TDD: write failing test (RED), implement to make it pass (GREEN), then refactor for clarity/performance (REFACTOR)."
</cannot_skip>
---
## Multi-Tenant Architecture
Multi-tenant isolation is 100% based on `tenantId` from JWT → tenant-manager middleware → database-per-tenant. Connection managers (`postgres.Manager`, `mongo.Manager`, `rabbitmq.Manager`) resolve tenant-specific credentials via the Tenant Manager API. `organization_id` is NOT part of multi-tenant.
**Standards reference:** All code examples and implementation patterns are in [multi-tenant.md](../../docs/standards/golang/multi-tenant.md). MUST load via WebFetch before implementing any gate.
**WebFetch URL:** `https://raw.githubusercontent.com/LerianStudio/ring/main/dev-team/docs/standards/golang/multi-tenant.md`
### MANDATORY: Canonical Environment Variables
See [multi-tenant.md § Environment Variables](../../docs/standards/golang/multi-tenant.md#environment-variables) for the complete table of 8 canonical `MULTI_TENANT_*` env vars with descriptions, defaults, and required status.
MUST NOT use any other names (e.g., `TENANT_MANAGER_ADDRESS` is WRONG — the correct name is `MULTI_TENANT_URL`).
HARD GATE: Any env var outside that table is non-compliant. Agent MUST NOT invent or accept alternative names.
### MANDATORY: Canonical Metrics
See [multi-tenant.md § Multi-Tenant Metrics](../../docs/standards/golang/multi-tenant.md#multi-tenant-metrics) for the 4 required metrics. All 4 are MANDATORY.
When `MULTI_TENANT_ENABLED=false`, metrics MUST use no-op implementations (zero overhead in single-tenant mode).
### MANDATORY: Circuit Breaker
See [multi-tenant.md § Environment Variables](../../docs/standards/golang/multi-tenant.md#environment-variables) for circuit breaker env vars (`MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD`, `MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC`).
The Tenant Manager HTTP client MUST enable `WithCircuitBreaker`. MUST NOT create the client without it.
HARD GATE: A client without circuit breaker can cascade failures across all tenants.
### MANDATORY: Service API Key
See [multi-tenant.md § Service Authentication](../../docs/standards/golang/multi-tenant.md#service-authentication-mandatory) for the authentication flow.
The Tenant Manager HTTP client MUST be configured with `client.WithServiceAPIKey(cfg.MultiTenantServiceAPIKey)`. Without this, the Tenant Manager rejects requests to the `/settings` endpoint with `401 Unauthorized`.
HARD GATE: A client without `WithServiceAPIKey` cannot resolve tenant connections.
### MANDATORY: Sub-Package Import Reference
Agents must use these exact import paths. Include this table in every gate dispatch to prevent hallucinated or outdated imports.
| Alias | Import Path | Purpose |
|-------|-------------|---------|
| `client` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/client` | Tenant Manager HTTP client with circuit breaker |
| `core` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/core` | Context helpers, resolvers, errors, types |
| `tmmiddleware` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/middleware` | TenantMiddleware, MultiPoolMiddleware, ConsumerTrigger |
| `tmpostgres` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/postgres` | PostgresManager (per-tenant PG pools) |
| `tmmongo` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/mongo` | MongoManager (per-tenant Mongo pools) |
| `tmrabbitmq` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/rabbitmq` | RabbitMQ Manager (per-tenant vhosts) |
| `tmconsumer` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/consumer` | MultiTenantConsumer (lazy mode) |
| `valkey` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/valkey` | Redis key prefixing (GetKeyFromContext) |
| `s3` | `github.com/LerianStudio/lib-commons/v4/commons/tenant-manager/s3` | S3 key prefixing (GetObjectStorageKeyForTenant) |
| `secretsmanager` | `github.com/LerianStudio/lib-commons/v4/commons/secretsmanager` | M2M credential retrieval (plugin only) |
**⛔ HARD GATE:** Agent must not use v2 import paths or invent sub-package paths. If WebFetch truncates, this table is the authoritative reference.
### MANDATORY: Isolation Modes
The Tenant Manager determines the isolation mode per tenant. Agents MUST handle both:
| Mode | Database | Schema | Connection String Modifier | When |
|------|----------|--------|---------------------------|------|
| `isolated` (default) | Separate DB per tenant | Default `public` | None | Strong isolation, recommended |
| `schema` | Shared DB | Schema per tenant | `options=-csearch_path="{schema}"` | Cost optimization |
The agent does not choose the mode — lib-commons `postgres.Manager` reads `TenantConfig.IsolationMode` from the Tenant Manager API and resolves the connection accordingly. The agent's responsibility is to use `core.ResolvePostgres`/`core.ResolveModuleDB` which handles both modes transparently.
### ConnectionSettings Override
Connection managers support per-tenant pool overrides via `TenantConfig.Databases[module].ConnectionSettings`:
- `MaxOpenConns` and `MaxIdleConns` per tenant
- When present, these override global defaults on `PostgresManager`/`MongoManager`
- When nil (older tenant associations), global defaults apply
- Managers call `ApplyConnectionSettings()` automatically after resolving a connection
Agents must not hardcode pool sizes — the Tenant Manager controls per-tenant pool tuning.
### MANDATORY: Agent Instruction (include in EVERY gate dispatch)
MUST include these instructions in every dispatch to `ring:backend-engineer-golang`:
> **STANDARDS: WebFetch `https://raw.githubusercontent.com/LerianStudio/ring/main/dev-team/docs/standards/golang/multi-tenant.md` and follow the sections referenced below. All code examples, patterns, and implementation details are in that document. Use them as-is.**
>
> **SUB-PACKAGES: Use the import table from the skill — see "Sub-Package Import Reference" above. Do NOT invent import paths.**
>
> **TDD: For implementation gates (2-6), follow TDD methodology — write a failing test first (RED), then implement to make it pass (GREEN). MUST have test coverage for every change.**
---
## Severity Calibration
| Severity | Criteria | Examples |
|----------|----------|----------|
| **CRITICAL** | Cross-tenant data leak, security vulnerability | Tenant A sees Tenant B data, missing tenant validation, hardcoded creds |
| **HIGH** | Missing tenant isolation, wrong env vars | No TenantMiddleware, TENANT_MANAGER_ADDRESS instead of MULTI_TENANT_URL |
| **MEDIUM** | Configuration gaps, partial implementation | Missing circuit breaker, incomplete metrics |
| **LOW** | Documentation, optimization | Missing env var comments, pool tuning |
MUST report all severities. CRITICAL: STOP immediately (security breach). HIGH: Fix before gate pass. MEDIUM: Fix in iteration. LOW: Document.
---
## Pressure Resistance
| User Says | This Is | Response |
|-----------|---------|----------|
| "It already has multi-tenant, skip this gate" | COMPLIANCE_BYPASS | "Existence ≠ compliance. MUST run compliance audit. If it doesn't match the Ring canonical model exactly, it is non-compliant and MUST be replaced." |
| "Multi-tenant is done, just review it" | COMPLIANCE_BYPASS | "CANNOT skip gates. Every gate verifies compliance OR implements. Non-standard implementations are not 'done' — they are wrong." |
| "Our custom approach works the same way" | COMPLIANCE_BYPASS | "Working ≠ compliant. Only lib-commons v4 tenant-manager sub-packages are valid. Custom implementations create drift and block upgrades." |
| "Skip the lib-commons upgrade" | QUALITY_BYPASS | "CANNOT proceed without lib-commons v4. Tenant-manager sub-packages do not exist in v2." |
| "Just do the happy path, skip backward compat" | SCOPE_REDUCTION | "Backward compatibility is NON-NEGOTIABLE. Single-tenant deployments depend on it." |
| "organization_id is our tenant identifier" | AUTHORITY_OVERRIDE | "STOP. organization_id is NOT multi-tenant. tenantId from JWT is the only mechanism." |
| "Skip code review, we tested it" | QUALITY_BYPASS | "MANDATORY: 7 reviewers. One security mistake = cross-tenant data leak." |
| "We don't need RabbitMQ multi-tenant" | SCOPE_REDUCTION | "MUST execute Gate 6 if RabbitMQ was detected. CANNOT skip detected stack." |
| "I'll make a quick edit directly" | CODE_BYPASS | "FORBIDDEN: All code changes go through ring:backend-engineer-golang. Dispatch the agent." |
| "It's just one line, no need for an agent" | CODE_BYPASS | "FORBIDDEN: Even single-line changes MUST be dispatched. Agent ensures standards compliance." |
| "Agent is slow, I'll edit faster" | CODE_BYPASS | "FORBIDDEN: Speed is not a justification. Agent applies TDD and standards checks." |
| "This plugin doesn't need Secret Manager" | SCOPE_REDUCTION | "If it's a plugin with MULTI_TENANT_ENABLED=true that calls product APIs, M2M via Secret Manager is MANDATORY." |
| "We can use env vars for M2M credentials" | SECURITY_BYPASS | "Env vars are shared across tenants. M2M credentials are PER-TENANT. MUST use Secrets Manager." |
| "Caching M2M credentials is optional" | QUALITY_BYPASS | "Every request to AWS adds ~50-100ms latency + cost. Caching is MANDATORY from day one." |
---
## Gate Overview
| Gate | Name | Condition | Agent |
|------|------|-----------|-------|
| 0 | Stack Detection + Compliance Audit | Always | Orchestrator |
| 1 | Codebase Analysis (multi-tenant focus) | Always | ring:codebase-explorer |
| 1.5 | Implementation Preview (visual report) | Always | Orchestrator (ring:visual-explainer) |
| 2 | lib-commons v4 + lib-auth v2 Upgrade | Skip only if `go.mod` contains `lib-commons/v4` AND `lib-auth/v2` (verified via grep) | ring:backend-engineer-golang |
| 3 | Multi-Tenant Configuration | Always — verify compliance or implement/fix | ring:backend-engineer-golang |
| 4 | Tenant Middleware (TenantMiddleware or MultiPoolMiddleware) | Always — verify compliance or implement/fix | ring:backend-engineer-golang |
| 5 | Repository Adaptation | Always per detected DB/storage — verify compliance or implement/fix | ring:backend-engineer-golang |
| 5.5 | M2M Secret Manager (Plugin Auth) | Skip if NOT a plugin | ring:backend-engineer-golang |
| 6 | RabbitMQ Multi-Tenant | Skip if no RabbitMQ | ring:backend-engineer-golang |
| 7 | Metrics & Backward Compat | Always | ring:backend-engineer-golang |
| 8 | Tests | Always | ring:backend-engineer-golang |
| 9 | Code Review | Always | 7 parallel reviewers |
| 10 | User Validation | Always | User |
| 11 | Activation Guide | Always | Orchestrator |
MUST execute gates sequentially. CANNOT skip or reorder.
### Input Validation (when invoked from dev-cycle)
If this skill receives structured input from ring:dev-cycle (post-cycle handoff):
```text
VALIDATE input:
1. execution_mode MUST be "FULL" or "SCOPED"
2. If execution_mode == "SCOPED":
- files_changed MUST be non-empty (otherwise there's nothing to adapt)
- multi_tenant_exists MUST be true
- multi_tenant_compliant MUST be true
- skip_gates MUST include ["0", "1.5", "2", "3", "4", "10", "11"]
3. If execution_mode == "FULL":
- Core gates always execute (0, 1, 1.5, 2, 3, 4, 5, 7, 8, 9)
- Conditional gates may be in skip_gates based on stack detection:
- "5.5" may be skipped if service is NOT a plugin
- "6" may be skipped if RabbitMQ was NOT detected
- "10", "11" may be skipped when invoked from dev-cycle
- skip_gates MUST NOT contain core gates (0-5, 7-9)
4. detected_dependencies (if provided) is used to pre-populate Gate 0 stack detection
— still MUST verify with grep commands (trust but verify)
If invoked standalone (no input_schema fields):
- Default to execution_mode = "FULL"
- Run full Gate 0 stack detection
```
<cannot_skip>
### HARD GATE: Existence ≠ Compliance
**"The service already has multi-tenant code" is NOT a reason to skip any gate.**
MUST replace existing multi-tenant code that does not follow the Ring canonical model — it is **non-compliant**. The only valid reason to skip a gate is when the existing implementation has been **verified** to match the exact patterns defined in [multi-tenant.md](../../docs/standards/golang/multi-tenant.md).
**Compliance verification requires EVIDENCE, not assumption.** See [multi-tenant.md § HARD GATE: Canonical Model Compliance](../../docs/standards/golang/multi-tenant.md#hard-gate-canonical-model-compliance) for the canonical list of compliant patterns. The Gate 0 Phase 2 compliance audit (A1-A8 grep checks) verifies each component against those patterns.
**If ANY audit check is NON-COMPLIANT → the corresponding gate MUST execute to fix it. CANNOT skip.**
</cannot_skip>
---
## Gate 0: Stack Detection + Compliance Audit
**Orchestrator executes directly. No agent dispatch.**
**This gate has TWO phases: detection AND compliance audit.**
### Phase 1: Stack Detection
```text
DETECT (run in parallel):
1. lib-commons version: grep "lib-commons" go.mod
1b. lib-auth v2: grep "lib-auth" go.mod
2. PostgreSQL: grep -rn "postgresql\|pgx\|squirrel" internal/ go.mod
3. MongoDB: grep -rn "mongodb\|mongo" internal/ go.mod
4. Redis: grep -rn "redis\|valkey" internal/ go.mod
5. RabbitMQ: grep -rn "rabbitmq\|amqp" internal/ go.mod
6. S3/Object Storage: grep -rn "s3\|ObjectStorage\|PutObject\|GetObject\|Upload.*storage\|Download.*storage" internal/ pkg/ go.mod
7. Existing multi-tenant:
- Config: grep -rn "MULTI_TENANT_ENABLED" internal/
- Middleware: grep -rn "tenant-manager/middleware\|WithTenantDB\|MultiPoolMiddleware" internal/
- Context: grep -rn "tenant-manager/core\|ResolveMongo\|ResolvePostgres\|ResolveModuleDB" internal/
- S3 keys: grep -rn "tenant-manager/s3\|GetObjectStorageKeyForTenant" internal/
- RMQ: grep -rn "X-Tenant-ID" internal/
8. Service type (plugin vs product):
- Plugin: grep -rn "plugin-\|Plugin" go.mod cmd/ internal/bootstrap/
- M2M client: grep -rn "client_credentials\|M2M\|secretsmanager\|GetM2MCredentials" internal/ pkg/
- Product API calls: grep -rn "ledger.*client\|midaz.*client\|product.*client" internal/
```
### Phase 2: Compliance Audit (MANDATORY if any multi-tenant code detected)
If Phase 1 detects any existing multi-tenant code (step 7 returns results), MUST run a compliance audit. MUST replace existing code that does not match the Ring canonical model — it is not "partially done", it is **wrong**.
```text
AUDIT (run in parallel — only if step 7 found existing multi-tenant code):
NOTE: A1 is a NEGATIVE check (presence of wrong names = NON-COMPLIANT).
A2-A8 are POSITIVE checks (absence of canonical patterns = NON-COMPLIANT).
A1. Config compliance:
- grep -rn "TENANT_MANAGER_ADDRESS\|TENANT_URL\|TENANT_MANAGER_URL" internal/
- (any match = NON-COMPLIANT config var names → Gate 3 MUST fix)
A2. Middleware compliance:
- grep -rn "tmmiddleware.NewTenantMiddleware\|tmmiddleware.NewMultiPoolMiddleware" internal/
- (no match but other tenant middleware exists = NON-COMPLIANT → Gate 4 MUST fix)
A3. Repository compliance:
- grep -rn "core.ResolvePostgres\|core.ResolveMongo\|core.ResolveModuleDB" internal/
- (repositories use static connections or custom pool lookup = NON-COMPLIANT → Gate 5 MUST fix)
A4. Redis compliance (if Redis detected):
- grep -rn "valkey.GetKeyFromContext" internal/
- (Redis operations without GetKeyFromContext = NON-COMPLIANT → Gate 5 MUST fix)
A5. S3 compliance (if S3 detected):
- grep -rn "s3.GetObjectStorageKeyForTenant" internal/
- (S3 operations without GetObjectStorageKeyForTenant = NON-COMPLIANT → Gate 5 MUST fix)
A6. RabbitMQ compliance (if RabbitMQ detected):
- grep -rn "tmrabbitmq.NewManager\|tmrabbitmq.Manager" internal/
- (RabbitMQ multi-tenant without tmrabbitmq.Manager = NON-COMPLIANT → Gate 6 MUST fix)
A7. Circuit breaker compliance:
- grep -rn "WithCircuitBreaker" internal/
- (Tenant Manager client without circuit breaker = NON-COMPLIANT → Gate 4 MUST fix)
A8. Backward compatibility compliance:
- grep -rn "TestMultiTenant_BackwardCompatibility" internal/
- (no backward compat test = NON-COMPLIANT → Gate 7 MUST fix)
A9. Service API key compliance:
- grep -rn "MULTI_TENANT_SERVICE_API_KEY" internal/
- grep -rn "WithServiceAPIKey" internal/
- (MULTI_TENANT_SERVICE_API_KEY missing from config OR WithServiceAPIKey not called on client = NON-COMPLIANT → Gate 3/4 MUST fix)
```
**Output format for compliance audit:**
```text
COMPLIANCE AUDIT RESULTS:
| Component | Status | Evidence | Gate Action |
|-----------|--------|----------|-------------|
| Config vars | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 3: SKIP / MUST FIX |
| Middleware | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 4: SKIP / MUST FIX |
| Repositories | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 5: SKIP / MUST FIX |
| Redis keys | COMPLIANT / NON-COMPLIANT / N/A | {grep results} | Gate 5: SKIP / MUST FIX |
| S3 keys | COMPLIANT / NON-COMPLIANT / N/A | {grep results} | Gate 5: SKIP / MUST FIX |
| RabbitMQ | COMPLIANT / NON-COMPLIANT / N/A | {grep results} | Gate 6: SKIP / MUST FIX |
| Circuit breaker | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 4: SKIP / MUST FIX |
| Backward compat test | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 7: SKIP / MUST FIX |
| Service API key | COMPLIANT / NON-COMPLIANT | {grep results} | Gate 3/4: SKIP / MUST FIX |
```
**HARD GATE: A gate can only be marked as SKIP when ALL its compliance checks are COMPLIANT with evidence. One NON-COMPLIANT row → gate MUST execute.**
### Phase 3: Non-Canonical File Detection (MANDATORY)
MUST scan for multi-tenant logic in files outside the canonical file map. See [multi-tenant.md § Canonical File Map](../../docs/standards/golang/multi-tenant.md#canonical-file-map) for the complete list of valid files.
```text
DETECT non-canonical multi-tenant files:
N1. Custom tenant middleware:
grep -rn "tenant" internal/middleware/ pkg/middleware/ --include="*.go" | grep -v "_test.go"
(any match = NON-CANONICAL file → MUST be removed and replaced with lib-commons middleware)
N2. Custom tenant resolvers/managers:
grep -rln "tenant" internal/tenant/ internal/multitenancy/ pkg/tenant/ pkg/multitenancy/ --include="*.go" 2>/dev/null
(any match = NON-CANONICAL file → MUST be removed)
N3. Custom pool managers:
grep -rln "pool.*tenant\|tenant.*pool" internal/ pkg/ --include="*.go" | grep -v "tenant-manager"
(any match outside lib-commons = NON-CANONICAL → MUST be removed)
```
**If non-canonical files are found:** report them in the compliance audit as `NON-CANONICAL FILES DETECTED`. The implementing agent MUST remove these files and replace their functionality with the canonical lib-commons v4 sub-packages during the appropriate gate.
**Service type classification:**
| Signal | Classification |
|--------|---------------|
| Module name contains `plugin-` (in go.mod) | **Plugin** → Gate 5.5 MANDATORY |
| Service calls product APIs (ledger, midaz, etc.) | **Plugin** → Gate 5.5 MANDATORY |
| No product API calls, serves own data | **Product** → Gate 5.5 SKIP |
MUST confirm with user: "Is this service a **plugin** (calls product APIs like ledger/midaz) or a **product** (serves its own data)?"
MUST confirm: user explicitly approves detection results before proceeding.
---
## Gate 1: Codebase Analysis (Multi-Tenant Focus)
**Always executes. This gate builds the implementation roadmap for all subsequent gates.**
**Dispatch `ring:codebase-explorer` with multi-tenant-focused context:**
> TASK: Analyze this codebase exclusively under the multi-tenant perspective.
> DETECTED STACK: {databases and messaging from Gate 0}
>
> CRITICAL: Multi-tenant is ONLY about tenantId from JWT → tenant-manager middleware → database-per-tenant.
> IGNORE organization_id completely — it is NOT multi-tenant. A tenant can have multiple organizations inside its database. organization_id is a domain entity, not a tenant identifier.
>
> FOCUS AREAS (explore ONLY these — ignore everything else):
>
> 1. **Service name, modules, and components**: What is the service called? (Look for `const ApplicationName`.) How many components/modules does it have? Each module needs a constant (e.g., `const ModuleManager = "manager"`). Identify: service name (ApplicationName), module names per component, and whether constants exist or need to be created. Hierarchy: Service → Module → Resource.
> 2. **Bootstrap/initialization**: Where does the service start? Where are database connections created? Where is the middleware chain registered? Identify the exact insertion point for TenantMiddleware.
> 3. **Database connections**: How do repositories get their DB connection today? Static field in struct? Constructor injection? Context? List EVERY repository file with file:line showing where the connection is obtained.
> 4. **Middleware chain**: What middleware exists and in what order? Where would TenantMiddleware fit (after auth, before handlers)?
> 5. **Config struct**: Where is the Config struct? What fields exist? Where is it loaded? Identify exact location for MULTI_TENANT_ENABLED vars.
> 6. **RabbitMQ** (if detected): Where are producers? Where are consumers? How are messages published? Where would X-Tenant-ID header be injected? Are producer and consumer in the SAME process or SEPARATE components? Is there already a config split? Are there dual constructors? Is there a RabbitMQManager pool? Does the service struct have both consumer types?
> 7. **Redis** (if detected): Where are Redis operations? Any Lua scripts? Where would GetKeyFromContext be needed?
> 8. **S3/Object Storage** (if detected): Where are Upload/Download/Delete operations? How are object keys constructed? List every file:line that builds an S3 key. What bucket env var is used?
> 9. **Existing multi-tenant code**: Any tenant-manager sub-package imports (`tenant-manager/core`, `tenant-manager/middleware`, `tenant-manager/postgres`, etc.)? TenantMiddleware or MultiPoolMiddleware? `core.ResolvePostgres`/`core.ResolveMongo`/`core.ResolveModuleDB`/`s3.GetObjectStorageKeyForTenant` calls? MULTI_TENANT_ENABLED config? (NOTE: organization_id is NOT related to multi-tenant — ignore it completely)
> 10. **M2M / Plugin authentication** (if service is a plugin): Does the service call product APIs (ledger, midaz, CRM)? How does it authenticate today (static token, env var, hardcoded)? Where is the HTTP client that calls the product? Is there an existing M2M or `client_credentials` flow? Any `secretsmanager` imports? List every file:line where product API calls are made and where authentication credentials are injected.
>
> OUTPUT FORMAT: Structured report with file:line references for every point above.
> DO NOT write code. Analysis only.
**The explorer produces a gap summary.** For the full checklist of required items, see [multi-tenant.md § Checklist](../../docs/standards/golang/multi-tenant.md).
**This report becomes the CONTEXT for all subsequent gates.**
<block_condition>
HARD GATE: MUST complete the analysis report before proceeding. All subsequent gates use this report to know exactly what to change.
</block_condition>
MUST ensure backward compatibility context: the analysis MUST identify how the service works today in single-tenant mode, so subsequent gates preserve this behavior when `MULTI_TENANT_ENABLED=false`.
---
## Gate 1.5: Implementation Preview (Visual Report)
**Always executes. This gate generates a visual HTML report showing exactly what will change before any code is written.**
**Uses the `ring:visual-explainer` skill to produce a self-contained HTML page.**
The report is built from Gate 0 (stack detection) and Gate 1 (codebase analysis). It shows the developer a complete preview of every change that will be made across all subsequent gates, with backward compatibility analysis.
**Orchestrator generates the report using `ring:visual-explainer` with this content:**
The HTML page MUST include these sections:
### 1. Current Architecture (Before)
- Mermaid diagram showing current request flow (how connections work today in single-tenant mode)
- Table of all files that will be modified, with current line counts
- How repositories get DB connections today (static field, constructor injection, etc.)
### 2. Target Architecture (After)
- Mermaid diagram showing the multi-tenant request flow (JWT → middleware → tenant pool → handler)
- Which middleware will be used: `TenantMiddleware` (single-module) or `MultiPoolMiddleware` (multi-module)
- How repositories will get DB connections (context-based: `core.ResolvePostgres(ctx, fallback)`)
### 3. Change Map (per gate)
Table with columns: Gate, File, Current Code, New Code, Lines Changed. One row per file that will be modified. Example:
| Gate | File | What Changes | Impact |
|------|------|-------------|--------|
| 2 | `go.mod` | lib-commons v2 → v3 + lib-auth v2, import paths | All files |
| 3 | `config.go` | Add the 8 canonical MULTI_TENANT_* env vars (see "Canonical Environment Variables" table above) to Config struct | ~20 lines added |
| 4 | `config.go` | Add TenantMiddleware/MultiPoolMiddleware setup | ~30 lines added |
| 4 | `routes.go` | Register middleware in Fiber chain | ~5 lines added |
| 5 | `organization.postgresql.go` | `c.connection.GetDB()` → `core.ResolveModuleDB(ctx, module, r.connection)` | ~3 lines per method |
| 5 | `metadata.mongodb.go` | Static mongo → `core.ResolveMongo(ctx, r.connection, r.dbName)` | ~2 lines per method |
| 5 | `consumer.redis.go` | Key prefixing with `valkey.GetKeyFromContext(ctx, key)` | ~1 line per operation |
| 5 | `storage.go` | S3 key prefixing with `s3.GetObjectStorageKeyForTenant(ctx, key)` | ~1 line per operation |
| 5.5 | `m2m/provider.go` | New file: M2MCredentialProvider with credential caching (plugin only) | ~80 lines |
| 5.5 | `config.go` | Add M2M_TARGET_SERVICE, cache TTL, AWS_REGION env vars (plugin only) | ~10 lines added |
| 5.5 | `bootstrap.go` | Conditional M2M wiring when multi-tenant + plugin (plugin only) | ~20 lines added |
| 6 | `producer.rabbitmq.go` | Dual constructor (single-tenant + multi-tenant) | ~20 lines added |
| 6 | `rabbitmq.server.go` | MultiTenantConsumer setup with lazy mode | ~40 lines added |
| 7 | `config.go` | Backward compat validation | ~10 lines added |
**MANDATORY: Below the summary table, show per-file code diff panels for every file that will be modified.**
For each file in the change map, generate a before/after diff panel showing:
- **Before:** The exact current code from the codebase (sourced from the Gate 1 analysis)
- **After:** The exact code that will be written (following multi-tenant.md patterns)
- Use syntax highlighting and line numbers (read `default/skills/visual-explainer/templates/code-diff.html` for patterns)
Example diff panel for a repository file:
```go
// BEFORE: organization.postgresql.go
func (r *OrganizationPostgreSQLRepository) Create(ctx context.Context, org *Organization) error {
db := r.connection.GetDB()
result := db.Model(&OrganizationPostgreSQLModel{}).Create(toModel(org))
// ...
}
// AFTER: organization.postgresql.go
func (r *OrganizationPostgreSQLRepository) Create(ctx context.Context, org *Organization) error {
db, err := core.ResolveModuleDB(ctx, "organization", r.connection)
if err != nil {
return fmt.Errorf("getting tenant db for organization: %w", err)
}
result := db.Model(&OrganizationPostgreSQLModel{}).Create(toModel(org))
// ...
}
```
The developer MUST be able to see the exact code that will be implemented to approve it. High-level descriptions alone are not sufficient for approval.
**When many files have identical changes** (e.g., 10+ repository files all changing `r.connection.GetDB()` to `core.ResolvePostgres(ctx, r.connection)`): show one representative diff panel, then list the remaining files with "Same pattern applied to: [file list]."
### 4. Backward Compatibility Analysis
**MANDATORY: Show complete conditional initialization code, not just the if/else skeleton.**
For each component (PostgreSQL, MongoDB, Redis, RabbitMQ, S3), show:
1. **Current initialization code** (exact lines from codebase, sourced from Gate 1)
2. **New initialization code** with the `if cfg.MultiTenantEnabled` branch
3. **Explicit callout:** "When MULTI_TENANT_ENABLED=false, execution follows the ELSE branch which is IDENTICAL to current behavior"
Side-by-side comparison showing:
- **MULTI_TENANT_ENABLED=false (default):** Exact current behavior preserved. No JWT parsing, no Tenant Manager calls, no pool routing. Middleware calls `c.Next()` immediately.
- **MULTI_TENANT_ENABLED=true:** New behavior with tenant isolation.
Code diff showing the complete conditional initialization (not skeleton):
```go
if cfg.MultiTenantEnabled && cfg.MultiTenantURL != "" {
// Multi-tenant path (NEW)
tmClient := client.NewClient(cfg.MultiTenantURL, logger, clientOpts...)
pgManager := postgres.NewManager(tmClient, logger)
// ... show complete initialization
} else {
// Single-tenant path (UNCHANGED — exactly how it works today)
// Show the exact same constructor calls that exist in the current codebase
logger.Info("Running in SINGLE-TENANT MODE")
}
```
Show the middleware bypass explicitly:
```go
// When MULTI_TENANT_ENABLED=false, TenantMiddleware is NOT registered.
// Requests flow directly to handlers without any JWT parsing or tenant resolution.
```
The developer MUST understand that:
- No new code paths execute in single-tenant mode
- The `else` branch preserves the exact current constructor calls
- No additional dependencies are loaded when disabled
- No performance impact when disabled (middleware calls c.Next() immediately)
### 5. New Dependencies
Table showing what gets added to go.mod and which sub-packages are imported:
- `tenant-manager/core` — types, errors, context helpers
- `tenant-manager/client` — Tenant Manager HTTP client
- `tenant-manager/middleware` — TenantMiddleware or MultiPoolMiddleware
- `tenant-manager/postgres` — PostgresManager (if PG detected)
- `tenant-manager/mongo` — MongoManager (if Mongo detected)
- etc.
### 6. Environment Variables
The exact 8 canonical env vars from the "Canonical Environment Variables" table in [multi-tenant.md](../../docs/standards/golang/multi-tenant.md#environment-variables). MUST NOT use alternative names. MUST NOT duplicate the list inline — reference the canonical table.
### 7. Risk Assessment
Table with: Risk, Mitigation, Verification. Examples:
- Single-tenant regression → Backward compat gate (Gate 7) → `MULTI_TENANT_ENABLED=false go test ./...`
- Cross-tenant data leak → Context-based isolation → Tenant isolation integration tests (Gate 8)
- Startup performance → Lazy consumer mode → `consumer.Run(ctx)` returns in <1s
### 8. Retro Compatibility Guarantee
Explicit explanation of backward compatibility strategy:
**Method:** Feature flag with `MULTI_TENANT_ENABLED` environment variable (default: `false`).
**Guarantee:** When `MULTI_TENANT_ENABLED=false`:
- No tenant middleware is registered in the HTTP chain
- No JWT parsing or tenant resolution occurs
- All database connections use the original static constructors
- All Redis keys are unprefixed (original behavior)
- All S3 keys are unprefixed (original behavior)
- RabbitMQ connects directly at startup (original behavior)
- `go test ./...` passes with zero changes to existing tests
**Verification (Gate 7):** The agent MUST run `MULTI_TENANT_ENABLED=false go test ./...` and verify all existing tests pass unchanged.
**Output:** Save the HTML report to `docs/multi-tenant-preview.html` in the project root.
**Open in browser** for the developer to review.
<block_condition>
HARD GATE: Developer MUST explicitly approve the implementation preview before any code changes begin. This prevents wasted effort on misunderstood requirements or incorrect architectural decisions.
</block_condition>
**If the developer requests changes to the preview, regenerate the report and re-confirm.**
---
## Gate 2: lib-commons v4 + lib-auth v2 Upgrade
**SKIP only if:** `go.mod` contains `lib-commons/v4` AND `lib-auth/v2` (verified via grep, not assumed). If the service uses lib-commons v2 or v3, or lib-auth v1, this gate is MANDATORY.
**Dispatch `ring:backend-engineer-golang` with context:**
> TASK: Upgrade lib-commons to v4, then update lib-auth to v2 (lib-auth v2 depends on lib-commons v4).
> For both libraries, fetch the latest tag (v4 is currently in beta — use latest beta until stable is released).
> Check latest tags: `git ls-remote --tags https://github.com/LerianStudio/lib-commons.git | grep "v4" | sort -V | tail -1` and `git ls-remote --tags https://github.com/LerianStudio/lib-auth.git | tail -5`
> Run in order:
> 1. `go get github.com/LerianStudio/lib-commons/v4@{latest-v4-tag}`
> 2. `go get github.com/LerianStudio/lib-auth/v2@{latest-tag}`
> Update go.mod and all import paths to v4 for lib-commons (from v2 or v3).
> Follow multi-tenant.md section "Required lib-commons Version".
> DO NOT implement multi-tenant code yet — only upgrade the dependencies.
> Verify: go build ./... and go test ./... MUST pass.
**Verification:** `grep "lib-commons/v4" go.mod` + `grep "lib-auth/v2" go.mod` + `go build ./...` + `go test ./...`
<block_condition>
HARD GATE: MUST pass build and tests before proceeding.
</block_condition>
---
## Gate 3: Multi-Tenant Configuration
**Always executes.** If config already has `MULTI_TENANT_ENABLED`, this gate VERIFIES that all 8 canonical env vars are present with correct names, types, and defaults where applicable. Non-compliant config (wrong names like `TENANT_MANAGER_ADDRESS`, missing vars, wrong defaults) MUST be fixed. Compliance audit from Gate 0 determines whether this is implement or fix.
**Dispatch `ring:backend-engineer-golang` with context from Gate 1 analysis:**
> TASK: Verify and ensure all 8 canonical multi-tenant environment variables exist in the Config struct with correct names and defaults. If any are missing, misnamed, or have wrong defaults — fix them.
> CONTEXT FROM GATE 1: {Config struct location and current fields from analysis report}
> Follow multi-tenant.md sections "Environment Variables", "Configuration", and "Conditional Initialization".
>
> The EXACT env vars to add (no alternatives allowed):
> - MULTI_TENANT_ENABLED (bool, default false)
> - MULTI_TENANT_URL (string, required when enabled)
> - MULTI_TENANT_ENVIRONMENT (string, default "staging", only if RabbitMQ)
> - MULTI_TENANT_MAX_TENANT_POOLS (int, default 100)
> - MULTI_TENANT_IDLE_TIMEOUT_SEC (int, default 300)
> - MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD (int, default 5)
> - MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC (int, default 30)
> - MULTI_TENANT_SERVICE_API_KEY (string, required — API key for tenant-manager /settings endpoint)
>
> MUST NOT use alternative names (e.g., TENANT_MANAGER_ADDRESS, TENANT_MANAGER_URL are WRONG).
> Add conditional log: "Multi-tenant mode enabled" vs "Running in SINGLE-TENANT MODE".
> DO NOT implement TenantMiddleware yet — only configuration.
**Verification:** `grep "MULTI_TENANT_ENABLED" internal/bootstrap/config.go` + `grep "MULTI_TENANT_SERVICE_API_KEY" internal/bootstrap/config.go` + `go build ./...`
**HARD GATE: `.env.example` compliance.** If the project has a `.env.example` file, MUST verify it includes `MULTI_TENANT_SERVICE_API_KEY`. If missing, add it.
---
## Gate 4: TenantMiddleware (Core)
**Always executes.** If middleware already exists, this gate VERIFIES it uses the canonical lib-commons v4 tenant-manager sub-packages (`tmmiddleware.NewTenantMiddleware` or `tmmiddleware.NewMultiPoolMiddleware`). Custom middleware, inline JWT parsing, or any non-lib-commons implementation is NON-COMPLIANT and MUST be replaced. Compliance audit from Gate 0 determines whether this is implement or fix.
**This is the CORE gate. Without compliant TenantMiddleware, there is no tenant isolation.**
**Dispatch `ring:backend-engineer-golang` with context from Gate 1 analysis:**
> TASK: Implement tenant middleware using lib-commons/v4 tenant-manager sub-packages.
> DETECTED DATABASES: {postgresql: Y/N, mongodb: Y/N} (from Gate 0)
> SERVICE ARCHITECTURE: {single-module OR multi-module} (from Gate 1)
> CONTEXT FROM GATE 1: {Bootstrap location, middleware chain insertion point, service init from analysis report}
>
> **For single-module services:** Follow multi-tenant.md § "Generic TenantMiddleware (Standard Pattern)" for imports, constructor, and options.
> **For multi-module services:** Follow multi-tenant.md § "Multi-module middleware (MultiPoolMiddleware)" for WithRoute/WithDefaultRoute pattern.
> **For sub-package import aliases:** See multi-tenant.md § sub-package import table.
>
> Follow multi-tenant.md § "JWT Tenant Extraction" for tenantId claim handling.
> Follow multi-tenant.md § "Conditional Initialization" for the bootstrap pattern.
>
> MUST define constants for service name and module names — never pass raw strings.
> Create connection managers ONLY for detected databases.
> Public endpoints (/health, /version, /swagger) MUST bypass tenant middleware.
> When MULTI_TENANT_ENABLED=false, middleware calls c.Next() immediately (single-tenant passthrough).
>
> **Service API Key Authentication (MANDATORY):** The Tenant Manager HTTP client MUST be configured with `client.WithServiceAPIKey(cfg.MultiTenantServiceAPIKey)` so that `X-API-Key` header is sent in requests to the `/settings` endpoint. Follow multi-tenant.md § "Service Authentication (MANDATORY)".
>
> **IF RabbitMQ DETECTED:** Follow multi-tenant.md § "ConsumerTrigger interface" for the wiring pattern.
**Verification:** `grep "tmmiddleware.NewTenantMiddleware\|tmmiddleware.NewMultiPoolMiddleware" internal/bootstrap/` + `grep "WithServiceAPIKey" internal/bootstrap/` + `go build ./...`
<block_condition>
HARD GATE: CANNOT proceed without TenantMiddleware.
</block_condition>
---
## Gate 5: Repository Adaptation
**Always executes per detected DB/storage.** If repositories already use context-based connections, this gate VERIFIES they use the canonical lib-commons v4 functions (`core.ResolvePostgres`, `core.ResolveMongo`, `core.ResolveModuleDB`, `valkey.GetKeyFromContext`, `s3.GetObjectStorageKeyForTenant`). Custom pool lookups, manual DB switching, or any non-lib-commons resolution is NON-COMPLIANT and MUST be replaced. Compliance audit from Gate 0 determines whether this is implement or fix.
**Dispatch `ring:backend-engineer-golang` with context from Gate 1 analysis:**
> TASK: Adapt all repository implementations to get database connections from tenant context instead of static connections. Also adapt S3/object storage operations to prefix keys with tenant ID.
> DETECTED STACK: {postgresql: Y/N, mongodb: Y/N, redis: Y/N, s3: Y/N} (from Gate 0)
> CONTEXT FROM GATE 1: {List of ALL repository files and storage operations with file:line from analysis report}
>
> Follow multi-tenant.md sections:
> - "Database Connection in Repositories" (PostgreSQL)
> - "MongoDB Multi-Tenant Repository" (MongoDB)
> - "Redis Key Prefixing" and "Redis Key Prefixing for Lua Scripts" (Redis)
> - "S3/Object Storage Key Prefixing" (S3)
>
> MUST work in both modes: multi-tenant (prefixed keys / context connections) and single-tenant (unchanged keys / default connections).
**Verification:** grep for `core.ResolvePostgres` / `core.ResolveMongo` / `core.ResolveModuleDB` (multi-module) / `valkey.GetKeyFromContext` / `s3.GetObjectStorageKeyForTenant` in `internal/` + `go build ./...`
---
## Gate 5.5: M2M Secret Manager (Plugin Auth)
**SKIP IF:** service is NOT a plugin (i.e., it is a product or infrastructure service).
**This gate is MANDATORY for plugins** that need to authenticate with product APIs (e.g., ledger, midaz) in multi-tenant mode. Each tenant has its own M2M credentials stored in AWS Secrets Manager.
**Dispatch `ring:backend-engineer-golang` with context from Gate 0/1 analysis:**
> TASK: Implement M2M credential retrieval from AWS Secrets Manager with per-tenant caching.
> SERVICE TYPE: Plugin (confirmed in Gate 0)
> APPLICATION NAME: {ApplicationName constant from codebase, e.g., "plugin-pix"}
> TARGET SERVICE: {product the plugin calls, e.g., "ledger", "midaz"}
>
> Follow multi-tenant.md section "M2M Credentials via Secret Manager (Plugin-Only)" for all implementation patterns.
>
> **What to implement:**
>
> 1. **M2M Authenticator struct** with credential caching (`sync.Map`) and token caching.
> Use `secretsmanager.GetM2MCredentials()` from `github.com/LerianStudio/lib-commons/v4/commons/secretsmanager`.
> The function signature is:
> ```go
> secretsmanager.GetM2MCredentials(ctx, smClient, env, tenantOrgID, applicationName, targetService) (*M2MCredentials, error)
> ```
> It returns `*M2MCredentials{ClientID, ClientSecret}` fetched from path:
> `tenants/{env}/{tenantOrgID}/{applicationName}/m2m/{targetService}/credentials`
>
> 2. **Credential cache** with configurable TTL (default 300s). MUST NOT hit AWS on every request.
>
> 3. **Bootstrap wiring** — conditional on `cfg.MultiTenantEnabled`:
> ```go
> if cfg.MultiTenantEnabled {
> awsCfg, _ := awsconfig.LoadDefaultConfig(ctx)
> smClient := awssm.NewFromConfig(awsCfg)
> m2mProvider := m2m.NewM2MCredentialProvider(smClient, cfg.MultiTenantEnvironment,
> constant.ApplicationName, cfg.M2MTargetService,
> time.Duration(cfg.M2MCredentialCacheTTLSec)*time.Second)
> productClient = product.NewClient(cfg.ProductURL, m2mProvider)
> } else {
> productClient = product.NewClient(cfg.ProductURL, nil) // single-tenant: static auth
> }
> ```
>
> 4. **Config env vars** — add to Config struct:
> - `M2M_TARGET_SERVICE` (string, required for plugins)
> - `M2M_CREDENTIAL_CACHE_TTL_SEC` (int, default 300)
> - `AWS_REGION` (string, required for plugins)
>
> 6. **Error handling** using sentinel errors from lib-commons:
> - `secretsmanager.ErrM2MCredentialsNotFound` → tenant not provisioned
> - `secretsmanager.ErrM2MVaultAccessDenied` → IAM issue, alert ops
> - `secretsmanager.ErrM2MInvalidCredentials` → secret malformed, alert ops
>
> **SECURITY:**
> - MUST NOT log clientId or clientSecret values
> - MUST NOT store credentials in environment variables (fetch from Secrets Manager at runtime)
> - MUST cache locally to avoid per-request AWS API calls
> - MUST handle credential rotation via cache TTL expiry
**Verification:** `grep "secretsmanager.GetM2MCredentials\|M2MAuthenticator\|NewM2MAuthenticator" internal/` + `go build ./...`
<block_condition>
HARD GATE: If service is a plugin and MULTI_TENANT_ENABLED=true, M2M Secret Manager integration is NON-NEGOTIABLE. The plugin cannot authenticate with product APIs without it.
</block_condition>
---
## Gate 6: RabbitMQ Multi-Tenant
**SKIP IF:** no RabbitMQ detected.
MANDATORY: RabbitMQ multi-tenant requires **TWO complementary layers** — both are required. See [multi-tenant.md § RabbitMQ Multi-Tenant: Two-Layer Isolation Model](../../docs/standards/golang/multi-tenant.md#rabbitmq-multi-tenant-two-layer-isolation-model) for the canonical reference.
**Summary:**
- **Layer 1 (Isolation):** `tmrabbitmq.Manager` → `GetChannel(ctx, tenantID)` for per-tenant vhosts
- **Layer 2 (Audit):** `X-Tenant-ID` AMQP header for tracing and context propagation
**⛔ CRITICAL DISTINCTION:**
- FORBIDDEN: Using `X-Tenant-ID` header as an isolation mechanism — it is metadata for audit/tracing only
- REQUIRED: `tmrabbitmq.Manager` with per-tenant vhosts as the only acceptable isolation mechanism
- FORBIDDEN: A service that only propagates `X-Tenant-ID` headers on a shared connection — this is not multi-tenant compliant
**Dispatch `ring:backend-engineer-golang` with context from Gate 1 analysis:**
> TASK: Implement RabbitMQ multi-tenant with TWO mandatory layers:
>
> **Layer 1 — Vhost Isolation (MANDATORY):**
> - MUST use `tmrabbitmq.Manager` for per-tenant vhost connections with LRU eviction
> - MUST call `tmrabbitmq.Manager.GetChannel(ctx, tenantID)` for tenant-specific channel (Producer)
> - MUST use `tmconsumer.MultiTenantConsumer` with lazy initialization — no startup connections (Consumer)
> - MUST branch on `cfg.MultiTenantEnabled` in bootstrap (CONFIG SPLIT with dual constructors)
> - MUST keep existing single-tenant code path untouched
>
> **Layer 2 — X-Tenant-ID Header (MANDATORY):**
> - MUST inject `headers["X-Tenant-ID"] = tenantID` in all published messages (Producer)
> - MUST extract `X-Tenant-ID` from AMQP headers for log correlation and tracing (Consumer)
> - Header is audit trail ONLY — isolation comes from Layer 1
>
> CONTEXT FROM GATE 1: {Producer and consumer file:line locations from analysis report}
> DETECTED ARCHITECTURE: {Are producer and consumer in the same process or separate components?}
>
> Follow multi-tenant.md sections:
> - "RabbitMQ Multi-Tenant Producer" for dual constructor pattern with both layers
> - "Multi-Tenant Message Queue Consumers (Lazy Mode)" for lazy initialization
> - "ConsumerTrigger Interface" for the trigger wiring
>
> Gate-specific constraints:
> 1. MANDATORY: CONFIG SPLIT — branch on `cfg.MultiTenantEnabled` for both producer and consumer in bootstrap
> 2. MUST keep the existing single-tenant code path untouched
> 3. MUST NOT connect directly to RabbitMQ at startup in multi-tenant mode
> 4. MUST use X-Tenant-ID in AMQP headers for audit — NOT as isolation mechanism
> 5. MUST implement both layers together — one without the other is non-compliant
**Verification:**
1. `grep "tmrabbitmq.Manager\|NewProducerMultiTenant\|EnsureConsumerStarted\|tmmiddleware.ConsumerTrigger" internal/` + `go build ./...`
2. **Vhost isolation (Layer 1):** `grep -rn "tmrabbitmq.NewManager\|tmrabbitmq.Manager" internal/` MUST return results.
3. **X-Tenant-ID header (Layer 2):** `grep -rn "X-Tenant-ID" internal/` MUST return results in both producer AND consumer.
4. **Shared connection rejection:** If RabbitMQ multi-tenant uses a shared connection with only `X-Tenant-ID` headers (no `tmrabbitmq.Manager`), this gate FAILS.
<block_condition>
HARD GATE: RabbitMQ multi-tenant requires BOTH layers:
1. `tmrabbitmq.Manager` for per-tenant vhost isolation (Layer 1 — ISOLATION)
2. `X-Tenant-ID` AMQP header for audit trail and context propagation (Layer 2 — OBSERVABILITY)
Layer 2 alone (shared connection + X-Tenant-ID header) is NOT multi-tenant compliant — it provides traceability but ZERO isolation between tenants.
Layer 1 alone (vhosts without header) provides isolation but loses audit trail and cross-service context propagation.
Both layers MUST be implemented together. MUST NOT connect directly to RabbitMQ at startup in multi-tenant mode.
</block_condition>
#### RabbitMQ Multi-Tenant Anti-Rationalization
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "X-Tenant-ID header is enough for isolation" | Headers are metadata for audit/tracing, NOT isolation. All tenants share the same queues and vhost. A consumer bug or poison message affects ALL tenants. | **MUST implement Layer 1: `tmrabbitmq.Manager` with per-tenant vhosts** |
| "Vhosts are enough, we don't need the header" | Vhosts isolate but don't propagate tenant context for logging, tracing, and downstream DB resolution. Header is required for observability. | **MUST implement Layer 2: `X-Tenant-ID` header in all messages** |
| "Shared connection is simpler" | Simplicity ≠ isolation. One tenant's traffic spike blocks all others. No per-tenant rate limiting or queue policies possible. | **MUST use per-tenant vhosts via `tmrabbitmq.Manager`** |
| "We'll migrate to vhosts later" | Later = never. This is a HARD GATE. | **MUST implement NOW** |
| "Our service has low RabbitMQ traffic" | Traffic volume ≠ exemption. Isolation is a platform requirement. | **MUST use `tmrabbitmq.Manager` + `X-Tenant-ID` header** |
---
## Gate 7: Metrics & Backward Compatibility
**Dispatch `ring:backend-engineer-golang` with context:**
> TASK: Add multi-tenant metrics and validate backward compatibility.
>
> Follow multi-tenant.md sections "Multi-Tenant Metrics" and "Single-Tenant Backward Compatibility Validation (MANDATORY)".
>
> The EXACT metrics to implement (no alternatives allowed):
> - `tenant_connections_total` (Counter) — Total tenant connections created
> - `tenant_connection_errors_total` (Counter) — Connection failures per tenant
> - `tenant_consumers_active` (Gauge) — Active message consumers
> - `tenant_messages_processed_total` (Counter) — Messages processed per tenant
>
> All 4 metrics are MANDATORY. When MULTI_TENANT_ENABLED=false, metrics MUST use no-op implementations (zero overhead).
>
> BACKWARD COMPATIBILITY IS NON-NEGOTIABLE:
> - MUST start without any MULTI_TENANT_* env vars
> - MUST start without Tenant Manager running
> - MUST pass all existing tests with MULTI_TENANT_ENABLED=false
> - Health/version endpoints MUST work without tenant context
>
> Write TestMultiTenant_BackwardCompatibility integration test.
**Verification:** `MULTI_TENANT_ENABLED=false go test ./...` MUST pass.
<block_condition>
HARD GATE: Backward compatibility MUST pass.
</block_condition>
---
## Gate 8: Tests
**Dispatch `ring:backend-engineer-golang` with context:**
> TASK: Write multi-tenant tests.
> DETECTED STACK: {postgresql: Y/N, mongodb: Y/N, redis: Y/N, s3: Y/N, rabbitmq: Y/N} (from Gate 0)
>
> Follow multi-tenant.md section "Testing Multi-Tenant Code" (all subsections).
>
> Required tests: unit tests with mock tenant context, tenant isolation tests (two tenants, data separation), error case tests (missing JWT, tenant not found), plus RabbitMQ, Redis, and S3 tests if detected.
**Verification:** `go test ./... -v -count=1` + `go test ./... -cover`
---
## Gate 9: Code Review
**Dispatch 7 parallel reviewers (same pattern as ring:requesting-code-review).**
MUST include this context in ALL 7 reviewer dispatches:
> **MULTI-TENANT REVIEW CONTEXT:**
> - Multi-tenant isolation is based on `tenantId` from JWT → tenant-manager middleware (TenantMiddleware or MultiPoolMiddleware) → database-per-tenant.
> - `organization_id` is NOT a tenant identifier. It is a business filter within the tenant's database. A single tenant can have multiple organizations. Do NOT flag organization_id as a multi-tenant issue.
> - Backward compatibility is required: when `MULTI_TENANT_ENABLED=false`, the service MUST work exactly as before (single-tenant mode, no tenant context needed).
| Reviewer | Focus |
|----------|-------|
| ring:code-reviewer | Architecture, lib-commons v4 usage, TenantMiddleware/MultiPoolMiddleware placement, sub-package usage |
| ring:business-logic-reviewer | Tenant context propagation via tenantId (NOT organization_id) |
| ring:security-reviewer | Cross-tenant DB isolation, JWT tenantId validation, no data leaks between tenant databases |
| ring:test-reviewer | Coverage, isolation tests between two tenants, backward compat tests |
| ring:nil-safety-reviewer | Nil risks in tenant context extraction from JWT and context getters |
| ring:consequences-reviewer | Impact on single-tenant paths, backward compat when MULTI_TENANT_ENABLED=false |
| ring:dead-code-reviewer | Orphaned code from tenant changes, dead tenant-specific helpers |
MUST pass all 7 reviewers. Critical findings → fix and re-review.
---
## Gate 10: User Validation
MUST approve: present checklist for explicit user approval.
```markdown
## Multi-Tenant Implementation Complete
- [ ] lib-commons v4
- [ ] MULTI_TENANT_ENABLED config
- [ ] Tenant middleware (TenantMiddleware or MultiPoolMiddleware for multi-module services)
- [ ] Repositories use context-based connections
- [ ] S3 keys prefixed with tenantId (if applicable)
- [ ] RabbitMQ X-Tenant-ID (if applicable)
- [ ] M2M Secret Manager with credential + token caching (if plugin)
- [ ] Backward compat (MULTI_TENANT_ENABLED=false works)
- [ ] Tests pass
- [ ] Code review passed
```
---
## Gate 11: Activation Guide
**MUST generate `docs/multi-tenant-guide.md` in the project root.** Direct, concise, no filler text.
The file is built from Gate 0 (stack) and Gate 1 (analysis). See [multi-tenant.md § Checklist](../../docs/standards/golang/multi-tenant.md) for the canonical env var list and requirements.
<!-- Template: values filled from Gate 0/1 results. Canonical source: multi-tenant.md -->
The guide MUST include:
1. **Components table**: Component name, Service const, Module const, Resources, what was adapted
2. **Environment variables**: the 8 canonical MULTI_TENANT_* vars (MULTI_TENANT_ENABLED, MULTI_TENANT_URL, MULTI_TENANT_ENVIRONMENT, MULTI_TENANT_MAX_TENANT_POOLS, MULTI_TENANT_IDLE_TIMEOUT_SEC, MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD, MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC, MULTI_TENANT_SERVICE_API_KEY) with required/default/description
3. **M2M environment variables (plugin only)**: If the service is a plugin, include M2M_TARGET_SERVICE, M2M_CREDENTIAL_CACHE_TTL_SEC, AWS_REGION
4. **How to activate**: set envs + start alongside Tenant Manager (+ AWS credentials for plugins)
5. **How to verify**: check logs, test with JWT tenantId (+ verify M2M credential retrieval for plugins)
6. **How to deactivate**: set MULTI_TENANT_ENABLED=false
7. **Common errors**: see [multi-tenant.md § Error Handling](../../docs/standards/golang/multi-tenant.md)
---
## State Persistence
Save to `docs/ring-dev-multi-tenant/current-cycle.json` for resume support:
```json
{
"cycle": "multi-tenant",
"service_type": "plugin",
"stack": {"postgresql": false, "mongodb": true, "redis": true, "rabbitmq": true, "s3": true},
"gates": {"0": "PASS", "1": "PASS", "1.5": "PASS", "2": "IN_PROGRESS", "3": "PENDING", "5.5": "PENDING"},
"current_gate": 2
}
```
---
## Anti-Rationalization Table
See [multi-tenant.md](../../docs/standards/golang/multi-tenant.md) for the canonical anti-rationalization tables on tenantId vs organization_id.
**Skill-specific rationalizations:**
| Rationalization | Why It's WRONG | Required Action |
|-----------------|----------------|-----------------|
| "Service already has multi-tenant code" | Existence ≠ compliance. Code that doesn't follow the Ring canonical model is WRONG and must be replaced. | **STOP. Run compliance audit (Gate 0 Phase 2). Fix every NON-COMPLIANT component.** |
| "Multi-tenant is already implemented, just needs tweaks" | Partial or non-standard implementation is not "almost done" — it is non-compliant. Every component must match the canonical model exactly. | **STOP. Execute every gate. Verify or fix each one.** |
| "Skipping this gate because something similar exists" | "Similar" is not "compliant". Only exact matches to lib-commons v4 tenant-manager sub-packages are valid. | **STOP. Verify with grep evidence. If it doesn't match the canonical pattern → gate MUST execute.** |
| "The current approach works fine, no need to change" | Working ≠ compliant. A custom solution that works today creates drift, blocks upgrades, and prevents standardized tooling. | **STOP. Replace with canonical implementation.** |
| "We have a custom tenant package that handles this" | Custom packages are non-canonical. Only lib-commons v4 tenant-manager sub-packages are valid. Custom files MUST be removed. | **STOP. Remove custom files. Use lib-commons v4 sub-packages.** |
| "This extra file just wraps lib-commons" | Wrappers add indirection that breaks compliance verification and creates maintenance burden. MUST use lib-commons directly. | **STOP. Remove wrapper. Call lib-commons directly from bootstrap/adapters.** |
| "Agent says out of scope" | Skill defines scope, not agent. | **Re-dispatch with gate context** |
| "Skip tests" | Gate 8 proves isolation works. | **MANDATORY** |
| "Skip review" | Security implications. One mistake = data leak. | **MANDATORY** |
| "Using TENANT_MANAGER_ADDRESS instead" | Non-standard name. Only the 8 canonical MULTI_TENANT_* vars are valid. | **STOP. Use MULTI_TENANT_URL** |
| "The service already uses a different env name" | Legacy names are non-compliant. Rename to canonical names. | **Replace with canonical env vars** |
| "Plugin doesn't need Secret Manager for M2M" | If multi-tenant is active, each tenant has different credentials. Env vars can't hold per-tenant secrets. | **MUST use Secret Manager for per-tenant M2M** |
| "We'll add M2M caching later" | Without caching, every request hits AWS (~50-100ms + cost). This is a production blocker. | **MUST implement caching from day one** |
| "Hardcoded credentials work for now" | Hardcoded creds don't scale across tenants and are a security risk. | **MUST fetch from Secrets Manager per tenant** |
```