reality-verification

Original source / Raw SKILL.md

---
name: reality-verification
description: This skill should be used when the user asks to "verify a fix", "reproduce failure", "diagnose issue", "check BEFORE/AFTER state", "VF task", "reality check", "check test quality", "mock-only tests", or needs guidance on verifying fixes by reproducing failures before and after implementation, or detecting mock-heavy test anti-patterns.
version: 0.2.0
user-invocable: false
---

# Reality Verification

For fix goals: reproduce the failure BEFORE work, verify resolution AFTER.

## Goal Detection

Classify user goals to determine if diagnosis is needed. See `references/goal-detection-patterns.md` for detailed patterns.

**Quick reference:**
- Fix indicators: fix, repair, resolve, debug, patch, broken, failing, error, bug
- Add indicators: add, create, build, implement, new
- Conflict resolution: If both present, treat as Fix

## Command Mapping

| Goal Keywords | Reproduction Command |
|---------------|---------------------|
| CI, pipeline | `gh run view --log-failed` |
| test, tests | project test command |
| type, typescript | `pnpm check-types` or `tsc --noEmit` |
| lint | `pnpm lint` |
| build | `pnpm build` |
| E2E, UI | Playwright MCP browser tools |
| API, endpoint | WebFetch tool |

For E2E/deployment verification, use MCP tools (Playwright MCP browser tools for UI, WebFetch tool for APIs).

## BEFORE/AFTER Documentation

### BEFORE State (Diagnosis)

Document in `.progress.md` under `## Reality Check (BEFORE)`:

```markdown
## Reality Check (BEFORE)

**Goal type**: Fix
**Reproduction command**: `pnpm test`
**Failure observed**: Yes
**Output**:
```
FAIL src/auth.test.ts
  Expected: 200
  Received: 401
```
**Timestamp**: 2026-01-16T10:30:00Z
```

### AFTER State (Verification)

Document in `.progress.md` under `## Reality Check (AFTER)`:

```markdown
## Reality Check (AFTER)

**Command**: `pnpm test`
**Result**: PASS
**Output**:
```
PASS src/auth.test.ts
All tests passed
```
**Comparison**: BEFORE failed with 401, AFTER passes
**Verified**: Issue resolved
```

## VF Task Format

Add as task 4.3 (after PR creation) for fix-type specs:

```markdown
- [ ] 4.3 VF: Verify original issue resolved
  - **Do**:
    1. Read BEFORE state from .progress.md
    2. Re-run reproduction command: `<command>`
    3. Compare output with BEFORE state
    4. Document AFTER state in .progress.md
  - **Verify**: `grep -q "Verified: Issue resolved" ./specs/<name>/.progress.md`
  - **Done when**: AFTER shows issue resolved, documented in .progress.md
  - **Commit**: `chore(<name>): verify fix resolves original issue`
```

## Test Quality Checks

When verifying test-related fixes, check for mock-only test anti-patterns. See `references/mock-quality-checks.md` for detailed patterns.

**Quick reference red flags:**
- Mock declarations > 3x real assertions
- Missing import of actual module under test
- All assertions are mock interaction checks (toHaveBeenCalled)
- No integration tests
- Missing mock cleanup (afterEach)

## Why This Matters

| Without | With |
|---------|------|
| "Fix CI" spec completes but CI still red | CI verified green before merge |
| Tests "fixed" but original failure unknown | Before/after comparison proves fix |
| Silent regressions | Explicit failure reproduction |
| Manual verification required | Automated verification in workflow |
| Tests pass but only test mocks | Tests verify real behavior, not mock behavior |
| False sense of security from green tests | Confidence that tests catch real bugs |


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### references/goal-detection-patterns.md

```markdown
# Goal Detection Patterns

Regex patterns and heuristics for classifying user goals as Fix or Add type.

## Detection Heuristics

| Pattern | Type | Regex |
|---------|------|-------|
| fix, repair, resolve, debug, patch | Fix | `\b(fix\|repair\|resolve\|debug\|patch)\b` |
| broken, failing, error, bug, issue | Fix | `\b(broken\|failing\|error\|bug\|issue)\b` |
| "not working", "doesn't work" | Fix | `not\s+working\|doesn't\s+work` |
| crash, crashing | Fix | `\b(crash\|crashing)\b` |
| add, create, build, implement, new | Add | `\b(add\|create\|build\|implement\|new)\b` |
| enable, introduce, support | Add | `\b(enable\|introduce\|support)\b` |

## Classification Algorithm

```text
1. Scan goal text for Fix indicators (higher priority)
2. If Fix indicators found -> classify as Fix
3. Else scan for Add indicators
4. If Add indicators found -> classify as Add
5. If both present -> classify as Fix (fixing enables the feature)
6. If neither -> default to Add
```

## Examples

| Goal | Classification | Reason |
|------|---------------|--------|
| "Fix login bug" | Fix | Contains "fix" and "bug" |
| "Add dark mode" | Add | Contains "add" |
| "Create auth endpoint" | Add | Contains "create" |
| "Resolve failing tests" | Fix | Contains "resolve" and "failing" |
| "Build new API and fix errors" | Fix | Both present, Fix takes priority |
| "Implement user registration" | Add | Contains "implement" |
| "Debug performance issue" | Fix | Contains "debug" and "issue" |

## Command Mapping

Map goal keywords to reproduction commands:

| Goal Keywords | Reproduction Command |
|---------------|---------------------|
| CI, pipeline, actions | `gh run view --log-failed` |
| test, tests, spec | project test command (package.json scripts.test) |
| type, types, typescript | `pnpm check-types` or `tsc --noEmit` |
| lint, linting | `pnpm lint` or `eslint .` |
| build, compile | `pnpm build` or `npm run build` |
| deploy, deployment | `gh api` or MCP fetch to check status |
| E2E, UI, browser, visual | MCP playwright to screenshot or run E2E suite |
| endpoint, API, response | MCP fetch with expected status/response validation |
| site, page, live | MCP fetch/playwright to verify live behavior |

## Fallback

If no keyword match:
1. Ask user for reproduction steps
2. Or skip diagnosis if goal is clearly Add type

```

### references/mock-quality-checks.md

```markdown
# Mock Quality Checks

Detect mock-only test anti-patterns where tests pass but don't actually test implementation.

## Red Flags

### 1. Mockery Pattern

Test file has more mock setup than real assertions.

**Detection:**
```bash
grep -c "mock\|stub\|spy" test-file
grep -c "expect(" test-file
```

**Red flag:** Mock lines > 3x assertion lines

### 2. Missing Real Imports

Test only imports test libraries, not actual module.

**Detection:** Check if test imports the real implementation:
```javascript
// Good: imports real module
import { authenticate } from '../auth'

// Bad: only imports test libraries
import { describe, it, expect, vi } from 'vitest'
```

**Red flag:** Only imports from jest/vitest/testing-library

### 3. Behavioral-Only Testing

All assertions are mock interactions, no state/value checks.

**Detection:**
```bash
grep "toHaveBeenCalled\|spy.calledWith" test-file
```

**Red flag:** No `toBe`, `toEqual`, `toMatch` assertions on real values

### 4. No Integration Coverage

Every dependency is mocked, no real integration tested.

**Detection:** Check if any tests run without mocks

**Red flag:** 100% mock coverage = 0% real integration tested

### 5. Partial Mocking Issues

Excessive use of `vi.spyOn`/`jest.spyOn`.

**Red flag:** Mixing real and mocked behavior creates unpredictability

### 6. No Mock Cleanup

Mocks persist across tests.

**Detection:**
```bash
grep "afterEach\|mockClear\|mockReset" test-file
```

**Red flag:** Missing cleanup = flaky tests

## Quality Check Report Format

### When Issues Detected

```text
Mock Quality Issues Detected

File: src/auth.test.ts
- Mock declarations: 15
- Real assertions: 3
- Mock ratio: 5.0x (threshold: 3x)
- Real module import: MISSING
- Integration tests: 0

Issues:
1. Missing import of actual auth module
2. All assertions verify mock interactions, none check real behavior
3. No integration test coverage

Suggested fixes:
- Import actual auth module: import { authenticate } from '../auth'
- Add state-based assertions: expect(result).toEqual({...})
- Create integration test with real dependencies
- Reduce mocking to only external services (network, DB)

Status: VERIFICATION_FAIL (test quality issues)
```

### When Tests Are Healthy

```text
Mock Quality Check: PASS

File: src/auth.test.ts
- Mock declarations: 2 (external services only)
- Real assertions: 12
- Real module import: YES
- Integration tests: 3
- Mock cleanup: afterEach present

Tests verify real behavior, not mock behavior.
```

## Reality Check for Test Fixes

For "fix tests" specs, verify BOTH conditions:
1. Tests pass
2. Tests actually test real behavior (not just mocks)

### Before State

```markdown
## Reality Check (BEFORE)

**Tests**: FAILING
**Mock ratio**: N/A (tests failing)
```

### After State (Good)

```markdown
## Reality Check (AFTER)

**Tests**: PASSING
**Mock quality check**:
- Mock declarations: 2
- Real assertions: 15
- Real module import: YES
- Integration tests: 3
- Mock cleanup: afterEach present

**Verdict**: Tests pass AND test real behavior
```

### After State (Bad)

```markdown
## Reality Check (AFTER)

**Tests**: PASSING (warning)
**Mock quality issues**:
- Mock declarations: 20
- Real assertions: 4
- Mock ratio: 5.0x (exceeds 3x threshold)
- Real module import: MISSING
- Integration tests: 0

**Verdict**: Tests pass but only test mocks, not implementation
**Required**: Fix test quality before marking VERIFICATION_PASS
```

```