validating-ai-ethics-and-fairness

Original source / Raw SKILL.md

---
name: validating-ai-ethics-and-fairness
version: 1.0.0
description: |
  Validate AI/ML models and datasets for bias, fairness, and ethical concerns.
  Use when auditing AI systems for ethical compliance, fairness assessment, or bias detection.
  Trigger with phrases like "evaluate model fairness", "check for bias", or "validate AI ethics".
allowed-tools: Read, Write, Edit, Grep, Glob, Bash(python:*)
license: MIT
---

## Prerequisites

Before using this skill, ensure you have:
- Access to the AI model or dataset requiring validation
- Model predictions or training data available for analysis
- Understanding of demographic attributes relevant to fairness evaluation
- Python environment with fairness assessment libraries (e.g., Fairlearn, AIF360)
- Appropriate permissions to analyze sensitive data attributes

## Instructions

### Step 1: Identify Validation Scope
Determine which aspects of the AI system require ethical validation:
- Model predictions across demographic groups
- Training dataset representation and balance
- Feature selection and potential proxy variables
- Output disparities and fairness metrics

### Step 2: Analyze for Bias
Use the skill to examine the AI system:
1. Load model predictions or dataset using Read tool
2. Identify sensitive attributes (age, gender, race, etc.)
3. Calculate fairness metrics (demographic parity, equalized odds, etc.)
4. Detect statistical disparities across groups

### Step 3: Generate Validation Report
The skill produces a comprehensive report including:
- Identified biases and their severity
- Fairness metric calculations with thresholds
- Representation analysis across demographic groups
- Recommended mitigation strategies
- Compliance assessment against ethical guidelines

### Step 4: Implement Mitigations
Based on findings, apply recommended strategies:
- Rebalance training data using sampling techniques
- Apply algorithmic fairness constraints during training
- Adjust decision thresholds for specific groups
- Document ethical considerations and trade-offs

## Output

The skill generates structured reports containing:

### Bias Detection Results
- Statistical disparities identified across groups
- Severity classification (low, medium, high, critical)
- Affected demographic segments with quantified impact

### Fairness Metrics
- Demographic parity ratios
- Equal opportunity differences
- Predictive parity measurements
- Calibration scores across groups

### Mitigation Recommendations
- Specific technical approaches to reduce bias
- Data augmentation or resampling strategies
- Model constraint adjustments
- Monitoring and continuous evaluation plans

### Compliance Assessment
- Alignment with ethical AI guidelines
- Regulatory compliance status
- Documentation requirements for audit trails

## Error Handling

Common issues and solutions:

**Insufficient Data**
- Error: Cannot calculate fairness metrics with small sample sizes
- Solution: Aggregate related groups or collect additional data for underrepresented segments

**Missing Sensitive Attributes**
- Error: Demographic information not available in dataset
- Solution: Use proxy detection methods or request access to protected attributes under appropriate governance

**Conflicting Fairness Criteria**
- Error: Multiple fairness metrics show contradictory results
- Solution: Document trade-offs and prioritize metrics based on use case context and stakeholder input

**Data Quality Issues**
- Error: Inconsistent or corrupted attribute values
- Solution: Perform data cleaning, standardization, and validation before bias analysis

## Resources

### Fairness Assessment Frameworks
- Fairlearn library for bias detection and mitigation
- AI Fairness 360 (AIF360) toolkit for comprehensive fairness analysis
- Google What-If Tool for interactive fairness exploration

### Ethical AI Guidelines
- IEEE Ethically Aligned Design principles
- EU Ethics Guidelines for Trustworthy AI
- ACM Code of Ethics for AI practitioners

### Fairness Metrics Documentation
- Demographic parity and statistical parity definitions
- Equalized odds and equal opportunity metrics
- Individual fairness and calibration measures

### Best Practices
- Involve diverse stakeholders in fairness criteria selection
- Document all ethical decisions and trade-offs
- Implement continuous monitoring for fairness drift
- Maintain transparency in model limitations and biases