pyportfolioopt

Original source / Raw SKILL.md

---
name: Portfolio Optimization with PyPortfolioOpt
description: A comprehensive guide to portfolio optimization using PyPortfolioOpt, covering expected returns calculation, risk models, and optimization techniques.Use when users ask to optimize their portfolio or need advice on asset allocation.
---

# Portfolio Optimization with PyPortfolioOpt

## Overview

PyPortfolioOpt is a financial portfolio optimization library that implements classical methods (Efficient Frontier, Black-Litterman) and modern techniques (shrinkage methods, Hierarchical Risk Parity). This skill provides a structured workflow for portfolio optimization in Python.

## Core Components

### 1. Expected Returns Calculation

Calculate expected returns from historical price data using simple or logarithmic returns:

```python
from pypfopt.expected_returns import returns_from_prices, mean_historical_return

# Raw returns calculation
log_returns = False
returns = returns_from_prices(prices_df, log_returns=log_returns)

# Expected returns (annualized)
mu = mean_historical_return(prices_df, log_returns=log_returns)
```

**Key Parameters:**

- `log_returns=False`: Uses simple percentage returns (default, suitable for most cases)
- `log_returns=True`: Uses logarithmic returns (more robust for volatile assets, longer time horizons)

### 2. Risk Models: Covariance Matrix Estimation

Estimate the covariance matrix of asset returns. PyPortfolioOpt provides multiple methods for different scenarios:

#### Sample Covariance (Standard Method)

Basic empirical covariance from historical returns. Simple but can be noisy with limited data:

```python
from pypfopt.risk_models import sample_cov

S = sample_cov(prices_df)
```

**Best for:** Large datasets, stable market conditions

---

#### Ledoit-Wolf Shrinkage

Shrinks sample covariance toward a structured target (identity matrix). Reduces estimation error while preserving correlation structure:

```python
from pypfopt.risk_models import CovarianceShrinkage

S = CovarianceShrinkage(prices_df).ledoit_wolf()
```

**Best for:** Limited data, high-dimensional portfolios, robust out-of-sample performance

---

#### Single-Factor Model (Market Model)

Uses a single market factor to estimate covariance. Assumes returns driven by market + idiosyncratic factors:

```python
from pypfopt.risk_models import CovarianceShrinkage

S = CovarianceShrinkage(prices_df).single_factor_model()
```

**Best for:** Simplifying correlation structure, reducing estimation noise, large universes

**Note:** Requires market index data or uses prices_df average as proxy

---

#### Denoised Covariance (Random Matrix Theory)

Filters out noise from sample covariance using eigenvalue decomposition. Removes small eigenvalues (noise) while preserving signal:

```python
from pypfopt.risk_models import CovarianceShrinkage

S = CovarianceShrinkage(prices_df).denoised_covariance()
```

**Best for:** High-frequency data, noisy markets, improving Sharpe ratio

---

#### Custom Shrinkage Target

Combine shrinkage with custom target matrix (e.g., correlation matrix, structured covariance):

```python
from pypfopt.risk_models import CovarianceShrinkage

# Shrink toward identity matrix with custom intensity
shrinkage = CovarianceShrinkage(prices_df)
shrinkage.shrinkage_target = np.eye(len(prices_df.columns))
S = shrinkage.ledoit_wolf()
```

---

#### Matrix Validation & Fixing

Ensure covariance matrix is positive semidefinite (required for optimization):

```python
from pypfopt.risk_models import fix_nonpositive_semidefinite
import numpy as np

# Check and fix if needed
S_fixed = fix_nonpositive_semidefinite(S)

# PyPortfolioOpt automatically fixes in most cases during optimization
```

## Risk Model Selection Guide

Choose the appropriate covariance model based on your data and constraints:

| Method                  | Use Case                         | Pros                  | Cons                                    |
| ----------------------- | -------------------------------- | --------------------- | --------------------------------------- |
| **sample_cov**          | Baseline, many assets (N > 100+) | Unbiased, simple      | High estimation error, requires large N |
| **ledoit_wolf**         | Limited data, high correlation   | Robust, reduces noise | Assumes specific shrinkage target       |
| **single_factor_model** | Large universes (N > 500)        | Parsimonious, stable  | Assumes factor dominance                |
| **denoised_covariance** | Noisy/high-frequency data        | Better Sharpe ratios  | Computationally intensive               |

### Decision Tree for Risk Model Selection

```
Do you have extensive historical data (3+ years, daily)?
├─ YES: Is your portfolio high-dimensional (50+ assets)?
│   ├─ YES: Use single_factor_model() for stability
│   └─ NO: Use sample_cov() or ledoit_wolf()
└─ NO: Use ledoit_wolf() or denoised_covariance()

Is data very noisy or high-frequency?
├─ YES: Try denoised_covariance()
└─ NO: Use ledoit_wolf() as default robust choice

Do you want maximum stability?
└─ Use single_factor_model()
```

### Comparing Risk Models in Code

```python
from pypfopt.risk_models import sample_cov, CovarianceShrinkage
from pypfopt.efficient_frontier import EfficientFrontier

# Calculate all methods
S_sample = sample_cov(prices_df)
S_ledoit = CovarianceShrinkage(prices_df).ledoit_wolf()
S_factor = CovarianceShrinkage(prices_df).single_factor_model()
S_denoised = CovarianceShrinkage(prices_df).denoised_covariance()

models = {
    'sample': S_sample,
    'ledoit_wolf': S_ledoit,
    'factor_model': S_factor,
    'denoised': S_denoised
}

results = {}
for name, S in models.items():
    ef = EfficientFrontier(mu, S)
    ef.max_sharpe()
    ret, vol, sharpe = ef.portfolio_performance()
    results[name] = {
        'return': ret,
        'volatility': vol,
        'sharpe': sharpe,
        'weights': ef.clean_weights()
    }

# Compare Sharpe ratios
for name, metrics in results.items():
    print(f"{name:15} Sharpe: {metrics['sharpe']:.4f}")
```

## Advanced Optimization Strategies

### Hierarchical Risk Parity (HRP)

Modern algorithm that doesn't require inverting the covariance matrix. Constructs diversified portfolios by hierarchical clustering:

```python
from pypfopt.hierarchical_portfolio import HRPOpt

hrp = HRPOpt(prices_df)
weights_hrp = hrp.optimize()
ret_hrp, vol_hrp, sharpe_hrp = hrp.portfolio_performance()
```

**Advantages:**

- No covariance matrix inversion (stable, numerically robust)
- Better out-of-sample performance
- More diversified allocations
- Robust to estimation errors

**Use Case:** When covariance estimates are unreliable or you have many correlated assets

---

### Black-Litterman Model

Incorporates your views about future returns with market equilibrium. Starts from market-implied returns, adjusts with expert views:

```python
from pypfopt.black_litterman import BlackLittermanModel
from pypfopt.efficient_frontier import EfficientFrontier

# Define absolute views: {ticker: expected_return}
views = {
    'AAPL': 0.15,   # Expect 15% return
    'GOOG': 0.10,   # Expect 10% return
}

# Create model with covariance matrix
bl = BlackLittermanModel(S, absolute_views=views)

# Get adjusted returns
mu_bl = bl.bl_returns()

# Optimize with adjusted returns
ef = EfficientFrontier(mu_bl, S)
ef.max_sharpe()
weights_bl = ef.clean_weights()

ret_bl, vol_bl, sharpe_bl = ef.portfolio_performance()
```

**Relative Views (Pairs Trading):**

```python
# View: AAPL will outperform GOOG by 5%
views = {
    'AAPL': {'GOOG': 0.05}  # AAPL +5% relative to GOOG
}

bl = BlackLittermanModel(S, relative_views=views)
mu_bl = bl.bl_returns()
```

**Market-Implied Risk Aversion:**

```python
from pypfopt.black_litterman import market_implied_risk_aversion

# Automatically calibrate risk aversion from market prices
delta = market_implied_risk_aversion(market_prices)

# Use in Black-Litterman
bl = BlackLittermanModel(S, absolute_views=views, risk_aversion=delta)
```

**Use Case:** Incorporating analyst forecasts, combining market data with internal views

---

## Classical Optimization Strategies

### Max Sharpe Ratio Portfolio

Maximizes risk-adjusted returns (return per unit of risk):

```python
from pypfopt.efficient_frontier import EfficientFrontier

ef_max_sharpe = EfficientFrontier(mu, S)
ef_max_sharpe.max_sharpe(risk_free_rate=risk_free_rate)
weights_max_sharpe = ef_max_sharpe.clean_weights()
ret_sharpe, std_sharpe, sharpe = ef_max_sharpe.portfolio_performance(
    risk_free_rate=risk_free_rate
)
```

**Use Case:** Best risk-adjusted returns; suitable for most investors

### Minimum Volatility Portfolio

Minimizes portfolio standard deviation:

```python
ef_min_vol = EfficientFrontier(mu, S)
ef_min_vol.min_volatility()
weights_min_vol = ef_min_vol.clean_weights()
ret_min_vol, std_min_vol, sharpe_min_vol = ef_min_vol.portfolio_performance(
    risk_free_rate=risk_free_rate
)
```

**Use Case:** Conservative investors prioritizing stability over returns

### Maximum Utility Portfolio

Balances return and risk according to risk aversion parameter:

```python
risk_aversion = 2.0  # Higher value = more conservative
ef_max_utility = EfficientFrontier(mu, S)
ef_max_utility.max_quadratic_utility(
    risk_aversion=risk_aversion,
    market_neutral=False
)
weights_max_utility = ef_max_utility.clean_weights()
ret_utility, std_utility, sharpe_utility = ef_max_utility.portfolio_performance(
    risk_free_rate=risk_free_rate
)
```

**Use Case:** Customizable risk-return tradeoff based on investor preferences

**Important:** Create separate EfficientFrontier instances for each optimization strategy. Don't reuse the same instance.

## Regularization & Advanced Constraints

### L2 Regularization

Penalizes small weights to reduce portfolio turnover and improve stability:

```python
from pypfopt import objective_functions

ef = EfficientFrontier(mu, S)
ef.add_objective(objective_functions.L2_reg, gamma=1.0)
ef.max_sharpe()
weights = ef.clean_weights()
```

**Effect:** More diversified portfolio with fewer small positions

**Tuning gamma:**

- `gamma=0`: No regularization (standard optimization)
- `gamma=0.5`: Light regularization
- `gamma=1.0-2.0`: Strong regularization
- Higher gamma → more equally weighted

---

### Efficient Risk (Target Volatility)

Maximize returns given a target volatility level:

```python
target_volatility = 0.15  # 15% annual volatility

ef = EfficientFrontier(mu, S)
weights = ef.efficient_risk(target_volatility)
ret, vol, sharpe = ef.portfolio_performance()
```

**Use Case:** Risk budgeting, matching specific risk tolerance

---

### Efficient Return (Target Return)

Minimize volatility while achieving a minimum target return:

```python
target_return = 0.10  # 10% annual return

ef = EfficientFrontier(mu, S)
weights = ef.efficient_return(target_return)
ret, vol, sharpe = ef.portfolio_performance()
```

**Use Case:** Liability matching, achieving specific return goals

---

### Bounded Weights

Constrain individual asset weights:

```python
# Long-only portfolio with max 10% per asset
ef = EfficientFrontier(mu, S, weight_bounds=(0, 0.1))
ef.max_sharpe()
weights = ef.clean_weights()

# Allow shorts with bounds [-0.5, 0.5]
ef = EfficientFrontier(mu, S, weight_bounds=(-0.5, 0.5))
ef.max_sharpe()
```

---

### Sector Constraints

Limit weight allocation to sectors:

```python
# Sector exposure: {sector: max_weight}
sector_weights = {
    'tech': 0.3,     # Max 30% in tech
    'finance': 0.25,  # Max 25% in finance
}

# Create constraint mapping
# Assuming asset_to_sector dict: {ticker: sector, ...}
for sector, max_weight in sector_weights.items():
    sector_assets = [ticker for ticker, s in asset_to_sector.items() if s == sector]
    sector_indices = [list(ef.tickers).index(t) for t in sector_assets]

    ef.add_constraint(
        lambda w: np.sum([w[i] for i in sector_indices]) <= max_weight
    )

ef.max_sharpe()
```

## Discrete Allocation (Dollar Amounts)

Convert continuous weights to actual share quantities and dollar amounts:

```python
from pypfopt.discrete_allocation import DiscreteAllocation, get_latest_prices

# Get latest prices
latest_prices = get_latest_prices(prices_df)
# Alternative: latest_prices = prices_df.iloc[-1]

# Allocate capital
portfolio_value = 100000  # Total portfolio value in currency units
da = DiscreteAllocation(weights, latest_prices, total_portfolio_value=portfolio_value)
allocation, leftover = da.greedy_portfolio()

# Results
print(allocation)  # {ticker: num_shares, ...}
print(leftover)    # Remaining capital after allocation
```

## Key Methods & Utilities

### Clean Weights

Remove negligible weights (typically < 1e-4) and normalize:

```python
weights = ef.clean_weights()
# Returns dict: {ticker: weight, ...}
```

### Portfolio Performance

Calculate portfolio metrics:

```python
annual_return, annual_volatility, sharpe_ratio = ef.portfolio_performance(
    risk_free_rate=0.02
)
```

### Exception Handling

Catch optimization errors:

```python
from pypfopt.exceptions import OptimizationError

try:
    ef.max_sharpe()
except OptimizationError as e:
    print(f"Optimization failed: {e}")
```

## Best Practices

### 1. Risk Model Selection

- **Start with:** Ledoit-Wolf shrinkage (`CovarianceShrinkage().ledoit_wolf()`) as robust default
- **Scale up:** For 50+ assets, consider `single_factor_model()` for numerical stability
- **High-frequency/noisy data:** Test `denoised_covariance()` for improved Sharpe ratios
- **Many correlated assets:** Use `HRPOpt` instead of Efficient Frontier
- **Expert views:** Use Black-Litterman to incorporate forecasts

### 2. Data Preparation

- Use daily closing prices in a pandas DataFrame
- Index should be timestamps; columns should be ticker symbols
- Ensure sufficient historical data:
  - Minimum 1-2 years for standard covariance
  - 3+ years recommended for shrinkage methods
  - 5+ years ideal for factor models

### 3. Instance Management

- Create separate `EfficientFrontier` instances for each optimization strategy
- Reusing instances may cause convergence issues
- Reset constraints between optimizations

### 4. Returns Calculation

- **Log returns:** Better for volatile assets, extended periods, mathematical convenience
- **Simple returns:** Adequate for most portfolio optimization (default)
- **Consistency:** Use same method for expected returns and covariance
- **Forecast method:** Historical mean can be biased; consider expert views (Black-Litterman)

### 5. Risk-Free Rate

- Should match the frequency of your data
- For daily data with annual returns: use annualized rate (e.g., 0.02 for 2% annual)
- Affects Sharpe ratio calculation and max_sharpe optimization
- Use realistic rate matching your opportunity cost

### 6. Regularization Strategy

- Add L2 regularization to reduce small weights and turnover
- Start with `gamma=1.0` and adjust based on portfolio concentration
- Higher gamma → more diversified, more stable out-of-sample
- Trade-off: diversification vs. concentration in best ideas

### 7. Constraints & Bounds

- Use weight bounds to enforce long-only (0, ∞) or allow shorts (-bounds, bounds)
- Add sector constraints for diversification requirements
- Prefer soft constraints (add_objective) over hard bounds when possible
- Test constraint impact on Sharpe ratio and portfolio concentration

### 8. Discrete Allocation

- Always use actual market prices in `DiscreteAllocation`
- Check `leftover` amount; large leftovers indicate thin trading or high prices
- The greedy algorithm adds shares one at a time until budget exhausted
- For small portfolios: manually adjust if allocation deviates significantly

### 9. Out-of-Sample Performance

- Compare multiple risk models on historical data
- Prefer models with better stability across time periods
- Test on recent/validation data before deploying
- HRP often outperforms traditional optimization out-of-sample

### 10. Monitoring & Rebalancing

- Recalculate covariance periodically (monthly/quarterly)
- Use shrinkage methods for stability if data windows are short
- Track realized vs. expected returns to validate assumptions
- Adjust risk aversion/views as market conditions change

## Common Workflows

### Risk Model Comparison Workflow

```python
import pandas as pd
from pypfopt.expected_returns import mean_historical_return
from pypfopt.risk_models import sample_cov, CovarianceShrinkage
from pypfopt.efficient_frontier import EfficientFrontier
from pypfopt.hierarchical_portfolio import HRPOpt

# Load data
prices_df = pd.read_csv("prices.csv", index_col="date", parse_dates=True)
mu = mean_historical_return(prices_df, log_returns=False)

# Compare all risk models
models = {
    'sample_cov': sample_cov(prices_df),
    'ledoit_wolf': CovarianceShrinkage(prices_df).ledoit_wolf(),
    'factor_model': CovarianceShrinkage(prices_df).single_factor_model(),
    'denoised': CovarianceShrinkage(prices_df).denoised_covariance(),
}

results = {}
for name, S in models.items():
    ef = EfficientFrontier(mu, S)
    ef.max_sharpe(risk_free_rate=0.02)
    ret, vol, sharpe = ef.portfolio_performance(risk_free_rate=0.02)
    results[name] = {
        'return': ret,
        'volatility': vol,
        'sharpe': sharpe,
        'weights': ef.clean_weights()
    }

# HRP comparison
hrp = HRPOpt(prices_df)
hrp_weights = hrp.optimize()
hrp_ret, hrp_vol, hrp_sharpe = hrp.portfolio_performance()
results['HRP'] = {
    'return': hrp_ret,
    'volatility': hrp_vol,
    'sharpe': hrp_sharpe,
    'weights': hrp_weights
}

# Print comparison
print("Risk Model Comparison (Sharpe Ratios):")
for name in sorted(results.keys(), key=lambda x: results[x]['sharpe'], reverse=True):
    print(f"  {name:15} Sharpe: {results[name]['sharpe']:.4f}")
```

### Black-Litterman Workflow

```python
from pypfopt.black_litterman import BlackLittermanModel
from pypfopt.efficient_frontier import EfficientFrontier

# Define views: {ticker: expected_return}
views = {
    'AAPL': 0.15,      # Expect 15% return
    'GOOG': 0.12,      # Expect 12% return
}

# Create Black-Litterman model
bl = BlackLittermanModel(S, absolute_views=views)
mu_bl = bl.bl_returns()

# Optimize with BL returns
ef = EfficientFrontier(mu_bl, S)
ef.max_sharpe(risk_free_rate=0.02)
weights_bl = ef.clean_weights()
ret_bl, vol_bl, sharpe_bl = ef.portfolio_performance()

print(f"BL Weights: {weights_bl}")
print(f"BL Sharpe Ratio: {sharpe_bl:.4f}")
```

### HRP (Hierarchical Risk Parity) Workflow

```python
from pypfopt.hierarchical_portfolio import HRPOpt

# HRP doesn't need returns, only prices
hrp = HRPOpt(prices_df)
weights_hrp = hrp.optimize()

# Get performance metrics
ret_hrp, vol_hrp, sharpe_hrp = hrp.portfolio_performance()

print(f"HRP Volatility: {vol_hrp:.4f}")

# Compare with Min Volatility (classical)
ef = EfficientFrontier(mu, S)
ef.min_volatility()
weights_mv = ef.clean_weights()
ret_mv, vol_mv, sharpe_mv = ef.portfolio_performance()

print(f"Min Vol Volatility: {vol_mv:.4f}")
```

### Complete Optimization Pipeline

```python
import pandas as pd
from pypfopt.expected_returns import mean_historical_return
from pypfopt.risk_models import sample_cov
from pypfopt.efficient_frontier import EfficientFrontier
from pypfopt.discrete_allocation import DiscreteAllocation, get_latest_prices

# 1. Load data
prices_df = pd.read_csv("prices.csv", index_col="date", parse_dates=True)

# 2. Calculate inputs
mu = mean_historical_return(prices_df, log_returns=False)
S = sample_cov(prices_df)

# 3. Optimize
ef = EfficientFrontier(mu, S)
ef.max_sharpe(risk_free_rate=0.02)
weights = ef.clean_weights()

# 4. Get metrics
ret, vol, sharpe = ef.portfolio_performance(risk_free_rate=0.02)

# 5. Allocate capital
latest_prices = get_latest_prices(prices_df)
da = DiscreteAllocation(weights, latest_prices, total_portfolio_value=100000)
allocation, leftover = da.greedy_portfolio()

print(f"Allocation: {allocation}")
print(f"Leftover: ${leftover:.2f}")
```

### Comparing Multiple Strategies

```python
strategies = {}

# Max Sharpe
ef = EfficientFrontier(mu, S)
ef.max_sharpe()
strategies['max_sharpe'] = {
    'weights': ef.clean_weights(),
    'metrics': ef.portfolio_performance()
}

# Min Volatility
ef = EfficientFrontier(mu, S)
ef.min_volatility()
strategies['min_vol'] = {
    'weights': ef.clean_weights(),
    'metrics': ef.portfolio_performance()
}

# Max Utility
ef = EfficientFrontier(mu, S)
ef.max_quadratic_utility(risk_aversion=2.0)
strategies['max_utility'] = {
    'weights': ef.clean_weights(),
    'metrics': ef.portfolio_performance()
}
```

## Advanced Options

### Bounds on Weights

Constrain individual asset weights:

```python
ef = EfficientFrontier(mu, S)
ef.add_constraint(lambda w: w >= 0)      # Non-negative weights (long-only)
ef.add_constraint(lambda w: w <= 0.1)    # Max 10% per asset
ef.max_sharpe()
```

### Market Neutrality

Allow short positions:

```python
ef.max_quadratic_utility(risk_aversion=2.0, market_neutral=True)
# Sums to zero: long and short positions offset
```

### Transaction Costs & Turnover

The library supports transaction costs and turnover constraints through optimization parameters.

## Documentation & Resources

- GitHub: https://github.com/robertmartin8/pyportfolioopt
- Installation: `pip install PyPortfolioOpt`
- Examples in code snippets: 146+ available examples