ML backtest methodology¶

The ML model that feeds PigPro's 30%-weighted ml_projection component is a per-position GradientBoostingRegressor. This page explains the exact training setup, what baselines we compare against, and the two generations of results (before and after adding rookie features).

The training frame¶

For every (player, season) pair in a training season, we join:

flowchart LR
    A[nflverse_weekly_stats
season X] -->|actual total PPR| LABEL[y = season PPR]
    B[nflverse_weekly_stats
season X-1] -->|aggregate per player| PRIOR[prior-season features]
    C[nflverse_snap_counts
season X-1] --> PRIOR
    D[nflverse_player_ids] -->|age, draft slot| ROOKIE[rookie features]
    E[nflverse_combine] --> ROOKIE
    PRIOR --> X[X features]
    ROOKIE --> X
    X --> MODEL[GBR per position]
    MODEL -->|predict| YHAT[ŷ season PPR]
    LABEL --- YHAT

Source: ml_pipeline/draft_value_trainer.py → _build_player_season_frame(). The SQL joins are documented there.

Features (27 total)¶

Prior-season performance (from weekly_stats aggregates)

• prior_games, prior_total_ppr, prior_ppg
• prior_carries, prior_rush_yards, prior_rush_tds
• prior_targets, prior_receptions, prior_rec_yards, prior_rec_tds
• prior_pass_attempts, prior_pass_yards, prior_pass_tds
• prior_avg_target_share, prior_avg_air_yards_share

Prior-season usage (from snap_counts)

• prior_avg_off_snap_pct, prior_total_off_snaps, prior_snap_games

Player profile

• age

Rookie signals (the 2024 gap fix — see below)

• is_rookie (draft_year == season)
• years_into_league
• nfl_draft_round, nfl_draft_ovr
• combine_forty, combine_weight, combine_vertical, combine_broad_jump

Label¶

actual_total_ppr = sum of fantasy_points_ppr across weeks 1-18 of the target season, regular season only.

Per-position modeling¶

One GradientBoostingRegressor per position (QB, RB, WR, TE). Hyperparams: n_estimators=200, max_depth=4, learning_rate=0.05, random_state=42. No hyperparameter search — these are known-reasonable defaults that haven't been tuned further.

The per-position split matters because the feature signal differs wildly — a prior_rush_yards=0 tells you a lot about RB future but nothing about WR future.

The temporal split¶

Train: 2022 + 2023 seasons (each season as features-of-prior + label-of-current). ~1,100 training examples.

Test: 2024 season (features from 2023, label from 2024). 559 players with complete prior-year data.

Why 2024 as holdout: it's the last completed NFL season nfl_data_py published. The 2025 NFL season was played but upstream hasn't pushed the parquet yet. When that publishes, the backlog task retrains on 2022-2024 and validates on 2025.

Validation metric: MAE (mean absolute PPR-point error), R² (variance explained), Pearson correlation, and position-specific top-20 hit rate.

The baselines we beat (or try to)¶

No model is good in a vacuum — we only care about whether it's meaningfully better than:

Persistence: "assume the player scores the same as last year." Literally ŷ = prior_total_ppr. It's dumb. It's hard to beat.

Position median: predict the median total PPR for the player's position across the training seasons. A pure no-signal baseline — whatever our model does, it should crush this.

We also report top-20 hit rate: of the 20 players the model ranks highest at each position, how many actually ended up in the top 20 by actual PPR?

Generation 1 — pre-rookie-features (v1)¶

Initial trained model with only prior-season + age features.

Per-position MAE:

Persistence won every position. That stung.

Top-5 biggest misses (actual vs predicted):

All five were 2024 rookies. They had zero prior-season NFL stats in our feature set, so the model predicted near-zero and they produced like first-round picks.

The pattern was obvious: fix the rookie blind spot, the model wins.

Generation 2 — with rookie features¶

Added 4 features from existing (but unused-by-the-model) nflverse tables:

• is_rookie binary flag
• nfl_draft_round (1-7, 99 for UDFAs)
• nfl_draft_ovr (1-262)
• combine_forty, combine_weight, combine_vertical, combine_broad_jump

Retrained on the same 2022+2023 data, retested on 2024.

Per-position MAE:

Top-20 hit rate (predicted top-20 ∩ actual top-20):

Per-position CV-R² on training data:

Feature importance ranking: nfl_draft_ovr landed in the top 5 features for all four positions. The signal was real.

New top-5 misses after the fix:

1. Joe Burrow — injury recovery year
2. Sam Howell — depth-chart churn
3. Aaron Rodgers — retirement U-turn
4. Jake Browning — backup who started unexpectedly
5. Bucky Irving — late-round rookie RB, limited college snaps (still surfaces at PigPro 8.00 now)

None of the old 5 rookie misses remain on the list. The rookie-feature fix closed that specific gap. The remaining misses are a different class of error — context shifts we don't model yet (injury recovery, depth-chart churn, retirements).

Generation 3 — v1.1 Hybrid (current shipping)¶

Promoted April 2026 (PR #47/#48). Adds Usage / Opportunity as an 8th component while keeping all seven v1 signals intact. Key insight from the failed v1.5 experiment that preceded it: dropping ADP Value entirely (replacing it with an ADP–vs–ML gap signal) regressed Spearman by −0.054. v1's market-consensus signal is too strong to throw away. v1.1 is "v1 + Usage overlay."

What's new¶

• Usage component at 8% — target_share × 0.55 + snap_pct × 0.45 for WR/TE; rush_share × 0.45 + target_share × 0.25 + snap_pct × 0.30 for RB; QB special case folded into ML Projection.
• Weights rebalanced from 30/25/20/10/10/2.5/2.5 (sum 1.0) to 27.6/23/18.4/9.2/9.2/8.0/2.3/2.3 (sum 1.0).
• QB override: when position == 'QB', the 8% Usage weight folds into ML Projection (so QB ML weight = 35.6%). Necessary because the Usage formula couldn't differentiate pocket QBs from rushing QBs by enough percentile range.
• Rookie-proxy logic for Usage on players with <12 career NFL games AND draft_year >= season - 2. Uses nflverse_player_ids.draft_ovr to set Usage from draft capital (pick 1 → 9.96, UDFA → 2).

v1 vs v1.1 against 2025 actuals¶

flowchart LR
    A[pigpro_baseline
version=v1] -->|persist via PR #43| HARNESS[pigpro_compare
back-test against 2025 PPR]
    B[pigpro_baseline
version=v1.1] --> HARNESS
    HARNESS --> RESULTS[per-position Spearman + top-N precision]

n = 533 players matched between pigpro_baseline and nflverse_weekly_stats 2025 totals.

Variant testing — why v1.1-B (92/8) won¶

The promotion PR tested four blend ratios of v1-composite × Usage. All four beat v1 on overall Spearman; the differentiator was RB top-24:

C and D had marginally higher rank correlation but cost 4 points of RB top-24 precision — RBs are the foundation of every fantasy draft, so we picked the variant that held them. Goldilocks: B.

Notable lifts under v1.1¶

The biggest legitimate score increases vs v1 (after the rookie-gate fix to exclude retired QBs from the proxy path):

• JJ McCarthy (rookie QB1) — v1 1.85 → v1.1 2.47
• Travis Hunter (#2 overall pick WR) — 2.67 → 3.25
• Ascending TE tier (the gap v1 missed): TJ Hockenson, Cole Kmet, Pat Freiermuth, Tyler Conklin, Cade Otton — all up 0.4–0.5 PigPro
• High-snap WRs v1 underweighted: Amon-Ra St Brown, Chris Godwin, Michael Pittman

The drops list (Bryce Ford-Wheaton, Kyle Monangai, Blake Corum) is entirely deep-bench players v1 was overrating. No star drops to a bust tier.

How does Gen 2 compare to persistence overall?¶

Honest answer: tied on variance explained, slightly behind on MAE, but wins where it matters.

• The model wins: R² (0.42 vs 0.34), correlation (matches 0.68), WR MAE and top-20 hit rate.
• The persistence baseline still edges out on overall MAE (48.2 vs 49.5) because it has a flatter floor on the long tail of low-scoring veterans.
• What the model uniquely does well: surfacing rookies. Persistence literally can't predict a rookie because it reads from prior_total_ppr=0 → 0.

We ship the model because:

1. Draft strategy benefits from rookie signal; persistence cannot give you that.
2. WR is the deepest position; winning WR MAE + top-20 hit rate compounds weekly.
3. The model output is one input among seven in PigPro, not the whole score. 30% weight means imperfect projection still helps the composite if it's directionally right.

Running the backtest yourself¶

The whole thing is reproducible end-to-end:

# In a backend pod or locally with DB_PASSWORD set
python -m ml_pipeline.backtest --persist

This trains on 2022+2023, scores 2024, writes a row to ml_backtests, and prints the per-position table. Persisted rows show up at /api/ml/backtest and on the /draft-analytics page.

Source files:

• ml_pipeline/backtest.py — the harness itself.
• ml_pipeline/draft_value_trainer.py — the model definition + training.
• ml_pipeline/draft_value_features.py — SQL + feature engineering.
• ml_pipeline/value_scorer.py — how the ML prediction is normalized to PigPro's 0-10 scale.

What's next (Tier 2)¶

• 2025 outcome-based weight refit — once the 2025 NFL season closes and nflverse_weekly_stats has full 2025 totals, run pigpro_v1_1_hybrid.fit_weights(2025) with the constrained-SLSQP fitter against actual rank correlation. First chance to learn weights from data instead of intuition.
• QB Usage reformulation — current QB override is a workaround. A reformulated formula incorporating passing_attempts_share, designed_run_rate, and red_zone_designed_touches would let QBs participate in Usage scoring without penalising pocket passers.
• Empirical ADP from mocks — mock-draft corpus (~19K decisions in draft_picks_v2) can replace the recommender's Gaussian prob_available estimate with measured per-player survival curves. Direct accuracy upgrade for the Draft Room.
• Public-source ADP ensemble — Sleeper + FFC are persisting nightly to adp_records (April 2026). Phase 4 of that work: median-across-sources ensemble feeding ADP Value Gap so we stop deriving ADP from a single source.
• Context-shift features for ML — injury_return flag, team_change flag, QB_change flag. Would catch Burrow-type misses.
• XGBoost / LightGBM — more capacity with better handling of heterogeneous features. GBR is a reasonable baseline; there's low-hanging accuracy beyond it.