Blocked Holdout Validation

Stage 35 is Abacus’s out-of-sample time-series validation layer.

It answers a narrower and more useful question than “does the model fit the training data?”:

“If I refit the MMM on the earlier history only, can it still predict the last blocked window reasonably well?”

For weekly MMM, that is usually a better stress test than a random split because media carryover, seasonality, and trend all depend on time order.

Why Abacus uses a blocked tail holdout

Abacus reserves the final holdout_observations unique dates as the holdout window, then fits a fresh model on the earlier dates only. The validation pipeline does not reuse Stage 20 posterior state.

That means Stage 35 is checking whether the full model specification can generalise forward in time, not whether the same fitted posterior can explain the rows it already saw.

This is especially useful in MMM because:

  • adstock depends on lagged spend history
  • seasonality is time-ordered rather than exchangeable
  • marketing calendars often drift near the end of the sample
  • overfit specifications can look fine in-sample and fail on the final weeks

See the implementation in validation.py.

How Stage 35 works

Given a YAML block such as:

validation:
  enabled: true
  holdout_observations: 8
  include_last_observations: true
  coverage_levels: [0.5, 0.8, 0.94]
  sampler:
    draws: 500
    tune: 500
    chains: 2
    cores: 2
    random_seed: 42

Abacus does the following:

  1. Sort all unique model dates.
  2. Reserve the last holdout_observations dates as the holdout window.
  3. Fit a fresh model on the remaining earlier dates only.
  4. Sample posterior predictive draws for the holdout rows.
  5. Compute uncertainty-aware predictive metrics and residual diagnostics.

If include_last_observations: true, Abacus prepends the trailing lag history needed for adstock carryover internally, then trims those prepended rows back out of the returned holdout predictions. This matters whenever media effects have memory.

Random seeds

Holdout posterior prediction explicitly receives the effective validation fitting seed. Seed precedence is the same as for the refit: base YAML fit, then runner sampler overrides, then validation.sampler. The metadata records prediction_random_seed alongside the effective sampler_config. A zero seed is valid; a missing or null effective seed leaves prediction unseeded and is recorded as null.

Reusing a seed supports repeatability within the same environment and execution settings; it does not promise identical draws across dependency versions or backends. No independent-stream derivation is applied.

Why this stage exists when Stage 30 already exists

Stage 30 and Stage 35 answer different questions.

  • Stage 30 is an in-sample fit check. It uses the same rows the model was fit on.
  • Stage 35 is an out-of-sample blocked holdout check. It uses future dates the validation fit did not see.

If Stage 30 looks good and Stage 35 looks weak, that is a classic warning sign of overfit, misspecification, or regime change.

Main artefacts

Stage 35 writes the following files under results/<run_name>_<timestamp>_<random_suffix>/35_holdout_validation/:

  • validation_metadata.json
  • holdout_posterior_predictive.nc
  • holdout_predictive_summary.csv
  • holdout_predictive_report.json
  • holdout_observed.csv
  • holdout_fitted.csv
  • holdout_residuals.csv
  • holdout_timeseries.png
  • holdout_residuals_acf.png

See also Output Directory Schema.

How to interpret the main metrics

The headline table is holdout_predictive_summary.csv.

The canonical predictive metric definitions specify the formulas, observation aggregation and missing-value boundaries.

Point-error metrics

RMSE gives larger errors more weight than MAE. NRMSE and NMAE divide those scores by the observed target range in the scored holdout, returning NaN when that range is approximately zero. Compare scores on comparable windows; range normalisation alone does not make different datasets comparable.

Use these as forecast-quality metrics, not causal-identification metrics.

Bias

bias is observed minus posterior predictive mean, averaged over observations:

  • positive bias means underprediction on average
  • negative bias means overprediction on average

For example, observed 12 and predicted mean 10 gives bias +2. Inspect persistent bias alongside trend changes, omitted predictors and changes in the holdout period. Its sign alone does not identify the cause.

CRPS

The continuous ranked probability score (CRPS) assesses the full predictive distribution. Lower is better on the same evaluation set. Inspect it alongside point errors, coverage and interval width; one score does not establish calibration.

Coverage

Stage 35 reports coverage_50, coverage_80 and coverage_94, the observed fractions inside equal-tailed posterior predictive intervals at those nominal probabilities. They do not measure coverage of mmm.summary HDIs. See the metric definitions for the quantiles, endpoint inclusion and finite-observation denominator.

Compare coverage with its nominal probability alongside bias, interval width, residual patterns, holdout size and dependence between observations. Low coverage can reflect narrow intervals, prediction bias or distributional change. High coverage does not by itself prove that intervals are too wide.

With eight finite aggregate observations, coverage changes in steps of 1/8. Seven covered observations give 0.875, so exact agreement with 0.94 is impossible. That result alone cannot establish or refute nominal 94% calibration. Serial or panel dependence further limits the information in a short holdout; panel rows do not automatically supply independent evidence.

How to read the plots

holdout_timeseries.png

This is the first plot to inspect.

Look for:

  • whether the observed series generally stays inside the predictive intervals
  • whether misses are isolated or systematically one-sided
  • whether the model misses turning points or holiday spikes
  • whether the predictive band width looks plausible relative to the volatility of the target

Common interpretations:

  • repeated misses on the same side: likely bias
  • clustered misses: inspect bias, changing volatility and omitted time structure
  • wide bands: inspect predictive spread and its sensitivity to the specification; width alone does not diagnose identification

holdout_residuals_acf.png

This checks whether the holdout residuals still contain serial structure.

Look for:

  • obvious positive autocorrelation across nearby lags
  • repeating seasonal patterns
  • long runs of same-sign residuals

If residual autocorrelation is strong, the model is usually still missing some time structure such as:

  • seasonality
  • holiday dynamics
  • delayed media effects
  • structural breaks or time-varying baseline behavior

Practical rules of thumb

These are pragmatic MMM heuristics, not hard pass/fail thresholds.

Choosing the holdout window

  • For weekly MMM with roughly 1 to 2 years of data, 6 to 12 weeks is a practical starting range.
  • 8 weeks is a sensible default when you want enough tail signal without throwing away too much history.
  • If the dataset is very short, a larger holdout can make the validation noisy and can leave too little training history for stable estimation.

Abacus itself enforces that blocked holdout validation must still leave enough training dates for the model to run; see validation.py.

Comparing Stage 30 and Stage 35

  • Expect Stage 35 to be worse than Stage 30. That is normal.
  • Worry when the degradation is large or the direction changes materially.
  • If Stage 30 is excellent and Stage 35 is weak, suspect overfit or misspecification before celebrating the in-sample fit.

Reading coverage

  • Report nominal probability, empirical coverage and the number of scored observations.
  • Investigate departures alongside bias and width; do not infer the cause from coverage alone.
  • Use additional comparable windows when feasible; one short holdout does not establish calibration.

Reading bias

  • A small nonzero bias is normal.
  • Large consistent bias over the holdout tail is a stronger warning sign than a single noisy miss.
  • If bias keeps the same sign across several model variants, inspect trend, holidays, and baseline structure before changing media priors.

Reading residual structure

  • White-noise-like residuals are what you want.
  • Visible residual runs or autocorrelation usually mean the model is still missing systematic time variation.
  • Do not treat a decent RMSE as sufficient if residuals still show structure.

What Stage 35 does not tell you

Blocked holdout validation is valuable, but it is not a causal guarantee.

It does not prove:

  • that channel attribution is identified
  • that ROAS is unbiased
  • that the chosen priors are correct
  • that the model is safe for large budget reallocation on its own

It does tell you whether the specification can forecast a held-out tail window coherently. That makes it an important diagnostic, but still only one part of MMM model assessment.

For most weekly MMM work:

  1. Run Stage 30 and Stage 35 together.
  2. Compare in-sample and holdout metrics before changing the specification.
  3. Use the same holdout window across candidate models so the comparison is fair.
  4. Prefer specifications that are stable across reasonable prior choices, not just the one that scores best on a single holdout.
  5. Treat Stage 35 as a forecasting sanity check alongside prior predictive checks, posterior predictive checks, and substantive business review.

Common mistakes

  • Using a random split instead of a blocked time split
  • Reading Stage 30 as out-of-sample validation
  • Ignoring coverage and focusing only on RMSE
  • Using a holdout that is too long for the amount of history available
  • Repeatedly tuning the spec to one holdout window until the score looks good