It covers posterior predictive checks, diagnostics, contribution analysis,
response curves, efficiency metrics, and the tabular summary surfaces that
Abacus exposes from fitted InferenceData.
Pages
Posterior Predictive: Sample fitted or future
predictions and compare them with observed data where available.
Diagnostics: Run design-matrix, MCMC, and predictive
diagnostics and export machine-readable reports.
Response Curves: Sample and summarise posterior
saturation and adstock curves, and understand the runner’s forward-pass
direct contribution artefacts.
ROAS and Metrics: Calculate ROAS, CPA-style metrics,
spend tables, and predictive error metrics.
Summary and Export: Work with MMMSummaryFactory,
HDI settings, time aggregation, and DataFrame export.
Subsections of Post-Modeling
Diagnostics
Abacus exposes diagnostics through mmm.diagnostics.
Use this surface to check the design matrix, posterior sampling quality, and
posterior predictive fit. For fitted-value plots and predictive sampling, see
Posterior Predictive.
Diagnostic surfaces
mmm.diagnostics provides three groups of checks.
Area
Summary method
Report method
What it covers
Raw input screening
design_summary(X)
design_report(X)
Collinearity, constants, and near-constant regressors on raw input columns
MCMC
mcmc_summary()
mcmc_report()
r_hat, ESS, divergences, BFMI, tree depth, acceptance rate
Predictive
predictive_summary()
predictive_report()
RMSE, MAE, NRMSE, NMAE, CRPS, residual moments
The summary methods return pandas DataFrames. The report methods return typed
report objects with a to_dict() method for JSON-ready export.
Raw input screening
Use design_summary(X) on the raw design matrix you want to inspect:
design_report(X) returns a compact roll-up with matrix rank, condition
number, maximum VIF, maximum absolute correlation, and lists of flagged
variables.
Screening requirements
Raw input screening requires:
all requested columns to exist in X
all checked columns to be numeric
Abacus raises a ValueError if a variable is missing or non-numeric.
The method names stay design_summary() and design_report(), but the
pipeline now treats them as raw input screening rather than transformed model
geometry.
MCMC summaries and reports retain unrounded diagnostics. Parameter flags use
inclusive boundaries: R-hat at or above the threshold and bulk/tail ESS at or
below the threshold are flagged, matching the pipeline gate comparisons.
Round values only for display, after classification.
Divergence reports distinguish available evidence from unavailable evidence.
Missing, empty, malformed or mismatched divergence flags produce
divergence_status="unavailable", null count/rate values and a reason. Only a
valid array covering the retained chain/draw coordinates can establish zero
divergences. Stage 50 warns when this evidence is unavailable, preventing an
all-pass diagnostic rollup. Absence alone does not establish that divergences
are inapplicable to the sampler.
R-hat and ESS thresholds screen Monte Carlo exploration; passing them does
not establish model validity or causal identification. Investigate retained
divergences even when other diagnostics pass. For interpretation and remedies,
see MCMC Diagnostics for Econometricians.
If idata is missing, Abacus raises an error and tells you to fit the model
first.
Example MCMC diagnostic output:
Predictive diagnostics
Scoring requires exactly matching observation dimensions and coordinate labels.
Matching labels may appear in a different order; predictions are reordered to
match the target. Missing, extra, duplicate, null or unlabelled observation
coordinates raise an error. Dimensions are not implicitly broadcast.
To score a subset, select the same intended observations explicitly on both
arrays before calling predictive_summary_from_arrays(). Empty observations
or samples are rejected.
Predictive diagnostics use the observed target and stored posterior predictive
samples:
The predictive summary is a one-row DataFrame with:
scale
num_observations
rmse
mae
nrmse
nmae
crps
residual_mean
residual_std
Abacus aligns target and prediction coordinates before flattening. That
includes mixed datetime coordinate dtypes when needed.
Predictive metric definitions
Let y_i be an observed target and m_i its posterior predictive mean, with
residual r_i = y_i - m_i. Abacus aligns labels and flattens all observation
dimensions, including panel dimensions, into one equally weighted aggregate.
num_observations counts those entries, not just unique dates.
Field
Definition
rmse
Square root of the mean of r_i ** 2
mae
Mean of abs(r_i)
nrmse
RMSE divided by max(y) - min(y) on the scored observations
nmae
MAE divided by the same observed range
crps
Mean continuous ranked probability score over observations, using all predictive draws
residual_mean
Mean of r_i; positive means underprediction
residual_std
Standard deviation of r_i with ddof=0
bias
The same signed mean as residual_mean, when requested by the array helper or Stage 35
NRMSE and NMAE return NaN when the observed range is approximately zero
(numpy.isclose(range, 0.0)). Range normalisation does not make arbitrary
windows comparable: their ranges, composition and prediction tasks can differ.
For RMSE, MAE and CRPS, lower scores are better on a comparable evaluation set;
none is a causal-identification test. CRPS can be written as
E|Y - y_i| - 0.5 * E|Y - Y'|, with independent predictive draws Y and Y'.
It assesses a predictive distribution rather than only its mean.
Stage 35 also requests empirical coverage. For probability p, take each
observation’s predictive quantiles at (1 - p) / 2 and (1 + p) / 2, then
average the indicator that the observed target lies between them, including
the endpoints. Entries with non-finite targets or bounds are excluded from
that coverage denominator; if none remain, coverage is NaN.
num_observations remains the full aligned count, not the finite coverage count.
These are equal-tailed intervals, distinct from the summary facade’s HDIs.
The ordinary mmm.diagnostics.predictive_summary() does not add coverage or
bias columns. See holdout interpretation.
Example residual diagnostics:
Export reports
Use the report objects when you want a compact export format:
The same pattern works for design_report(...) and predictive_report().
Pipeline outputs
The pipeline diagnostics stage uses the same retained diagnostic surfaces to
write report tables and text summaries. If you run the pipeline, those stage
artefacts should match the behaviour documented here.
In the structured pipeline, the raw-input screening rows in
diagnostics_report.csv use the phase label raw_input_screening instead of
design so the machine-readable output matches the wording here.
Common pitfalls
Running mcmc_summary() or mcmc_report() before fitting
Running predictive diagnostics before sampling posterior predictive values
Passing non-numeric columns into design_summary(X)
Treating predictive diagnostics as a substitute for design or MCMC checks
Posterior Predictive Checks
Use posterior predictive draws to check in-sample fit and to generate
predictions for new rows that follow the fitted panel layout.
If you only want the returned samples and do not want to update mmm.idata,
set extend_idata=False.
Check training-fit values against observed data
For an in-sample check, pass the same design matrix you used for fitting. This
is the same pattern used by the pipeline’s Stage 30 training-fit assessment:
The retained images below come from earlier examples. Their generating
configuration and seed are not recorded here, so they are not reproducible
outputs of the preceding 94% call. Use their captions and legends to identify
the displayed interval and scale.
The first image shows a predictive median and a band labelled 85% HDI. Its
vertical axis has values around 0.2–0.8 and does not identify the target units.
The code above requests 94%; do not use this image as its expected interval or
infer original-scale business values from the axis.
The second image shows observed values and a fitted mean on a target axis
marked 1e6, with a band labelled 94% interval. It uses a different displayed
scale from the first image; no conversion between the two is established
here. Inspect systematic misses and the relationship between observations and
the predictive band. The displayed in-sample metrics do not establish
holdout performance or causal validity.
mmm.summary.posterior_predictive() returns a table with:
observed target values
posterior predictive mean and median
HDI bound columns such as abs_error_94_lower and abs_error_94_upper
You can also access the predictive draws directly:
sample_posterior_predictive(...) does not take y. For a holdout or future
window, keep the actual target outside the model and align it yourself if you
want external evaluation.
Use include_last_observations correctly
Set include_last_observations=True when the forecast window needs lag history
for adstock carryover.
When enabled, Abacus:
prepends the last adstock.l_max training observations internally
samples posterior predictive values on the padded data
removes the prepended rows from the returned result
This only works when the input dates do not overlap with the training dates.
If they do overlap, Abacus raises a ValueError.
Practical guidance
Use the training X for fitted-versus-observed checks.
Use future-only dates for forward prediction.
Use the training-window refit pattern for blocked holdout validation.
Keep combined=True if you want a simpler sample dimension.
Use combined=False if you need explicit chain and draw dimensions.
Call sample_posterior_predictive(...) before using
mmm.diagnostics.predictive_summary() or mmm.summary.posterior_predictive().
Common pitfalls
Calling sample_posterior_predictive(...) without X
Expecting y to be passed into the predictive method
Using include_last_observations=True on dates that overlap with training
data
Forgetting that the returned object is extracted samples, while the stored
idata.posterior_predictive group keeps the native posterior predictive
structure
Contributions and Decomposition
Abacus stores additive contribution terms for fitted PanelMMM models and
exposes them through the data wrapper, summary tables, and plotting suite.
Use this page to inspect media, baseline, control, seasonality, and event
effects. For channel efficiency ratios built from media contributions, see
ROAS and Metrics.
Contribution surfaces
You can work with contributions at three levels.
Surface
Use it for
mmm.data
Raw xarray contribution samples
mmm.summary
DataFrames with posterior means, medians, and HDIs
mmm.plot
Time-series and waterfall visualisations
Read raw contribution samples
The lowest-level accessor is mmm.data.get_contributions(...):
Depending on the fitted model, the returned dataset can contain:
channels
baseline
controls
seasonality
events
baseline includes the intercept contribution and any legacy Mundlak
contribution when the fitted model uses use_mundlak_cre=True. For the named
CRE preset, the structured pipeline reports the CRE adjustment separately and
retains it on the baseline or non-incremental side of the decomposition.
identifying columns such as date, channel, control, and any panel
dims
mean
median
HDI bound columns such as abs_error_94_lower and abs_error_94_upper
mmm.summary.contributions(...) does not expose event effects. For event
effects, use mmm.data.get_contributions(include_events=True) or
mmm.summary.mean_contributions_over_time().
Create a wide decomposition table
Use mmm.summary.mean_contributions_over_time(...) when you want one row per
time point and panel slice:
Use original_scale=True when you want business-unit interpretation.
Use mmm.summary.contributions(...) for tidy per-component tables.
Use mmm.summary.mean_contributions_over_time() for decomposition exports.
Use mmm.summary.total_contribution() when you only need component-level
totals.
Common pitfalls
Expecting mmm.summary.contributions(...) to include event effects
Forgetting that baseline can include more than the intercept when legacy
Mundlak CRE is enabled, or that the named CRE adjustment is non-incremental
Using frequency="all_time" with mean_contributions_over_time() or
change_over_time()
Response Curves
Use response curves to inspect the fitted media transformations directly.
Abacus exposes posterior saturation and adstock curves through both the fitted
model and mmm.summary. For decomposition of realised contributions over time,
see Contributions and Decomposition.
Sample saturation curves
Use sample_saturation_curve(...) on a fitted PanelMMM:
The adstock curve is the fitted decay pattern for an impulse of size
amount. It does not use an original_scale option because the returned
weights are not target-unit contributions.
Runner-generated direct contribution artefacts
If you use the retained pipeline runner, Stage 60_response_curves also writes
a forward-pass direct contribution artefact alongside the saturation and
adstock transformation curves:
forward_pass_contribution_curve.nc
forward_pass_contribution_curve_summary.csv
forward_pass_contribution_curve.png
This artefact is different from the saturation-only curve:
the saturation-only curve shows the fitted saturation transform itself
the forward-pass direct contribution curve runs spend through the full fitted
model path, including adstock and saturation
The retained Stage 60 forward-pass plot uses one explicit scenario so the curve
is interpretable: it rescales the full observed historical spend path from
0% to 200%, then plots total channel spend against total channel
contribution in original units. The marker at 100% highlights the fitted
total contribution for the observed historical spend path.
These methods return DataFrames with posterior mean, median, and pointwise
HDI bounds. A sample axis uses the same HDI calculation as chain/draw;
these intervals are not simultaneous bands for the entire curve. See
Summary interval semantics
for the single-interval definition and the correction to earlier equal-tailed
curve summaries.
saturation_curves(...) includes an x column. adstock_curves(...) uses
time since exposure.
MMMSummaryFactory requirement
Curve summaries need access to both the fitted data and the fitted model
transformations.
mmm.summary already satisfies that requirement. If you construct
MMMSummaryFactory manually, pass model=mmm:
These are direct ratios built from fitted media contributions and the model’s
channel inputs, stored as constant_data.channel_data. Abacus does not look
up a separate monetary spend series or convert exposures to spend.
With original_scale=True, the financial interpretations require:
Ratio
Required units
Interpretation
contribution / channel input
Revenue and monetary spend in a common currency
ROAS: revenue per unit of spend
channel input / contribution
Monetary spend and a conversion-count target
Cost per conversion: currency per conversion
Use consistent currency, time periods and panel aggregation for both sides of
the ratio. For example, £200 of contribution divided by £100 of spend gives
ROAS 2; £200 divided by 1,000 impressions gives £0.20 per impression, not ROAS.
If contributions are left on the scaled target space, the result is not in
original business units.
These are model-conditional contribution ratios. Their posterior intervals do
not establish incremental causal returns; see
Causal Identification.
The arrays are element-wise over time, channel, and any panel dims, with
posterior sample dimensions on top.
Abacus returns NaN when it would otherwise divide by zero.
Summarise ROAS
Use mmm.summary.roas(...) for a tidy summary table:
This is the same retained summary surface that mmm.summary.efficiency() uses
for target_type="conversion".
Use the default efficiency metric
Abacus chooses the default efficiency metric and label from the target type.
This selects an accessor; it does not validate currency or convert channel
inputs to spend. The financial labels below require the units stated above:
Use channel_spend() when you want the raw channel-input table with no posterior
aggregation:
spend_df=mmm.summary.channel_spend()
This returns the observed channel inputs with columns such as date, channel,
panel dims, and channel_data. They represent spend only when the model’s
channel inputs are monetary amounts.
Predictive error metrics
Predictive metrics live under mmm.diagnostics.predictive_summary():
These metrics are calculated from the stored posterior predictive samples and
the observed target. See the canonical
predictive metric definitions,
including the observed-range denominator for NRMSE/NMAE and its constant-target
NaN case.
Practical guidance
Use roas() for revenue targets with monetary channel inputs in a common currency.
Use cost_per_target() for conversion-count targets with monetary channel inputs.
Use efficiency() when you want target-type-aware reporting.
Sample posterior predictive values before using predictive metrics.
Common pitfalls
Reporting revenue per exposure as ROAS because a channel input was labelled
spend without checking its units
Forgetting that zero spend or zero contribution produces NaN
Using predictive diagnostics before calling
sample_posterior_predictive(...)
Summary and Export
mmm.summary is the retained tabular summary surface for fitted PanelMMM
models.
It is backed by MMMSummaryFactory and returns pandas or polars DataFrames
that you can export with normal DataFrame methods.
For predictive diagnostics and JSON-ready reports, see
Diagnostics.
Use mmm.summary
The simplest path is the bound summary factory on the fitted model:
Summary tables include interval columns named from those probabilities. For
example, hdi_probs=[0.94] produces columns such as:
abs_error_94_lower
abs_error_94_upper
These columns contain interval endpoints, not error magnitudes. Abacus uses
ArviZ’s single contiguous empirical HDI for both chain/draw and sample
layouts. Reshaping the same draws does not change the bounds. Means, medians,
coordinate labels and output column names are independent of that layout.
NaNs are not silently removed before calculating the HDI; inspect non-finite
bounds before interpreting or exporting them.
Each interval summarises one output coordinate. Curve intervals are pointwise,
not simultaneous bands over a whole response curve. A single contiguous
interval also need not represent a disconnected highest-density set for a
multimodal posterior.
Earlier versions used equal-tailed quantiles for arrays with a sample
dimension, including curve summaries. Those bounds can differ materially from
HDIs for skewed draws. Recompute affected tables from the same saved draws to
obtain consistent HDIs; no refit is required. Means, medians and column names
remain unchanged.
Blocked holdout coverage
uses equal-tailed predictive intervals. Its coverage_* fields do not measure
coverage of these HDI columns.
Aggregate over time
Many summary methods accept frequency with one of these values: