Runner Overview

Use the pipeline runner when you want a full disk-backed PanelMMM run instead of only an in-memory fit.

The runner loads a YAML config and a CSV dataset, builds the model, executes a fixed stage sequence, and writes each stage’s artefacts into a structured run directory. When validation is enabled, the runner performs a second train-window fit for the blocked holdout stage, so the run takes longer than a pure full-sample fit.

If you want a quick first run, start with Quickstart: Pipeline Runner.

Public entry points

The public Python API is:

• abacus.pipeline.PipelineRunConfig
• abacus.pipeline.run_pipeline
• abacus.pipeline.PipelineRunResult

The thin CLI wraps the same code path:

python -m abacus.pipeline.runner --config path/to/config.yml

Basic Python example

from pathlib import Path

from abacus.pipeline import PipelineRunConfig, run_pipeline

result = run_pipeline(
    PipelineRunConfig(
        config_path=Path("data/demo/geo_panel/config.yml"),
        output_dir=Path("results"),
        run_name="geo_panel_baseline",
        prior_samples=10,
        draws=500,
        tune=500,
        chains=2,
        cores=2,
        random_seed=42,
        curve_samples=100,
        curve_points=100,
    )
)

print(result.run_dir)
print(result.manifest_path)

PipelineRunResult contains:

What the runner does

run_pipeline(...) performs these steps:

1. Load the YAML config with load_yaml_config(...).
2. Load X and y from CSV using load_pipeline_data(...).
3. Merge CLI sampler overrides with YAML fit through build_model_kwargs(...).
4. Create the output directory tree and initialise run_manifest.json.
5. Run the retained stages in order, updating the manifest after every stage.

For named estimators, Stage 00 prepares the model and stores it in the shared PipelineContext; Stage 10 completes the deferred graph before prior predictive sampling. Configurations without a named estimator use the direct builder path, which can complete the graph in Stage 00. A populated model_class field identifies the model object, not graph completion.

Runner-only roots such as prior_sensitivity, ai_advisor, diagnostics, and validation stay on the pipeline context and are stripped before the public MMM builder validates the model YAML.

Stage order

This is the canonical stage sequence. The runner uses a fixed stage list; optional stages retain their place and record a skipped status when disabled. Stage 80 inventories evidence and stage statuses. It does not generate automated business conclusions or certify decision readiness.

The prior-sensitivity stage is marked skipped when the YAML config does not contain prior_sensitivity or it is disabled. The AI advisor stage follows the same convention for ai_advisor. The diagnostics advisor stage runs by default for enabled ai_advisor blocks and is marked skipped only when ai_advisor is absent, disabled, or has diagnostics_review_enabled: false. The validation stage is marked skipped when the YAML config does not contain validation or it is disabled. The optimisation stage is also optional; it returns None and is marked skipped when the YAML config does not contain an optimization block.

See Output Directory Schema for the stage folders and artefact layout.

Data and model assumptions

The retained runner is designed around PanelMMM.

• The flow-oriented public YAML is expected to describe a PanelMMM.
• The data loader reads CSV only.
• Later stages call PanelMMM plotting, summary, diagnostics, and optimisation methods directly.

If you need the exact YAML keys, see YAML Configuration.

PipelineRunConfig

PipelineRunConfig controls runtime settings that sit outside the YAML model specification.

The draws, tune, chains and cores overrides do not necessarily bound Stage 35: explicit validation.sampler values take precedence for the refit. Use the bounded software smoke for an execution check with validation explicitly skipped.

Only sampler settings are merged into model construction. Other overrides are used by the runner itself during data loading, holiday resolution, diagnostics reporting, and output setup.

Run directory naming

The runner creates the run directory as:

<output_dir>/<effective_run_name>_<YYYYMMDD_HHMMSS>_<random_suffix>

The timestamp is generated in UTC. Each invocation exclusively allocates a new run directory, including simultaneous invocations with the same name and timestamp. The random suffix is opaque; use PipelineRunResult.run_dir and manifest_path instead of reconstructing paths from the name and timestamp. Existing runs are never reused or resumed by this allocator.

A run name must be a non-empty filename component, without path separators. Set output_dir to choose its parent. Allocated run directories have owner-only permissions on POSIX (0700); manifest files have mode 0600.

All stage directories are created up front, even if a later stage is skipped or the run aborts. An allocation or stage-directory setup error propagates; a partially initialised new run directory may remain for inspection.

Manifest updates are published by replacing the previous file with a completed temporary file in the same directory. Readers see a complete old or new snapshot. A publication failure propagates and leaves the previous snapshot intact, which may therefore be stale. This does not make stage artefacts transactional or provide power-loss durability.

Failure and skip behaviour

If a stage raises an exception:

• the current stage is marked failed
• the run manifest is marked failed
• all still-pending later stages are marked not_reached
• run_pipeline(...) re-raises the exception

If a stage returns None:

• the stage is marked skipped
• the manifest warning records that no configuration was supplied for that optional stage

Reporter hook

run_pipeline(...) accepts an optional reporter that implements the PipelineReporter protocol.

The reporter can observe:

• pipeline start
• stage start
• stage end
• pipeline end
• pipeline failure

See Extending the Runner for the callback contract.