Benchmarks

These benchmarks measure Shifty’s validate pipeline (inference and validation) on building models across tagged releases. The chart can also include a HEAD run from the current checkout.

Each bar separates setup, inference, and validation time. Preparing a large shapes graph takes time even when the data graph is small, so improvements to validation may have little effect on the total runtime.

Show the numbers

Per-model results

These tables show exact validate times for the latest measured version and its predecessor, which may be a HEAD checkout rather than a release. The % column flags regressions or improvements per model, and the geomean row summarises the overall change between the two versions.

Regenerating benchmark data

./benchmark/run_history.sh
uv run benchmark/process_results.py
cd docs && make html

run_history.sh benchmarks every release tag, then the current checkout as a final HEAD entry. Tagged results are reused when they already exist, so a repeat run only re-measures HEAD — but the first run measures every tag and takes hours. To refresh just the HEAD entry after a code change:

BENCH_ONLY_HEAD=1 ./benchmark/run_history.sh
cd docs && make html

HEAD is built from the working tree, so uncommitted changes are included; the run logs the commit it started from and whether the tree was dirty. BENCH_HEAD=0 restricts a run to tags only, and BENCH_ITERS controls how many samples each measurement takes (default 3, median reported).

Shapes and models always come from the current checkout. That means changing a fixture invalidates every previously recorded result — the old numbers were measured against a different input and are no longer comparable. Delete benchmark/results/v*/ and re-run the full history when that happens.

Reading these numbers

Compare absolute times alongside ratios. For example, if an optimization speeds up conformance checking more than evidence generation, the evidence cost as a multiple of conformance time increases even though both are faster.

Each measurement runs in a fresh process and includes setup for that model. This matches CLI usage. Library callers can reuse PreparedValidator across data graphs to avoid repeating setup; the setup segment estimates how much time this can save.

For the separate question of what evidence costs on top of validation, see Evidence performance.