Benchmarks

Performance of Shifty’s validate pipeline — inference plus validation — across real building models and tagged releases. The chart can also include a HEAD run from the current checkout.

The chart splits each validate run into setup, inference, and validation. Preparing a large shapes graph is a fixed cost paid even for a small data graph. Read the segments when comparing versions: a change in validation work may be small relative to setup in a fresh process.

Each bar below is therefore split into the three things the time is spent on.

Show the numbers

Per-model results

These tables show exact validate times for the latest measured version and its predecessor, which may be a HEAD checkout rather than a release. The % column flags regressions or improvements per model, and the geomean row summarises the overall change between the two versions.

Regenerating benchmark data

./benchmark/run_history.sh
uv run benchmark/process_results.py
cd docs && make html

run_history.sh benchmarks every release tag, then the current checkout as a final HEAD entry. Tagged results are reused when they already exist, so a repeat run only re-measures HEAD — but the first run measures every tag and takes hours. To refresh just the HEAD entry after a code change:

BENCH_ONLY_HEAD=1 ./benchmark/run_history.sh
cd docs && make html

HEAD is built from the working tree, so uncommitted changes are included; the run logs the commit it started from and whether the tree was dirty. BENCH_HEAD=0 restricts a run to tags only, and BENCH_ITERS controls how many samples each measurement takes (default 3, median reported).

Shapes and models always come from the current checkout. That means changing a fixture invalidates every previously recorded result — the old numbers were measured against a different input and are no longer comparable. Delete benchmark/results/v*/ and re-run the full history when that happens.

Reading these numbers

Two cautions, both learned the hard way.

A ratio is only meaningful next to its denominator. When an optimization speeds up the baseline more than the thing being measured, the reported ratio gets worse even though both arms got faster. Absolute times belong beside any ratio quoted from this corpus; Evidence performance study has a worked example of that happening.

These are per-process runs. Each model pays setup once because every measurement is a fresh process. That is the right model for the CLI and the wrong one for a library caller, who can amortize setup across many data graphs with PreparedValidator. The setup segment of each bar is roughly what that amortization is worth.

For the separate question of what evidence costs on top of validation, see Evidence performance.