Core concepts
Seven nouns cover almost everything in Invariant: system, scenario, suite, run, metric, baseline and verdict.
System
The software under test: an autonomy stack, a perception model, a manipulation policy, or all three. Invariant sees it as a container image plus an adapter that knows how to start it, feed it inputs and collect outputs. Each commit you test resolves to an image digest, so a run always points at exactly the code that ran.
Scenario
One repeatable situation to test. There are three kinds:
| Kind | What it is | Good for | Status |
|---|---|---|---|
replay | A time slice of a recorded log, played into your stack | Localization, perception, regression tests from field failures | MVP |
sim | A world in a simulator, plus parameters such as poses, lighting, sensor noise and seed | Closed-loop navigation, manipulation, rare and dangerous cases | Next |
rig | A procedure on a hardware-in-the-loop bench or test robot | Timing, drivers, compute load, release sign-off | Later |
Scenarios are versioned. Changing a world file, a log slice or a parameter range creates a new scenario version, and results are only compared across the same version.
Suite
A named set of scenarios, metrics, run counts and a gate. Most teams have four that share one spec:
| Suite | When it runs | Typical size | Goal |
|---|---|---|---|
laptop | Before you push | 3–5 scenarios, minutes | Catch obvious breaks fast |
pr | Every pull request | 10–30 scenarios, under 15 min | Block real regressions |
nightly | Every night on main | 50–200 scenarios | Find slow drift, find the first bad commit |
release | Before shipping | Everything, plus rigs | Evidence for a release decision |
Run
One execution of one system version on one scenario with one seed. A run produces an MCAP file, metric values, logs and a run record: image digest, scenario version, seed, simulator version, adapter version, runner type, hardware, start and end time. The run record is what makes a result reproducible and auditable.
Every run ends in exactly one state: completed, failed (your software crashed or timed out) or error (the infrastructure failed: no GPU, image pull failed, simulator didn't start). Infrastructure errors are retried. They don't count as a regression, and a suite with unresolved errors is reported as incomplete rather than passed.
Metric
A number computed from a run, with a unit, a direction (lower or higher is better) and a tolerance: the smallest change you care about. Invariant ships metric packs for localization, navigation, perception and manipulation, and you can write your own in Python. See metrics in the spec.
Baseline
The version you compare against, usually main or the last release. If the baseline was already run on the same scenarios and seeds (for example by last night's suite), those runs are reused instead of re-run.
Verdict
The answer for one scenario and metric, from paired runs of the candidate and the baseline:
| Verdict | Meaning |
|---|---|
| worse | The whole confidence interval is past the tolerance on the bad side |
| better | The whole interval is past the tolerance on the good side |
| unchanged | The whole interval is inside the tolerance |
| inconclusive | The interval crosses a tolerance edge: not enough evidence yet |
A scenario can also be marked flaky if the same version disagrees with itself too often. A suite's gate decides which verdicts block a merge. Details: Verdicts and statistics.