Verdicts and statistics

Robot tests are noisy. Invariant compares paired runs of two versions, puts an interval on the difference, and only calls something better or worse when the evidence supports it.

Why pass/fail isn't enough

The same build on the same scenario can give different results. Thread scheduling, message timing, GPU physics and sensor noise all vary from run to run. Even simulators that support deterministic mode only promise it under narrow conditions: Isaac Lab, for example, documents reproducibility only on the same hardware and software versions (Isaac Lab docs). In a study using two open-source autonomous-driving simulators, between 1% and 32% of generated tests flipped between pass and fail on rerun, depending on the test setup (arXiv 2311.18768).

A single pass/fail threshold on one run therefore produces both false alarms, which teach teams to ignore the check, and missed regressions, which reach the field.

Paired runs MVP

For each scenario, Invariant runs the candidate and the baseline with the same seed and the same sampled parameters, and computes the difference per pair. Pairing removes most of the variation caused by the scenario itself, so far fewer runs are needed than comparing two independent averages.

Intervals and tolerances MVP

Invariant computes a 95% confidence interval on the mean paired difference (a t-interval for small samples, a bootstrap that keeps pairs together for larger ones, and paired methods for success rates; every report names the method and sample size). Each metric has a tolerance: the smallest change that matters to you. Each metric also has a direction (better: lower for errors, better: higher for clearance or success rate), with sensible defaults for built-in metrics. Reports show the raw change, and plots put the harmful direction on the right. The verdict comes from where the interval sits relative to the tolerance band:

Adding runs when it's too close to call Next

When a result is inconclusive and the suite allows it (runs.max), Invariant adds pairs in batches until the interval clears the tolerance band or the limit is reached. It uses sequential methods designed for repeated looks at the data, so checking after every batch doesn't inflate the false-alarm rate. Clear cases stop early, which is where most of the compute savings come from.

Many scenarios, many metrics Next

A suite with 60 scenarios and 4 metrics makes 240 comparisons. If nothing had really changed, unadjusted 5% tests would still flag about 12 of them by chance. Invariant adjusts regression decisions across the suite (Benjamini–Hochberg by default), which limits the expected share of flagged results that are false alarms. Reports label per-metric intervals and suite-adjusted decisions separately.

Rare failures and zero-failure results

Zero failures doesn't mean zero risk. If a scenario has 0 failures in n independent runs, the 95% upper bound on its failure rate is about 3/n (the "rule of three"). 0 collisions in 100 runs still allows a collision rate up to about 3%. Invariant shows that upper bound next to every zero-failure result.

How many runs

invariant plan pr estimates run counts and cost for a suite from its recent variance before you run it Next.

Flaky scenarios Next

Nightly runs repeat each scenario on the same version, which measures how much it disagrees with itself. A scenario whose self-disagreement is above a threshold is marked flaky. A flaky label doesn't switch off a gate. Moving a scenario to warn-only is an explicit, logged decision with an owner and an expiry date, and hard limits (collisions, crashes, timeouts) keep blocking unless the project overrides them.

Every result, one label