Overview
Invariant is test and evaluation infrastructure for physical AI. It runs your robot software against recorded logs, simulated scenarios and hardware rigs, and tells you whether each change made things better, worse, or whether there isn't enough evidence yet.
What Invariant does
Robot software is hard to test. Results depend on timing, physics and sensor noise, so the same build can pass and fail on the same scenario. Most teams end up with a mix of replay scripts, simulator launch files and dashboards that one engineer keeps alive. Invariant replaces that with one system:
- A test spec in your repo (
invariant.yaml) that says what to run, on what, how to score it and what counts as a regression. - Runners that execute the spec the same way on a laptop, in CI, on cloud workers or on a hardware rig.
- Statistics that compare paired runs of two versions and return better, worse, unchanged or inconclusive for each scenario and metric.
- A run history that records every input, version and result, so any run can be reproduced and audited.
How a test flows
You commit a spec. A run is started from the CLI, a pull request or a schedule. The scheduler resolves the build (a container image), the baseline (usually main) and the list of scenarios, then shards the work across runners. Each run writes an MCAP file and metric values. The statistics engine pairs runs of the two versions by scenario and seed, and the result is posted wherever you asked: terminal, pull request, chat or report.
Who it's for
| Team | Typical first test | Typical metrics |
|---|---|---|
| Autonomy and localization | Replay recorded drives through the localization stack | pose RMSE, max error, divergence events, goals reached, clearance |
| Perception and computer vision | Score a new model on labeled scenes, sliced by condition | recall, precision, mean IoU, per-class and low-light recall, latency |
| Manipulation and learned policies | Hundreds of simulated trials per checkpoint | success rate per object, drops, collisions, cycle time |