Docs / Running tests

Logs and replay

Recorded data is the fastest way to start: no simulator choice, no worlds to build, and every field failure can become a regression test.

Design preview. These docs describe how Invariant is being built. Commands, output and limits are illustrative until release, and stage tags mark what ships first. See the roadmap.

Formats MVP

MCAP is native. rosbag2 recordings (MCAP or SQLite storage) are read directly. Other formats are converted to MCAP once, and the converted file is written to an output prefix you choose, so inputs stay read-only.

Where logs live MVP

Logs stay in your storage: S3, GCS, Azure Blob, an S3-compatible on-prem store, or a local path for laptop runs. The spec stores a reference and a content hash, never a copy. Runners read logs directly from your bucket with credentials you scope, and cache them on local disk so repeated runs don't pull the same file twice.

Slices MVP

Most logs are long and most of them are boring. A slice is a named time window that becomes its own scenario:

invariant.yamlscenarios:
  dock:
    replay: s3://acme-logs/2026-06-11/run_0611.mcap
    slices:
      - { name: dock_approach_02, start: "0:42", end: "2:10" }
      - { name: ramp_transition_05, start: "14:05", end: "15:30" }
    ground_truth: /gt/pose

Invariant indexes the log once, so slicing a 2-hour log doesn't mean reading 2 hours of data per run. Slices include a configurable warm-up before start so filters and maps converge before scoring begins.

Ground truth

Domain	Common sources
Localization	RTK GNSS, survey-grade total station, motion capture, or an offline best-estimate trajectory computed from the full log
Perception	Labeled frames (COCO-style JSON, or KITTI-style for 3D boxes), joined to the log by timestamp
Simulation	Exact poses, contacts and object states from the simulator itself

If ground truth is missing, comparison metrics still work: Invariant can score the candidate against the baseline's output (for example, pose difference between two builds), which catches changes but can't tell you which version is right.

From field failure to regression test MVP

invariant scenario add --from-log s3://acme-logs/2026-10-21/run_1021.mcap \
  --name lost_localization_bay3 --start 31:12 --end 32:40 --ground-truth /gt/pose

The scenario joins the suite you name, gets a baseline result on main, and from then on every change is checked against it. This is the most valuable test most teams can write: it covers something that already went wrong.

Viewing runs

Every run's output is a normal MCAP file in your bucket, so you can open it in the viewer you already use. Reports link directly to the file and the time of each failure. Invariant doesn't include its own 3D viewer.

← PreviousSimulators Next →Where tests run