Docs / Concepts

invariant.yaml reference

The test spec lives in your repo and is reviewed like code. One file can define several suites that share systems, scenarios and metrics.

Design preview. These docs describe how Invariant is being built. Commands, output and limits are illustrative until release, and stage tags mark what ships first. See the roadmap.

Full example

invariant.yamlversion: 1

systems:
  nav:
    image: ghcr.io/acme/nav_stack:${GIT_SHA}
    adapter: ros2
    ros2:
      distro: jazzy
      launch: nav_stack bringup.launch.py
      args: { use_sim_time: true }
      record: [/localization/pose, /cmd_vel, /tf]
      ready: { topic: /localization/pose, timeout_s: 30 }
    resources: cpu-8

scenarios:
  dock_replays:
    replay: s3://acme-logs/dock/*.mcap
    ground_truth: /gt/pose
  aisles:
    sim: gazebo
    worlds: [narrow_aisle, corridor_loop, pallet_row]
    goals: worlds/goals.yaml
    vary:
      lighting: { uniform: [0.2, 1.0] }   # sampled range
      lidar_dropout: { uniform: [0, 0.05] }
      start_pose: { jitter_m: 0.3, jitter_yaw_deg: 10 }
    resources: gpu-l4

metrics:
  - localization_rmse: { tolerance: 0.005 }      # m
  - min_clearance:     { tolerance: 0.02 }       # m
  - time_to_goal:      { tolerance: 2.0 }        # s
  - goals_reached:     { limit: { min: 9 } }     # every goal must be reached
  - custom: metrics/dock_alignment.py

suites:
  laptop:  { system: nav, scenarios: [dock_replays], limit: 3, runs: { paired: 3 } }
  pr:      { system: nav, scenarios: [dock_replays, aisles], runs: { paired: 10, max: 40 },
             gate: { block_on: [worse], warn_on: [inconclusive] }, budget: { gpu_hours: 2 } }
  nightly: { system: nav, scenarios: all, runs: { paired: 20, max: 80 }, schedule: "0 2 * * *" }

Short form

A file with one suite can skip the named sections and use suite, system and an inline scenarios list, as in the Quickstart:

invariant.yamlsuite: pr
system:
  image: ghcr.io/acme/nav_stack:${GIT_SHA}
  adapter: ros2
scenarios:
  - replay: s3://acme-logs/dock/*.mcap
  - sim: gazebo
    worlds: [narrow_aisle, corridor_loop]
metrics:
  - localization_rmse: { tolerance: 0.005 }
runs: { paired: 10, max: 40 }
gate: { block_on: worse }

systems

Key	Description
`image`	Container image to test. `${GIT_SHA}` and `${BRANCH}` are filled in per run. Resolved to a digest before any run starts.
`build`	Alternative to `image`: a Dockerfile and context to build per commit, with layer caching.
`adapter`	`ros2`, `process` or `python`. See Connect your stack.
`resources`	Default resource profile, for example `cpu-8` or `gpu-l4`. See profiles.
`env`, `secrets`	Environment variables and named secrets made available to the container.

scenarios

Key	Description
`replay`	One or more logs (glob or list). Each file, or each `slice` within a file, becomes a scenario.
`slices`	Named time windows within a log: `{ name, start, end }`.
`ground_truth`	Topic or file used as truth for localization and perception metrics.
`sim`	Simulator adapter: `isaac_sim`, `gazebo`, `o3de`, `mujoco` or `container`.
`worlds`	World, level or scene files you provide. Invariant doesn't author worlds.
`vary`	Parameters to vary. A list such as `[0.2, 0.6, 1.0]` is a fixed sweep. A distribution such as `{ uniform: [0.2, 1.0] }` is sampled. The sampled value and seed are recorded for every pair.
`seeds`	Fixed seed list. If omitted, seeds are drawn and recorded so pairs are reproducible.

metrics

Built-in metric packs MVP for localization and perception. Navigation and manipulation packs are Next.

Pack	Metrics
localization	`localization_rmse`, `max_pose_error`, `relative_pose_error`, `divergence_events`, `time_to_relocalize`
navigation	`goals_reached`, `time_to_goal`, `min_clearance`, `path_length`, `collisions`, `interventions`
perception	`recall`, `precision`, `map`, `mean_iou`, `recall_by_class`, `latency_p95`, with `slice_by` tags such as `low_light`
manipulation	`success_rate`, `success_by_object`, `drops`, `collisions`, `cycle_time`

Each metric takes a tolerance in its own unit, and optionally an absolute limit that fails the run regardless of the baseline (for example collisions: { limit: 0 }).

Custom metrics

metrics/dock_alignment.pyfrom invariant import metric, Run

@metric(unit="m", lower_is_better=True, tolerance=0.01)
def dock_alignment_error(run: Run) -> float:
    final = run.topic("/localization/pose").last()
    dock = run.scenario.named_pose("dock")
    return final.position.distance_to(dock.position)

Custom metrics read the run's MCAP and scenario metadata, and run after the container exits, so they never slow down or change the system under test.

suites

Key	Description
`runs.paired`	Starting number of paired runs per scenario.
`runs.max`	Upper limit when Invariant adds runs to resolve inconclusive results Next.
`gate`	`block_on` and `warn_on` lists of verdicts. Default: block on `worse`, warn on `inconclusive` and `flaky`.
`budget`	Upper limits on `gpu_hours`, `cpu_hours` or wall-clock `minutes`. The suite stops scheduling new runs when a limit is reached and reports what it covered.
`schedule`	Cron expression for scheduled suites.
`limit`	Run only the first N scenarios. Useful for the `laptop` suite.

← PreviousCore concepts Next →Verdicts and statistics