Calibration evidence framework for robotics

Estimate it, check it, watch it drift.¶

Three commands for the sensor calibration of a robot, on ROS 2 and ROS 1 bags, with honest per-axis verdicts for LiDAR, IMU, camera, GNSS, and vehicle. Calibrex says plainly which axes it measured, which it could not judge, and what to record to fix that.

The workflow Run it in the browser Install GitHub

Terminal recording: calibrex estimate on a bag with no calibration (rotation observed, lever arm marked NOT MEASURED), calibrex check of the exported frames.yaml on another recording (pass on the three judged axes), and calibrex drift over three recordings flagging the one whose IMU was remounted

Real runs on the Koide hard-localization recordings (abridged output): estimate a calibration from one bag, check it on another, then ask whether it changed across recordings. The third recording is a copy with its IMU rotated 2° about z.

The workflow¶

	Command	Question	Result
1	`calibrex estimate <bag>`	The bag has no calibration yet: what do the data say?	`frames.yaml`, static transforms, URDF joints, a Kalibr camchain; per-axis std; axes the data did not observe are marked NOT MEASURED; `--html` report
2	`calibrex check <bag2> --tf frames.yaml`	Is the deployed calibration still right on another recording?	`pass` / `warn` / `fail` / `inconclusive` per axis, what could not be judged, next steps, progress; `--html` report
3	`calibrex drift <bag1> <bag2> ...`	Did the calibration change over time, and in which bag?	`stable` / `drift` / `inconclusive` per axis, the deviating bag and the size of the change

When a lever-arm axis is not observable, estimate and check say why (which rotation the recording lacks) and how much motion or duration would fix it (translation observability). The same check and estimate run in the browser page on your own bag, with no install and no upload. Details and real outputs: the workflow page.

What it does¶

One command, ten sensor pairs

calibrex check reads /tf_static (or the frames.yaml that calibrex estimate wrote) and checks every sensor pair it finds, from LiDAR-vehicle to camera-IMU. camera-LiDAR is judged on rotation only; translation is never judged.
Reads your bag as it is

rosbag2 (.db3), ROS 2 MCAP (zstd/lz4) and ROS 1 .bag (bz2/lz4) are detected from the file: no conversion, no ROS install. Autoware's raw Velodyne packets are decoded too (Autoware note).
Honest per-axis verdicts

Partial coverage is named, not hidden: a pass covers only the judged axes, and detection power tells you whether the data could have caught a wrong calibration.
Check or estimate in your browser

Drop a ROS 2 bag on the bag check page to plan, run calibrex check or run calibrex estimate under Pyodide, with progress and downloads. Files are read in place and never uploaded. Compared with the command line on real bags of every pair: same verdicts (camera pairs agree within one to two sigma).
Pre-registered SOTA claims

Thresholds are frozen before held-out data is scored, and claims that fail stay on the leaderboard.
Evidence and provenance

Configs and results are schema-validated, every report records its inputs, and the README GIFs are digest-bound to their sources.

Catching a bad calibration¶

calibrex check catches a bad calibration and confirms the good one on KITTI: with a 3 degree yaw error the LiDAR points miss the bollards and lidar-vehicle fails; restoring the vendor calibration file puts them back on the bollards and the re-check passes

Real calibrex check runs on KITTI development drives: a deployed LiDAR transform with a 3° yaw error fails; restoring the calibration file and re-checking passes (pitch and yaw judged; roll is not observable from driving).

Camera ↔ LiDAR, rotation only¶

calibrex check --pairs camera-lidar on KITTI drive 0005: LiDAR points projected through the vendor calibration land on the cyclist and bollards and the pair passes; turned 3 degrees about the camera y axis they miss and the pair fails on pitch

Two real calibrex check --pairs camera-lidar runs on KITTI development drive 0005: the vendor calibration passes (roll and pitch judged); the same calibration turned +3° about the camera y axis fails (pitch 3.00° against 0.50°). The frames in between only slide the overlay. Translation is not judged; results and limits.

Real data, rendered by the code¶

Every frame comes from a real recording, rendered by Calibrex's own code.

Two Ouster LiDARs on NTU VIRAL: the second LiDAR scans snap onto the first LiDAR map — **LiDAR ↔ LiDAR snaps into focus** (NTU VIRAL tnp_01). From a labelled perturbed start to the `calibrex lidar-lidar` estimate; median point-to-plane residual 19.1 → 4.4 cm (the design value gives 4.6 cm).

A hand-held Livox MID360 sweep before and after gyro deskewing — **Gyro deskew** (RTK-SLAM, hand-held MID360. The fastest sweep of the run, 180 deg/s: raw vs deskewed with Calibrex's own IMU-LiDAR estimate; local surface thickness 3.42 → 2.19 cm.)

Gyro and camera rate curves lock together as the calibration converges — **Camera ↔ IMU curves lock together** (Hilti 2022 exp21, development data. Rate RMSE 15.8 → 6.2 deg/s (the floor is camera-rate noise); the estimate is 0.45° and −0.21 ms from Kalibr's target-based calibration.)

The KITTI rig frame tree turning red as calibrex check fails lidar-vehicle — **The rig under a yaw error** (KITTI `/tf_static`. Rotation drawn ×8 for visibility; colours follow the real `calibrex check` verdict at each step.)

SOTA standings¶

Calibrex calls a method state of the art only for a claim that a frozen, pre-registered audit marks supported. Refuted audits are listed too.

Pair	Standing	Notes
`imu-lidar`	supported	Livox MID360 against LI-Init: rotation and clock offset, and the full extrinsic in a second round (the first round was refuted)
`lidar-vehicle`	supported	KITTI motion-only rotation against calib_imu_to_velo, eight unseen drives
`camera-imu`	supported	Hilti 2022 forward cameras against Kalibr: every judged axis within 1 deg, five unseen recordings (third round; the first two were refuted)
`lidar-lidar`	refuted	NTU VIRAL, 2/4 gates
`camera-camera`	refuted	Hilti 2022 against Kalibr, 4/6 gates
`camera-lidar`	refuted	KITTI check verdicts: v2 8/9 gates on 18 unseen drives of four days (no calibration flagged; one turned-run axis 1.503 deg off, gate 1.5), v1 5/9
`ins-lidar`	refuted	KITTI + Boreas, 6/8 gates: estimated axes agree, but one recording has no observable rotation
`lidar-wheel_odometry`	refuted	NCLT, five unseen sessions: yaw within 0.43 deg, roll and pitch up to 2.4 deg off (gate 0.5); check never failed the calibration
`camera`	supported	TUM-VI focal lengths, five unseen sequences: check never failed Kalibr's values and passed none of the 3 %-wrong ones (half inconclusive; no accuracy claim)
`gnss-imu`, `imu-vehicle`	no_claim	Native method, no audited claim yet

Full leaderboard and protocols

Quickstart¶

python -m pip install "https://github.com/rsasaki0109/Calibrex/releases/download/v0.6.2/calibrex-0.6.2-py3-none-any.whl"
calibrex estimate my_bag/ --output est/ --html est.html     # no calibration yet
calibrex check other_bag/ --tf est/frames.yaml --html check.html
calibrex drift day1/ day2/ day3/ --output drift/

See the workflow, Install and first check, or the check tutorial for the KITTI yaw-injection demo.

Hand-eye benchmark¶

Five AX=YB methods were recomputed on the ETHZ ASL real robot-arm dataset with one shared 80/20 split.

Method	Rotation holdout RMSE deg ↓	Translation holdout RMSE mm ↓	Known-bad detection fraction ↑	Failure rate	Runtime s
Shah	0.585795	10.0628	1	0.0%	—
Li-Wang-Wu	0.58728	17.458	1	0.0%	—
Dornaika-Horaud	0.585795	10.0628	1	0.0%	—
Zhuang-Roth-Sudhakar	0.590747	10.132	1	0.0%	—
Calibrex nonlinear refinement	0.58721	10.0394	1	0.0%	—

Calibrex's nonlinear refinement is best on translation in this benchmark; Shah is marginally best on rotation. These are held-out closure errors, not ground-truth extrinsic errors. Read the benchmark protocol and provenance.

Integration paths¶

Current status¶

Calibrex is alpha research software. Failed controls are reported rather than hidden; see the changelog and development roadmap for current limitations. Read the frame conventions before integrating transforms.