Skip to content

KITTI Camera-LiDAR Falsification

This page publishes Calibrex's full-scale KITTI camera-LiDAR falsification benchmark as a maintained artifact: kitti-camera-lidar-falsification.json (slac.kitti_lidar_camera_falsification/v0.1, schema-valid). One frozen protocol scores two candidates for T_camera_lidar on the official KITTI raw drive 2011_09_26_drive_0005_sync:

  • dataset_reference: KITTI's own calib_velo_to_cam.txt, which must pass;
  • known_bad: the same transform turned +5 deg about the LiDAR's vertical axis and moved +0.25 m along its forward axis (composed on the LiDAR side, so the camera sees a turn about its vertical axis and a shift along its optical axis), declared before scoring, which must fail.

Each trial is a full calibrex calibrate run whose evidence bundle (result, evidence, policy, assessment, observability, protocol, transforms, bundle and verification files) is bound by SHA-256 in the artifact, as are the 50 camera images and Velodyne scans and the KITTI calibration files. The data stay outside the repository (KITTI raw, CC BY-NC-SA 3.0).

Result

Status: pass: dataset reference passed and declared known-bad candidate failed.

  • Sequence: 2011_09_26_drive_0005_sync, 50 camera-LiDAR frame pairs, seed 20260729, holdout ratio 0.2, 4000 projected points per frame.
  • Protocol: kitti_lidar_camera_projection_edge_holdout/v0.1.
  • Known-bad candidate: right-composed camera-frame +5 deg yaw and +0.25 m x translation; declared before scoring.
  • Generated by Calibrex 0.5.1 at commit 5455dd6, 2026-10-12.
dataset_reference known_bad
expected pass fail
assessment pass fail
evidence bundle verified yes yes
lidar_camera_edge_alignment_score 0.3111 train / 0.3813 holdout 0.3061 train / 0.3895 holdout
lidar_camera_depth_edge_alignment_score 0.6234 train / 0.7442 holdout 0.5154 train / 0.7222 holdout
lidar_camera_projection_ratio 0.1697 train / 0.1688 holdout 0.1779 train / 0.1770 holdout
lidar_camera_projection_frame_count 50 50
lidar_camera_perturbation_case_count 1200 1200
lidar_camera_perturbation_detectable_fraction 0.8550 0.8133
lidar_camera_perturbation_mandatory_case_count 600 600
lidar_camera_perturbation_mandatory_detectable_count 508 490

Policy gates:

Gate dataset_reference known_bad
raw_recomputation pass pass
protocol_declared pass pass
holdout_support_gate pass pass
holdout_independence pass pass
dataset_reference_consistency pass pass
known_bad_controls pass pass
decision_boundary_summary pass fail

LiDAR-camera evidence checks (the decision-boundary gate combines the others):

Check dataset_reference known_bad
Candidate Support pass pass
Holdout Edge Alignment pass pass
Dataset Reference Consistency (compares with the dataset's calibration) pass fail
Known-Bad Controls pass pass
Observability Statement pass pass
Decision Boundary pass fail

dataset_reference: all 7 falsification policy gates passed.

known_bad: 1 falsification policy gate(s) failed; 0 inconclusive.

The protocol rejects known_bad only through Dataset Reference Consistency (translation delta 0.25 m (threshold <= 0.1), rotation delta 5 deg (threshold <= 2)): a comparison with the dataset's own calibration. None of the protocol's checks of the images and scans rejects it.

Data-only status: pass: the images and scans alone, judged with the calibrex check per-axis rule (error / tolerance): rotation in deg by edge alignment, z (along the optical axis) in m by stereo depth; x and y are not judged.

Frames dataset_reference known_bad Decision
50 spread over the drive pass (roll 0.10 / 0.50, pitch 0.05 / 0.50, yaw 0.10 / 0.50, z 0.050 / 0.053 m) fail (roll 0.05 / 0.50, pitch 4.90 / 0.50 fail, yaw 0.00 / 0.50, z 0.257 / 0.076 m fail) pass
first 50 (the protocol's) pass (roll 0.05 / 0.50, pitch 0.05 / 0.50, yaw 0.40 / 0.50, z 0.022 / 0.040 m) fail (roll 0.00 / 0.50, pitch 4.90 / 0.50 fail, yaw 0.35 / 0.50, z 0.264 / 0.029 m fail) pass

The same decisions on other drives recorded with the same calibration files:

Drive, frames dataset_reference known_bad Decision
2011_09_26_drive_0009_sync, spread 50 pass (roll 0.15 / 0.50, pitch 0.05 / 0.50, yaw 0.05 / 0.50, z 0.005 / 0.100 m) fail (roll 0.20 / 0.50, pitch 4.90 / 0.50 fail, yaw 0.00 / 0.50, z 0.234 / 0.086 m fail) pass
2011_09_26_drive_0009_sync, first 50 pass (roll 0.05 / 0.50, pitch 0.25 / 0.50, yaw 0.05 / 0.50, z 0.009 / 0.049 m) pass (roll 0.10 / 0.50, yaw 0.00 / 0.50; unchecked pitch, z) inconclusive

Cross-check, not judged: the joint six-DoF stereo fit's translation readings (candidate minus estimate, m, with jackknife std; untrusted where its own control or std rejects the axis):

Drive, frames dataset_reference known_bad
2011_09_26_drive_0005_sync, spread x +0.040 ± 0.022, y +0.026 ± 0.038 untrusted, z +0.014 ± 0.042 x +0.044 ± 0.026, y +0.048 ± 0.033 untrusted, z +0.262 ± 0.041
2011_09_26_drive_0005_sync, first x -0.027 ± 0.019, y +0.072 ± 0.014 untrusted, z +0.016 ± 0.036 x -0.027 ± 0.018, y +0.082 ± 0.013 untrusted, z +0.278 ± 0.037
2011_09_26_drive_0009_sync, spread x +0.053 ± 0.019, y +0.030 ± 0.014, z -0.038 ± 0.021 x +0.055 ± 0.021, y +0.028 ± 0.010, z +0.213 ± 0.019
2011_09_26_drive_0009_sync, first x +0.040 ± 0.012, y +0.049 ± 0.011, z -0.025 ± 0.051 untrusted x +0.039 ± 0.013, y +0.057 ± 0.006 untrusted, z +0.219 ± 0.068 untrusted

What the protocol's pass does and does not show

The protocol passes, but none of its checks of the images and scans rejects the known-bad candidate. The candidate is turned 5 deg and moved 25 cm. On the held-out frames it scores as well as the reference: an edge alignment of 0.39 against 0.38, and a depth-edge alignment of 0.72 against 0.74. Every check of the images and scans passes it, including support, held-out edge alignment and the 1200 perturbation controls.

It fails one check only, Dataset Reference Consistency, which compares the candidate with KITTI's calibration: it is 0.25 m and 5 deg away, beyond 0.1 m and 2 deg. The decision-boundary gate counts that comparison, so the known-bad fails. Under this protocol, the falsification therefore rests on knowing the reference, not on the data. The published test (tests/integration/test_kitti_falsification_published.py) pins this finding, so a protocol change that alters it also has to update this page.

The data-only decision

The artifact also carries a decision that never looks at the reference: the Data-only status above (calibrex.evaluation.kitti_edge_falsification). Each candidate is judged with the per-axis rule of the bag-level camera-LiDAR check: the tolerance is the larger of 3 jackknife standard deviations and a floor (0.5 deg, 0.02 m), and an error beyond twice the tolerance is a fail. The decision passes when the reference passes and the known-bad fails.

  • Rotation comes from the check's edge alignment, started at the candidate, pooled over the frames with a block jackknife and held-out known-bad controls.
  • Translation along the optical axis (z) comes from KITTI's stereo pair, at the rotation the edges estimate (calibrex.evaluation.stereo_lidar_depth). LiDAR points on surfaces that face the camera are block-matched between the rectified images of cameras 2 and 3. The disparity residual is fitted as a constant offset of the rig plus the candidate's depth error, which fades as 1 / Z^2; KITTI's rig reads -0.6 to -0.8 px, absorbed by the fit. A control moves the candidate 0.1 m along the axis and requires the estimate to follow (between half and one and a half of it); on this drive it follows by 0.6 to 1.05. This uses the stereo pair's own calibration, never the camera-LiDAR one.
  • The order. Edges see rotation at a translation. When the stereo depth is trusted, the edges run again from the candidate with that error along the optical axis removed, and judge rotation there. The depth is then measured again at their new rotation. Rotation is still judged as the candidate's error, and the trial records the shift (edge_start_depth_shift_m).
  • x and y are not judged.

Edges cannot judge z here. In a development run with translation searched, the edge estimate of z ran to its 0.2 m search bound from the reference, and the 5 cm held-out translation controls were not detected. The reason is the street itself: moving the candidate along the optical axis keeps a road point on the road and a side-wall point on its wall, so the shifted cloud still lines up. The same holds for stereo depth when every point is used: a 0.25 m error then read as about 0.06 m. Points on camera-facing surfaces (car backs, building fronts; about one in eight matched points) do not have that freedom; on them, offsets of -0.10, +0.10 and +0.25 m added to the reference read as -0.09, +0.09 and +0.21 m in a development run.

The images and scans alone fail the known-bad on both axes it is wrong in, on both frame selections. Its 5 deg turn reads as a 4.9 deg pitch error against a 0.5 deg tolerance (the check names axes in the camera's optical frame, where the vertical axis is pitch), and its 0.25 m along the optical axis reads as 0.26 m against 0.029 to 0.076 m.

They pass the reference on both frame selections, so the data-only status is pass. On the spread frames it passes close to the tolerance: stereo depth puts it 0.050 m along the optical axis against 0.053 m.

The order (depth first, then rotation) was added to let the edges see the known-bad's pitch on drive 0009 (below). Before it, the edges judged rotation at the candidate's translation, and on 0005 the reference read warn twice:

  • on the spread frames, z at 0.0504 m against 0.0496 m;
  • on the first 50 frames (5 s of consecutive driving), yaw at 0.50 deg against 0.50 deg.

The data-only status was then inconclusive. With the order, the readings barely move (z 0.050 m, yaw 0.40 deg), but both now fall inside the tolerance. On the first 50 frames the jackknife's yaw standard deviation stays small (blocks a second apart are not independent), so yaw is judged against the 0.5 deg floor.

Drive 0009: the same calibration on another drive

Drive 2011_09_26_drive_0009_sync was recorded on the same day with the same calibration files (the artifact binds their SHA-256), so its reference and known-bad are the same transforms. Repeating the decisions there (the second table above, --replicate) separates what belongs to the calibration from what belongs to a drive.

  • The 5 cm reading along the optical axis belongs to drive 0005, not to the calibration. On 0009 stereo depth puts the reference under 1 cm along the optical axis on both frame selections. On 0005 it reads 5 cm on both 50 and, in a development run, 100 spread frames. The most likely cause is a scene-dependent bias of the stereo estimate on 0005, not an error in KITTI's calibration.
  • On the spread frames the decision passes. Every rotation axis of the reference is judged and passes. The known-bad fails on its 0.25 m along the optical axis and on its pitch. When the edges started at its translation, its pitch control reached t = 1.99, just under 2. Started with the 0.25 m removed, it reaches 2.2.
  • On 0009's first 50 frames (5 s) the data do not see the known-bad's error. Roll and yaw are judged and pass (they are right). Pitch is not judged, so its 5 deg turn is not corrected, and stereo depth at that rotation is not trusted either (jackknife std 0.084 m), so the edges are not restarted. The trial's verdict is what calibrex check would report: pass, with pitch and z unchecked. The decision is inconclusive: the protocol declares the axes the known-bad is perturbed on (pitch and z in the camera's frame), and none of them is judged.

A first replication read 0009 through a pairing bug: KITTI's timestamps.txt keeps a blank line for each missing scan (0009 has none for frames 177-180), and the reader took each scan's file by its position in the directory, so every pair after the gap loaded the scan 0.4 s late. With most frames misaligned, the edges judged no rotation on 0009. Files are now taken by frame number, and pairs more than 60 ms apart (the images whose own scan is missing) are not used.

Translation across the optical axis: measured, not judged

Edges and camera-facing surfaces leave x and y unjudged. A joint fit (estimate_stereo_alignment) uses LiDAR points on planar surfaces of any facing. House fronts along the street see x, the road sees y. A candidate that puts a point d away from its surface (normal n) reads a disparity residual base * (n . d) / (Z (n . p)), linear in a translation error and a small rotation error, so all six are fitted together with the rig's offset. Each translation axis has a moved-candidate control. On a rendered street it recovers x, z and rotation; y, which rests on the near road only, is trusted only when its control follows.

On KITTI it is recorded on every data-only trial as a cross-check (the last table above) and not judged, because its readings of the same calibration do not agree:

  • x varies more than its uncertainty says. The reference reads x from -2.7 cm (0005, first 50) to +5.3 cm (0009, spread), while each run's jackknife std is 1 to 3 cm. Three runs agree within 1.3 cm (+4.0 to +5.3 cm). The fourth is 5 s of a slow turn (3.3 m/s, 7 deg/s), whose stereo rotation also reads oddly (pitch +0.67 deg in a development run). Judged with the check's rule, the earlier spread warned the known-bad on x, which it is right in (0009, first 50: 2.7 cm against 2.2 cm).
  • What was ruled out and what helped (development runs).
  • The LiDAR sweep: ruled out. A KITTI scan takes 0.1 s, and the camera fires when the LiDAR faces forward. Moving each point to the image time with the OXTS velocity and yaw rate, either way round, left the spread at 11 to 12 cm.
  • The stereo rig's imperfections: helped. Fitting the rig's disparity scale and a roll between its cameras beside its offset brought the spread from 11.4 cm to 8 cm. On the road both look like a height error, and on one-sided streets like a lateral one. The fit now includes them (rig_terms).
  • y is untrusted on three of the four runs. Its control moves the estimate by a quarter to a half of the move. On the road, y trades off against a roll of the stereo rig.
  • Its z agrees with the judged one. It reads the known-bad 0.22 to 0.28 m along the optical axis.

KITTI's calibration is a reference, not ground truth. These readings do not show it wrong. The published tests pin them, so a change to either estimator that moves them also has to update this page.

What was decided after seeing results:

  • The frame selection. Spread frames are the check's own policy and set the status; the first 50 are a sensitivity row. Both were run before one was made primary.
  • The order: depth first, then rotation. It was added for the known-bad's pitch on 0009, and it also moved two borderline reference readings on 0005 inside the tolerance (above).
  • The occlusion filter of the stereo estimate. A point with a nearer LiDAR return in its matching patch is dropped: it is hidden from the camera, which sits 0.27 m in front of the LiDAR, or straddles a depth edge. It was added after a first run in which the reference read 0.041 m against 0.052 m (pass); with it, the reference reads warn. The rule and its floors are the check's, unchanged.

The decision rule for a known-bad that passes: it is fail (the method passed a wrong candidate) when at least one of its declared axes was judged, and inconclusive (the data did not look where it is wrong) when none was. The declared axes come from the protocol's declaration of the known-bad, never from the data. This distinction was added after 0009's first 50 frames read fail under the earlier rule.

The same decision on a second dataset, a rectified fisheye pair, is reported in Hilti fisheye stereo falsification.

The protocol's own status is unchanged: the data-only decision is reported beside it, not folded into it, so the protocol's finding above stands.

Reproduce

python tools/publish_kitti_falsification_benchmark.py \
    --dataset data/public/kitti_raw/2011_09_26/2011_09_26_drive_0005_sync \
    --output-dir outputs/kitti-falsification \
    --replicate data/public/kitti_raw/2011_09_26/2011_09_26_drive_0009_sync

This runs both trials (about 40 minutes on 8 cores) and rewrites the two files under docs/assets; --from-output DIR republishes a finished run, re-hashing every trial file first. The opt-in test test_official_kitti_falsification_benchmark (@pytest.mark.kitti, CALIBREX_KITTI_RAW_0005) runs the same benchmark; tests/integration/test_kitti_falsification_published.py checks the committed artifact on every CI run: schema-valid, reference pass, known-bad fail, every trial file digest-bound with a relative path, both data-only rows (the known-bad failing on pitch and z, the reference's warn readings), the 0009 replication, and the Markdown above regenerated byte for byte. Publishing a run made before the data-only decision existed adds it (about four minutes); each decision records its own git commit.