ETHZ native and OpenCV hand-eye benchmarks¶

These benchmarks execute all 13 Calibrex native hand-eye methods and the seven OpenCV 4 hand-eye variants on common, leakage-safe splits of the ETHZ ASL real robot-arm pose streams. The two equations use different estimands, so Calibrex publishes and ranks them separately.

Hand-eye `AX=XB`¶

ETHZ real hand-eye AX=XB native and OpenCV shared-split benchmark¶

Method	Holdout rotation RMSE deg ↓	Holdout translation RMSE mm ↓	Failure rate	Runtime s
Calibrex Park-Martin	0.867227 [0.759962, 0.958155]	13.8759 [11.7678, 15.8266]	0.0%	0.153451
Calibrex Tsai-Lenz	0.875291 [0.77004, 0.967317]	13.8652 [11.7326, 15.8293]	0.0%	0.126921
Calibrex Daniilidis	0.868969 [0.761845, 0.960775]	13.8212 [11.9121, 15.6887]	0.0%	0.322403
Calibrex Andreff	0.867199 [0.759958, 0.958223]	13.879 [11.7674, 15.8309]	0.0%	0.334728
Calibrex Shiu-Ahmad	1.6191 [0.787831, 3.10277]	31.9219 [12.7334, 52.761]	0.0%	14.1537
Calibrex Chou-Kamel	0.867205 [0.759953, 0.958241]	13.8795 [11.7684, 15.8315]	0.0%	0.129661
Calibrex Horaud-Dornaika	0.890804 [0.782116, 0.996268]	14.1434 [11.8641, 16.1088]	0.0%	0.139859
Calibrex H-D nonlinear	0.885789 [0.777979, 0.991045]	14.0634 [11.8408, 16.0027]	0.0%	19.4512
OpenCV Tsai	0.871872 [0.769051, 0.961725]	13.8681 [11.8255, 15.6992]	0.0%	0.0368567
OpenCV Park	0.867225 [0.759969, 0.958144]	13.8754 [11.7667, 15.8261]	0.0%	0.0280751
OpenCV Horaud	0.867203 [0.759961, 0.958229]	13.879 [11.7674, 15.831]	0.0%	0.0248639
OpenCV Andreff	0.868623 [0.763653, 0.959819]	15.4855 [13.6068, 17.338]	0.0%	0.0327896
OpenCV Daniilidis	0.871005 [0.764385, 0.963618]	13.9265 [11.9635, 15.6961]	0.0%	0.0284915

Shared protocol: ethz-real-hand-eye/ax-xb/absolute-pose-split/v0.1; 5 split(s); 5000 paired bootstrap samples.

Limitations:

The dataset has no accepted ground-truth extrinsic; closure is consistency, not absolute accuracy.
Pairwise relative motions share absolute poses and therefore are not independent samples.
Runtime is single-process wall time on the recorded host; Python peak memory excludes native allocator visibility.

Robot-world hand-eye `AX=YB`¶

ETHZ real robot-world hand-eye AX=YB native and OpenCV benchmark¶

Method	Holdout rotation RMSE deg ↓	Holdout translation RMSE mm ↓	Failure rate	Runtime s
Calibrex Shah	0.624739 [0.559308, 0.688861]	10.7793 [9.53397, 11.9952]	0.0%	0.0206115
Calibrex Li-Wang-Wu	0.627131 [0.561467, 0.692796]	19.651 [15.2668, 23.3423]	0.0%	0.0321106
Calibrex Dornaika-Horaud	0.624742 [0.559311, 0.688865]	10.7793 [9.53399, 11.9952]	0.0%	0.0286939
Calibrex Zhuang-Roth-Sudhakar	0.651861 [0.569557, 0.733996]	10.961 [9.65144, 12.1956]	0.0%	0.0213057
Calibrex D-H nonlinear	0.6265 [0.560987, 0.692012]	10.7752 [9.52577, 12.0161]	0.0%	0.217955
OpenCV Shah	0.624739 [0.559308, 0.688861]	10.7793 [9.53397, 11.9952]	0.0%	0.00343018
OpenCV Li	0.627131 [0.561467, 0.692796]	19.651 [15.2668, 23.3423]	0.0%	0.00570322

Shared protocol: ethz-real-hand-eye/ax-yb/absolute-pose-split/v0.1; 5 split(s); 5000 paired bootstrap samples.

Limitations:

The dataset has no accepted ground-truth extrinsic; closure is consistency, not absolute accuracy.
AX=YB results are ranked separately from the AX=XB equation family.
Runtime is single-process wall time on the recorded host; Python peak memory excludes native allocator visibility.

Protocol¶

Dataset: ETHZ ASL robot_arm_w_color_camera_real.zip, DOI 10.3929/ethz-c-000788527
Source archive SHA-256: 2454578f731e656a940ddf51017b56e8d535c58d16a50629b85326005dedd6c3
Repetitions: seeds 0 through 4
Per split: 32 fit and 16 holdout absolute poses
Leakage control: absolute poses are split before relative motions are built
AX=XB native input: the exact pairwise relative-motion convention used by OpenCV 4 (inv(A_j) A_i, inv(B_j) B_i)
Tuning: fixed defaults; outer holdout data is evaluation-only
Failure policy: every method-by-split failure remains in the denominator
OpenCV boundary: optional opencv-python-headless>=4.8,<5, Apache-2.0; version 4.13.0.92 in the committed run

The five split means are accompanied by deterministic 95% bootstrap intervals. The pairwise relative motions share source poses, so they are not statistically independent observations; the split, rather than each pair, is the resampling unit.

Interpretation¶

On AX=XB, Calibrex Andreff has the lowest mean rotation closure RMSE and Calibrex Daniilidis has the lowest mean translation closure RMSE. The native Horaud-Dornaika nonlinear refinement does not win either metric and is much slower; this negative result is retained.

On AX=YB, OpenCV Shah and native Shah are numerically equivalent on rotation. The native nonlinear refinement has the lowest mean translation closure RMSE, but its intervals overlap the Shah and Dornaika-Horaud results. No accepted ground-truth extrinsic is supplied for this run, so closure measures equation consistency rather than absolute calibration accuracy.

Reproduce¶

python3 tools/download_public_dataset.py ethz_hand_eye_robot_arm_real
python3 tools/generate_ethz_hand_eye_benchmarks.py \
  data/public/ethz_hand_eye_robot_arm_real/robot_arm_w_color_camera_real.zip \
  --output-dir docs/assets \
  --seeds 0,1,2,3,4 \
  --fit-count 32 \
  --holdout-count 16

Each equation family has a raw benchmark definition, an aggregated slac.benchmark/v0.1 artifact, and generated Markdown in docs/assets. Every trial records the input/config/output digests, runtime, peak Python memory, method identity, paper DOI, tool version, license, and source archive digest.