Skip to content

ETHZ real robot-world hand-eye benchmark

Calibrex recomputes five AX=YB methods on the public ETHZ ASL real robot-arm pose streams. Every method uses the same 1,350 fit pairs and 338 held-out pairs.

ETHZ real robot-world hand-eye shared-split benchmark

Method Rotation holdout RMSE deg ↓ Translation holdout RMSE mm ↓ Known-bad detection fraction ↑ Failure rate Runtime s
Shah 0.585795 10.0628 1 0.0% —
Li-Wang-Wu 0.58728 17.458 1 0.0% —
Dornaika-Horaud 0.585795 10.0628 1 0.0% —
Zhuang-Roth-Sudhakar 0.590747 10.132 1 0.0% —
Calibrex nonlinear refinement 0.58721 10.0394 1 0.0% —

Shared protocol: ethz-real-robot-world-hand-eye/shared-split/v0.1; 1 split(s); 2000 paired bootstrap samples.

Limitations:

  • Metrics are closure errors on held-out pose pairs, not ground-truth extrinsic errors.
  • This historical run contains one shared split, so its bootstrap interval cannot measure split-to-split uncertainty.
  • Per-method runtime and peak memory were not recorded by the original combined run and remain explicitly unavailable.
  • The scoped win is translation holdout RMSE on this dataset and split; Shah has the lowest rotation holdout RMSE.
  • All compared solvers are independent native Calibrex implementations of the cited methods.

The nonlinear refinement reduces translation holdout RMSE from the Dornaika–Horaud closed-form initialization by 0.23%. Shah has the lowest rotation holdout RMSE, by less than 0.000001° relative to the closed-form Dornaika–Horaud result.

This benchmark supports only the scoped claim above. Its metrics are rotation/translation closure errors on unseen pose pairs, not errors against a ground-truth extrinsic. The run therefore does not establish that one method has universally better calibration accuracy.

Protocol

  • Dataset: ETHZ ASL robot_arm_w_color_camera_real.zip (dataset page, DOI 10.3929/ethz-c-000788527)
  • Pairing: disjoint absolute-pose pairs
  • Split: seeded shared 80/20 fit/holdout split
  • Fit pairs: 1,350
  • Holdout pairs: 338
  • Known-bad controls: ±5 cm translation and ±5° rotation probes on both recovered transforms, 24 per method
  • External solver code: not executed
  • Verdict thresholds: rotation RMSE ≤ 2°; translation RMSE ≤ 30 mm; known-bad detection ≥ 75%

All methods passed the three gates. The table highlights the minimum of each holdout metric; it does not combine unlike units into an unreported aggregate score.

Methods

  • Shah, Solving the Robot-World/Hand-Eye Calibration Problem Using the Kronecker Product, DOI 10.1115/1.4024473
  • Li, Wang, and Wu, Simultaneous Robot-World and Hand-Eye Calibration Using Dual-Quaternions and Kronecker Product, DOI 10.5897/IJPS.9000501
  • Dornaika and Horaud, Simultaneous Robot-World and Hand-Eye Calibration, DOI 10.1109/70.704233
  • Zhuang, Roth, and Sudhakar, Simultaneous Robot/World and Tool/Flange Calibration by Solving Homogeneous Transformation Equations of the Form AX=YB, DOI 10.1109/70.313105
  • Calibrex nonlinear refinement: a native nonlinear refinement initialized from the Dornaika–Horaud closed-form estimate; no external implementation is copied or executed

Reproduce

Download the declared archive, run the comparison, then verify the evidence bundle:

python3 tools/download_public_dataset.py ethz_hand_eye_robot_arm_real
calibrex calibrate \
  examples/public_datasets/ethz_hand_eye_robot_arm_real/config.yaml \
  --output-dir outputs/ethz_hand_eye_robot_arm_real
calibrex verify outputs/ethz_hand_eye_robot_arm_real/bundle.json

The committed raw benchmark definition is aggregated into ethz-robot-world-hand-eye-benchmark.json and the table included above by calibrex benchmark. Both artifacts record the exact values, source digests, run identity, method citations, and reproduction command. They validate against the generated benchmark_definition and benchmark JSON Schemas.