Public Dataset Scoreboard¶
bagx scores public rosbag datasets with a single command (bagx eval). This page is
generated from benchmarks/scoreboard.json
so scores stay reproducible across releases.
Scores are readiness hints, not SLAM trajectory accuracy. They summarize sensor rates, sync, IMU noise, and stack-specific topic coverage (Nav2 / Autoware / MoveIt / perception).
The Data column says where each bag comes from: public rows are scored on the
upstream download, sim rows on bags captured with scripts/run_ros_dogfood.py, and
unavailable rows are tracked but unscored because the upstream bag can no longer be
downloaded.
Analysis articles¶
- Why EuRoC MH_01_easy IMU looks noisy
- Why NTU VIRAL IMU looks noisy
- Why Autoware all-sensors-bag4 scores 83
30 datasets tracked (25 public · 5 simulator captures) · 22 scored (17 public) · bagx 0.6.0
| Dataset | Domain | ROS | Data | Overall | IMU | Sync | Key finding | bagx |
|---|---|---|---|---|---|---|---|---|
| EuRoC MH_01_easy | SLAM | ros1 | public | 60.3 | 23.1 | 97.6 | WARNING IMU accel noise 0.7935 m/s² — noisy, LiDAR-only odometry may outperform LIO | 0.6.0+dev |
| Hilti SLAM Challenge 2022 | SLAM | ros2 | public | — | — | — | Pending download — construction-site LiDAR+IMU benchmark | — |
| Livox MID-360 (AIST driving) | SLAM | ros2 | public | 98.7 | 97.3 | 100.0 | IMU accel noise 0.0160 m/s² — excellent, set imu_acc_noise_density to 0.0160 | 0.6.0+dev |
| M2DGR (urban street) | SLAM | ros1 | public | — | — | — | Pending download (~21GB) — multi-sensor GNSS/IMU/LiDAR urban dataset | — |
| NTU VIRAL eee_03 (drone) | SLAM | ros1 | public | 84.0 | 72.0 | 100.0 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Newer College (handheld) | SLAM | ros2 | public | — | — | — | Pending rescore — download requires the dataset request form | — |
| Ouster OS0-32 (static) | SLAM | ros2 | unavailable | — | — | — | Source unavailable — original recording was never pinned to a public file | — |
| TUM VI calib-imu1 | SLAM | ros1 | public | 95.4 | 90.8 | 100.0 | IMU accel noise 0.0515 m/s² — good for LIO, set imu_acc_noise_density to 0.0515 | 0.6.0+dev |
| UrbanLoco (Alameda) | SLAM | ros1 | public | — | — | — | Pending download (multi-GB) — long urban GNSS/IMU/LiDAR sequence | — |
| UrbanNav HK TST | SLAM | ros1 | public | — | — | — | Pending download (~35GB) — urban canyon GNSS + LiDAR fusion bag | — |
| AutoCore Ouster OS1-64 | Autoware | ros2 | public | 100.0 | — | — | LiDAR (/sensing/lidar/top/rectified/pointcloud) at 10Hz | 0.6.0 |
| Autoware all-sensors-bag1 | Autoware | ros2 | public | 93.8 | 85.8 | 89.6 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware all-sensors-bag2 | Autoware | ros2 | public | 90.4 | 69.8 | 91.9 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware all-sensors-bag3 | Autoware | ros2 | public | 86.9 | 78.7 | 80.0 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware all-sensors-bag4 | Autoware | ros2 | public | 82.8 | 48.8 | 87.4 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware all-sensors-bag5 | Autoware | ros2 | public | 90.2 | 77.9 | 87.1 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware all-sensors-bag6 | Autoware | ros2 | public | 81.6 | 40.3 | 88.3 | GNSS fix rate 100% — suitable as ground truth reference | 0.6.0+dev |
| Autoware driving_20_kmh | Autoware | ros2 | public | 99.0 | — | 97.9 | Sensor sync good (8.3ms) | 0.6.0+dev |
| Autoware driving_30_kmh | Autoware | ros2 | public | 99.0 | — | 97.9 | Sensor sync good (8.3ms) | 0.6.0+dev |
| Nav2 Gazebo (deep capture) | Nav2 | ros2 | sim | 100.0 | 100.0 | — | Pipeline scan → costmap: 43ms median, 82ms P95 | 0.6.0 |
| Nav2 Gazebo (goal capture) | Nav2 | ros2 | sim | 88.6 | 100.0 | 77.1 | IMU accel noise 0.0170 m/s² — excellent, set imu_acc_noise_density to 0.0170 | 0.6.0 |
| Nav2 Gazebo (headless) | Nav2 | ros2 | sim | 100.0 | 100.0 | — | Control command (/cmd_vel) at 10Hz — good for control-loop observability | 0.6.0 |
| TurtleBot3 Walker | Nav2 | ros2 | unavailable | — | — | — | Source unavailable — upstream bag no longer downloadable | — |
| Franka Panda (MoveIt) | MoveIt | ros2 | unavailable | — | — | — | Source unavailable — upstream bag no longer downloadable | — |
| MoveIt execution capture | MoveIt | ros2 | sim | 100.0 | — | — | Pipeline joint_states → planned_path: 6ms median, 6ms P95 (1 sample) | 0.6.0 |
| MoveIt plan-only capture | MoveIt | ros2 | sim | 66.7 | — | — | Pipeline joint_states → planned_path: 9ms median, 9ms P95 (1 sample) | 0.6.0 |
| NVIDIA r2b_galileo | Perception | ros2 | public | 90.9 | 72.8 | 100.0 | WARNING IMU accel noise 0.2882 m/s² — noisy, consider checking IMU calibration | 0.6.0+dev |
| NVIDIA r2b_galileo2 | Perception | ros2 | public | 97.5 | — | 95.3 | Sensor sync good (16.2ms) | 0.6.0+dev |
| NVIDIA r2b_whitetunnel | Perception | ros2 | public | 91.7 | 75.1 | 100.0 | WARNING IMU accel noise 0.2812 m/s² — noisy, consider checking IMU calibration | 0.6.0+dev |
| NVIDIA r2b_robotarm | Manipulation | ros2 | public | 96.0 | — | 92.2 | Sensor sync good (10.2ms) | 0.6.0+dev |
Reproduce a row¶
Each manifest entry includes a reproduce command. Example:
export BAGX_SCOREBOARD_BAGS=/path/to/downloaded/bags
export BAGX_DB3_CACHE=/path/with/free/space # for .db3.zstd decompression
# also accepts BAGX_REALBAGS for NVIDIA / Autoware mirrors
bagx eval "$BAGX_SCOREBOARD_BAGS/r2b_galileo2"
Refresh the table after downloading bags:
export BAGX_SCOREBOARD_BAGS=/path/to/bags
python scripts/generate_scoreboard.py --refresh --write-manifest
python scripts/generate_scoreboard.py
Manifest: benchmarks/scoreboard.json
Sample commands by domain¶
SLAM
- Newer College (handheld):
bagx eval newer_college.db3 - Livox MID-360 (AIST driving):
bagx eval livox_mid360_aist
Autoware
- Autoware all-sensors-bag4:
aws s3 sync s3://autoware-files/recordings/bags/2022-08-22_leo_drive_isuzu_bags/all-sensors-bag4_compressed/ ./all-sensors-bag4_compressed --no-sign-request && bagx eval all-sensors-bag4_compressed - Autoware driving_20_kmh:
aws s3 sync s3://autoware-files/recordings/bags/2022-08-22_leo_drive_isuzu_bags/driving_20_kmh_2022_06_10-16_01_55_compressed/ ./driving_20 --no-sign-request && bagx eval driving_20
Nav2
- Nav2 Gazebo (deep capture):
python scripts/run_ros_dogfood.py nav2-gazebo && bagx eval <captured_bag> - Nav2 Gazebo (headless):
python scripts/run_ros_dogfood.py nav2-gazebo-headless && bagx eval <captured_bag>
MoveIt
- MoveIt execution capture:
python scripts/run_ros_dogfood.py moveit-demo && bagx eval <captured_bag> - MoveIt plan-only capture:
python scripts/run_ros_dogfood.py moveit-demo-plan && bagx eval <captured_bag>
Perception
- NVIDIA r2b_galileo:
bagx eval r2b_galileo - NVIDIA r2b_whitetunnel:
bagx eval r2b_whitetunnel
Manipulation
- NVIDIA r2b_robotarm:
bagx eval r2b_robotarm