MONSTER Dataset

Robustness Analysis of Monocular SLAM Systems: Current Limitations and Future Research Directions

Explore the Dataset

Algorithms evaluated

Public Datasets

106

Benchmark sequences

Isolated failure modes (MONSTER)

Difficulty levels per mode

50x

Trials per sequence

Dataset previews unavailable

Interactive Visualizer

Explore trajectories, videos, and evaluation metrics across datasets, variants, and algorithms, built for fast qualitative and quantitative inspection.

Open Visualizer

Video and 3D scene viewer

Inspect estimated trajectories and scene geometry alongside the reference video in lockstep.

Comparative results table

Browse APE/RPE-style metrics grouped by algorithm family with filtering and highlighting.

Birdseye projection

Project trajectories onto a top-down reference image for a clear spatial overview (zoomable).

Custom uploads

Overlay your own trajectories for quick comparisons against published baselines.

Abstract

Simultaneous Localization and Mapping (SLAM) is essential for autonomous robots, yet despite years of research, state-of-the-art visual SLAM algorithms still fail in many real-world environments. Current benchmarks evaluate on scenarios with multiple confounding factors, such as motion blur, poor lighting, and dynamic objects. While this is useful for evaluating real world performance, it does not give specific and actionable insights into what is causing performance degradation, hindering algorithmic innovation.

Our paper conducts the most extensive robustness analysis to date, analyzing 8 popular monocular SLAM algorithms across sequences from 8 benchmark datasets and performing detailed analysis to identify current limitations and propose research directions needed to advance monocular SLAM toward robust real-world deployment. To complement this evaluation, we present MONSTER (MONocular Synthetic TEsting of Robustness), a novel synthetic dataset that isolates 28 individual sources of performance degradation, each with three difficulty levels. By systematically introducing camera, environment, and trajectory challenges, MONSTER enables precise identification of algorithm-specific weaknesses that are masked in normal datasets.

We make the MONSTER dataset, our collected data, and an interactive data visualizer available at our website

Highlights

Benchmark at scale

8 state-of-the-art monocular SLAM/SfM systems across 8 datasets (106 sequences), with repeated trials for statistical confidence.

MONSTER: isolated stress tests

A synthetic suite that isolates 28 individual failure modes with easy/medium/hard variants, so you can pinpoint what breaks and when.

Actionable research directions

A failure-driven roadmap of what's still missing for robust real-world monocular SLAM (not just leaderboard numbers).

Standardized, reproducible protocol

Uniform hardware + Dockerized runs and consistent settings to enable fair comparisons across systems and datasets.

Open resource

Release of MONSTER, collected outputs, and an interactive data visualizer to accelerate debugging and comparisons.

Resources

Links for the paper, code, dataset downloads, and reproducibility.

Release status

Dataset and full results are released upon acceptance. This section will host, downloads, and change notes.

Cite

If you use MONSTER or the benchmark results in your research, please cite:

@article{monster2025,
  title   = {Robustness Analysis of Monocular SLAM Systems: Current Limitations and Future Research Directions},
  author  = {People's Names are Anonymized for Review},
  journal = {IEEE Transactions on Robotics},
  year    = {2025}
}