NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation

Highlights

🔥 NAVSIM gathers simulation-based metrics (such as progress and time to collision) for end-to-end driving by unrolling simplified bird's eye view abstractions of scenes for a short simulation horizon. It operates under the condition that the policy has no influence on the environment, which enables efficient, open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors.

NAVSIM attempts to address some of the challenges faced by the community:

1. Providing a principled evaluation (by incorporating ideas + data from nuPlan)

• Key Idea: PDM Score, a multi-dimensional metric implemented in open-loop with strong correlation to closed-loop metrics
• Critical scenario sampling, focusing on situations with intention changes where the ego history cannot be extrapolated into a plan
• Official leaderboard on HuggingFace that remains open and prevents ambiguity in metric definitions between projects

2. Maintaining ease of use (by emulating nuScenes)

• Simple data format and reasonably-sized download (<nuPlan’s 20+ TB)
• Large-scale publicly available test split for internal benchmarking
• Continually-maintained devkit

🏁 NAVSIM will serve as a main track in the CVPR 2024 Autonomous Grand Challenge. The leaderboard for the challenge is open! For further details, please check the challenge website!

Table of Contents

1. Highlights
2. Getting started
3. Changelog
4. License and citation
5. Other resources

Getting started

• Download and installation
• Understanding and creating agents
• Understanding the data format and classes
• Dataset splits vs. filtered training / test splits
• Understanding the PDM Score
• Submitting to the Leaderboard

(back to top)

Changelog

• [2024/04/21] NAVSIM v1.0 release (official devkit version for AGC 2024)
  • IMPORTANT NOTE: The name of the data split competition_test was changed to private_test_e2e. Please adapt your directory name accordingly. For details see installation.
  • Parallelization of metric caching / evaluation
  • Adds Transfuser baseline (see agents)
  • Adds standardized training and test filtered splits (see splits)
  • Visualization tools (see tutorial_visualization.ipynb)
  • Refactoring
• [2024/04/03] NAVSIM v0.4 release
  • Support for test phase frames of competition
  • Download script for trainval
  • Egostatus MLP Agent and training pipeline
  • Refactoring, Fixes, Documentation
• [2024/03/25] NAVSIM v0.3 release (official devkit version for warm-up phase)
  • Changes env variable NUPLAN_EXP_ROOT to NAVSIM_EXP_ROOT
  • Adds code for Leaderboard submission
  • Major refactoring of dataloading and configs
• [2024/03/11] NAVSIM v0.2 release
  • Easier installation and download
  • mini and test data split integration
  • Privileged Human agent
• [2024/02/20] NAVSIM v0.1 release (initial demo)
  • OpenScene-mini sensor blobs and annotation logs
  • Naive ConstantVelocity agent

(back to top)

License and citation

All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The datasets (including nuPlan and OpenScene) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.

@misc{Contributors2024navsim,
    title={NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation},
    author={NAVSIM Contributors},
    howpublished={\url{https://github.com/autonomousvision/navsim}},
    year={2024}
} 

@inproceedings{Dauner2023CORL,
    title = {Parting with Misconceptions about Learning-based Vehicle Motion Planning},
    author = {Daniel Dauner and Marcel Hallgarten and Andreas Geiger and Kashyap Chitta},
    booktitle = {Conference on Robot Learning (CoRL)},
    year = {2023}
} 

(back to top)

Other resources

• SLEDGE | tuPlan garage | CARLA garage | Survey on E2EAD
• PlanT | KING | TransFuser | NEAT

(back to top)