womd-prediction

Author: georgeliu233

README.md

@@ -1,16 +1,28 @@
 <h1 align="center"> BeTop: Reasoning Multi-Agent Behavioral Topology 
 for Interactive Autonomous Driving </h1> 
 
+<div align="center">
+ 
+[![arXiv](https://img.shields.io/badge/arXiv-2409.18031-479ee2.svg)](https://arxiv.org/abs/2409.18031)
+
+</div>
+
+
 <div id="top" align="center">
 <p align="center">
 <img src="assets/betop_teaser.png" width="1000px" >
 </p>
 </div>
 
-> [Haochen Liu](https://scholar.google.com/citations?user=iizqKUsAAAAJ&hl), [Li Chen](https://scholar.google.com/citations?user=ulZxvY0AAAAJ&hl=zh-CN), [Yu Qiao](https://scholar.google.com/citations?user=gFtI-8QAAAAJ&hl=zh-CN&inst=8669986779262753491&oi=ao), [Chen Lv](https://scholar.google.com/citations?user=UKVs2CEAAAAJ&hl=zh-CN&inst=8669986779262753491&oi=ao) and [Hongyang Li](https://scholar.google.com/citations?user=Hfrih1EAAAAJ&hl=zh-CN&inst=8669986779262753491&oi=ao)
+> - [Haochen Liu](https://scholar.google.com/citations?user=iizqKUsAAAAJ&hl), [Li Chen](https://scholar.google.com/citations?user=ulZxvY0AAAAJ&hl=zh-CN), Yu Qiao, Chen Lv and Hongyang Li  
+> - [arXiv paper](https://arxiv.org/abs/2409.18031) | Slides TODO  
+> - If you have any questions, please feel free to contact: *Haochen Liu* ( [email protected] )
 
 <!-- > 📜 Preprint: <a href="https://arxiv.org/abs/2409.09016"><img src="https://img.shields.io/badge/arXiv-Paper-<color>"></a> -->
 
+---
+
+**[2024-11]** Prediction project released.
 
 Full code and checkpoints release is coming soon. Please stay tuned.
 
@@ -32,12 +44,37 @@ The synergetic framework, *BeTopNet*, integrates topology reasoning with predict
 </p>
 </div>
 
-## TODO list
+## Get Started
+
+### Prediction
+
+We provide the full prediction implementation of BeTopNet in Waymo Open Motion Dataset (WOMD).
+
+**Features:**
+- :white_check_mark: Full support for WOMD Prediction Challenges  
+- :white_check_mark: Flexible Toolbox for prediction tasks  
+- :white_check_mark: Pipeline for reproduced popular Baselines
+
+**[Guideline](womd/README.md)**
+   - [Data Preparation](/docs/womd/DataPrep_pred.md)
+   - [Training & Evaluation](/docs//womd/TrainEval_pred.md)
+   - [Testing & Submission](/docs//womd/Submission_pred.md)
+
+## TODO List
 
 - [x] Initial release
-- [ ] Prediction pipeline in WOMD
+- [x] Prediction pipeline in WOMD
 - [ ] Planning pipeline in nuPlan
 
 ## Citation
 
-If you find the project helpful for your research, please consider citing our paper:
+If you find the project helpful for your research, please consider citing our paper:
+
+```bibtex
+@inproceedings{liu2024betop,
+ title={Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Driving}, 
+ author={Haochen Liu and Li Chen and Yu Qiao and Chen Lv and Hongyang Li},
+ booktitle={NeurIPS},
+ year={2024}
+}
+```

docs/womd/DataPrep_pred.md

@@ -0,0 +1,43 @@
+# Preparation for WOMD prediction
+
+## Installation
+
+Run the following command to install the BeTopNet (WOMD) package:
+
+```bash
+cd womd && pip install -e .
+```
+
+To build the necessary CUDA dependencies, please refer to [EQNet](https://github.com/dvlab-research/DeepVision3D/tree/master/EQNet/eqnet/ops).
+
+Make sure the corresponding WOMD package are successfully installed
+
+## Data Preparation
+
+### 1. Download the Data
+
+Download the ```scenario/``` part of Waymo Open Motion Dataset under certain path and your specified task: [Link](https://console.cloud.google.com/storage/browser/waymo_open_dataset_motion_v_1_3_0;tab=objects?prefix=&forceOnObjectsSortingFiltering=false)
+
+Download the ```intention_points.pkl```. You may find the corresponding files in the following [Link](https://github.com/OpenDriveLab/BeTop/releases/tag/womd)
+
+### 2. Preprocess the Raw Data:
+
+**[NOTE]** The ```data_process``` code assumes you have downloaded both motion & interaction data, so you may modify according to your need.
+
+Run the following command to preprocess the raw data:
+
+```bash
+cd tools/data_tools
+python3 data_process.py your_downloaded_path your_preferred_info_path
+```
+
+### 3. **[Optional]** Cache Preprocessed Data
+
+For memory-efficient training, you can preprocess the info data in ```.npz``` format cache:
+
+```bash
+cd tools/data_tools
+python3 cache_offline_data.py --cache_path YOUR_CACHE_PATH --cfg ../cfg/BeTopNet_full_64.yaml
+```
+
+**[NOTE]** The caching process would require around ```3-4TB``` storage space.

docs/womd/Submission_pred.md

@@ -0,0 +1,52 @@
+# Submission for WOMD Leaderboard
+
+## Submission Info
+
+Specify your registered information in submission code:
+
+1. Trace to Line 188 in ```womd/tools/submission.py```.
+2. Specify your information in the following format:
+
+```python
+submission_info = dict(
+        account_name='your Waymo account email',
+        unique_method_name='xxx',
+        authors=['A', 'B', 'xxx'],
+        affiliation='your affiliation',
+        uses_lidar_data=False,
+        uses_camera_data=False,
+        uses_public_model_pretraining=False,
+        public_model_names='N/A',
+        num_model_parameters='N/A',
+    )
+```
+
+
+## Submission File Generation
+
+Run the submission pipeline:
+
+```bash
+cd womd/tools
+python3 test_submission.py \
+        --cfg_file cfg/YOUR_CHOSEN_BASELINE.yaml \
+        --batch_size BATCH \
+        --ckpt MODEL_CKPT \
+        --output_dir path_to_save_submission_file \
+        #--eval \ # whether to submit the eval / test portal
+        #--interactive # whether to submit the motion / interaction prediction task
+```
+
+**Note:** Fot the last two arguments, the default setup is for ```test``` submission for *Motion Prediction* challenge.
+
+- Use ```--eval``` if you want to check the evaluation on the Leaderboard portal.
+- Add the ```--interactive``` argument if you want to submit for *Interaction Prediction* challenge.
+
+## Upload
+
+Submit your results to the Leaderboard portal:
+
+You can find the corresponding ```.tar.gz``` file under your specified ```--output_dir```. Submit to the Leaderboard portal:
+
+- [Motion Prediction challenge](https://waymo.com/open/challenges/2024/motion-prediction/)
+- [Interaction Prediction challenge](https://waymo.com/open/challenges/2021/interaction-prediction/)

docs/womd/TrainEval_pred.md

@@ -0,0 +1,29 @@
+# Training & Evaluation for WOMD prediction
+
+## Baseline Model Selection
+Choose a baseline model to train through config files in ```womd/tools/cfg/```:
+
+```bash
+BeTopNet-full: cfg/BeTopNet_full_64.yaml
+BeTopNet-e2e: cfg/BeTopNet_e2e_6.yaml
+MTR++: cfg/MTR_PlusPlus.yaml
+Wayformer: cfg/Wayformer.yaml
+```
+
+## Training
+To start the training pipeline, run the following command:
+
+```bash
+cd womd/tools
+bash scripts/dist_train.sh N_GPUS --cfg_file cfg/YOUR_CHOSEN_BASELINE.yaml --epoch 30 --batch_size BATCH --extra_tag XXX
+```
+
+**[TIPS]** If you have not cached the data, your may set ```BATCH = 10*N_GPUS``` using A100(80G) gpus. Otherwise, you may leverage a larger batch size.
+
+## Evaluation
+To evaluate the trained model, use the following command:
+
+```bash
+cd womd/tools
+bash scripts/dist_test.sh N_GPUS --cfg_file cfg/YOUR_CHOSEN_BASELINE.yaml --batch_size BATCH --ckpt YOUR_MODEL_CKPT
+```

womd/README.md

@@ -0,0 +1,39 @@
+# BeTopNet (WOMD Prediction)
+
+This folder contains the full prediction implementation of BeTopNet in [Waymo Open Motion Dataset (WOMD)](https://waymo.com/open/challenges).
+
+
+## Highlights
+
+Our prediction pipeline contains the following features:
+
+:white_check_mark: **Full support for WOMD Prediction Challenges:**
+
+- Full pipeline for [WOMD Motion Prediction](https://waymo.com/open/challenges/2024/motion-prediction/) challenge.
+- Full pipeline for [WOMD Interaction Prediction](https://waymo.com/open/challenges/2021/interaction-prediction/) challenge.
+
+:white_check_mark: **Flexible Toolbox for prediction tasks:**
+
+- Memory Efficient: Cached dataset and model training.
+- User-Friendly Tools: Simple tools for model training/evaluation and **Leaderboard submission**.  
+- Scene Renderer: Visualize predicted trajectories.
+
+:white_check_mark: **Pipeline for reproduced popular Baselines:**
+
+- [Wayformer](https://arxiv.org/pdf/2207.05844) baseline reproduction.
+- [MTR++](https://arxiv.org/pdf/2306.17770) baseline reproduction (Deprecated).
+
+## Get Started
+
+- [Installation & Data Preparation](../docs//womd/DataPrep_pred.md)
+- [Training & Evaluation](../docs//womd/TrainEval_pred.md)
+- [Submission to Leaderboard](../docs//womd/Submission_pred.md)
+
+
+## Acknowledgment / Related resources
+
+:star_struck: We would like to thank the following repositories and works for providing a foundation for building our pipelines:
+
+- [MTR](https://github.com/sshaoshuai/MTR)
+- [UniTraj](https://github.com/vita-epfl/UniTraj)
+- [TopoNet](https://github.com/OpenDriveLab/TopoNet)

womd/betopnet/config.py

@@ -0,0 +1,89 @@
+'''
+Mostly from Motion Transformer (MTR): https://arxiv.org/abs/2209.13508
+'''
+
+from pathlib import Path
+
+import yaml
+from easydict import EasyDict
+
+def log_config_to_file(cfg, pre='cfg', logger=None):
+    for key, val in cfg.items():
+        if isinstance(cfg[key], EasyDict):
+            logger.info('\n%s.%s = edict()' % (pre, key))
+            log_config_to_file(cfg[key], pre=pre + '.' + key, logger=logger)
+            continue
+        logger.info('%s.%s: %s' % (pre, key, val))
+
+
+def cfg_from_list(cfg_list, config):
+    """Set config keys via list (e.g., from command line)."""
+    from ast import literal_eval
+    assert len(cfg_list) % 2 == 0
+    for k, v in zip(cfg_list[0::2], cfg_list[1::2]):
+        key_list = k.split('.')
+        d = config
+        for subkey in key_list[:-1]:
+            assert subkey in d, 'NotFoundKey: %s' % subkey
+            d = d[subkey]
+        subkey = key_list[-1]
+        assert subkey in d, 'NotFoundKey: %s' % subkey
+        try:
+            value = literal_eval(v)
+        except:
+            value = v
+
+        if type(value) != type(d[subkey]) and isinstance(d[subkey], EasyDict):
+            key_val_list = value.split(',')
+            for src in key_val_list:
+                cur_key, cur_val = src.split(':')
+                val_type = type(d[subkey][cur_key])
+                cur_val = val_type(cur_val)
+                d[subkey][cur_key] = cur_val
+        elif type(value) != type(d[subkey]) and isinstance(d[subkey], list):
+            val_list = value.split(',')
+            for k, x in enumerate(val_list):
+                val_list[k] = type(d[subkey][0])(x)
+            d[subkey] = val_list
+        else:
+            assert type(value) == type(d[subkey]), \
+                'type {} does not match original type {}'.format(type(value), type(d[subkey]))
+            d[subkey] = value
+
+
+def merge_new_config(config, new_config):
+    if '_BASE_CONFIG_' in new_config:
+        with open(new_config['_BASE_CONFIG_'], 'r') as f:
+            try:
+                yaml_config = yaml.load(f, Loader=yaml.FullLoader)
+            except:
+                yaml_config = yaml.load(f)
+        config.update(EasyDict(yaml_config))
+
+    for key, val in new_config.items():
+        if not isinstance(val, dict):
+            config[key] = val
+            continue
+        if key not in config:
+            config[key] = EasyDict()
+        merge_new_config(config[key], val)
+
+    return config
+
+
+def cfg_from_yaml_file(cfg_file, config):
+    with open(cfg_file, 'r') as f:
+        try:
+            new_config = yaml.load(f, Loader=yaml.FullLoader)
+        except:
+            new_config = yaml.load(f)
+
+        merge_new_config(config=config, new_config=new_config)
+
+    return config
+
+
+
+cfg = EasyDict()
+cfg.ROOT_DIR = (Path(__file__).resolve().parent / '../').resolve()
+cfg.LOCAL_RANK = 0