Skip to content

Latest commit

 

History

History
190 lines (173 loc) · 6.07 KB

prepare_dataset.md

File metadata and controls

190 lines (173 loc) · 6.07 KB

Dataset Setup

1. DAIR-V2X-I Dataset

1.1. Download DAIR-V2X-I dataset from official website.

1.2. Convert the dataset to KITTI format.

ln -s [single-infrastructure-side root] ./data/dair-v2x
python scripts/data_converter/dair2kitti.py --source-root data/dair-v2x-i --target-root data/dair-v2x-i-kitti

The directory will be as follows.

BEVHeight_Plus
├── data
│   ├── dair-v2x-i
│   │   ├── velodyne
│   │   ├── image
│   │   ├── calib
│   │   ├── label
|   |   └── data_info.json
|   ├── dair-v2x-i-kitti
|   |   ├── training
|   |   |   ├── calib
|   |   |   ├── label_2
|   |   |   └── images_2
|   |   └── ImageSets
|   |        ├── train.txt
|   |        └── val.txt
|   |...
|...

1.3. Prepare infos for DAIR-V2X-I dataset.

python scripts/gen_info_dair.py

2. Rope3D Dataset

2.1. Download Rope3D dataset from official website.

2.2. Convert the dataset to KITTI format.

ln -s [rope3d root] ./data/rope3d
python scripts/data_converter/rope2kitti.py --source-root data/rope3d --target-root data/rope3d-kitti

The directory will be as follows.

BEVHeight_Plus
├── data
|   ├── rope3d
|   |   ├── training
|   |   ├── validation
|   |   ├── training-image_2a
|   |   ├── training-image_2b
|   |   ├── training-image_2c
|   |   ├── training-image_2d
|   |   └── validation-image_2
|   ├── rope3d-kitti
|   |   ├── training
|   |   |   ├── calib
|   |   |   ├── denorm
|   |   |   ├── label_2
|   |   |   └── images_2
|   |   └── map_token2id.json
|   |...  
├── ...

2.3. Prepare infos for Rope3D dataset.

python scripts/gen_info_rope3d.py

3. KITTI Dataset

3.1. Download KITTI dataset from official website.

3.2. Prepare infos for KITTI dataset.

python scripts/gen_info_kitti.py --data_root data/kitti

4. KITTI-360 Dataset

4.1. Download KITTI-360 dataset from official website.

4.2. Download the processed KITTI-360 train_val and dummy testing labels. Extract them.

  • Thanks for the KITTI-360 labels from SeaBird repository.

4.3. Convert the dataset to KITTI format.

python data/kitti-360/calib_converter.py
python data/kitti-360/bbox_converter.py

# creat soft link
ln -s data/kitti-360/train_val data/kitti-360/training
ln -s data/kitti-360/training/label_2_converted data/kitti-360/training/label_2

The directory will be as follows.

BEVHeight_Plus
├── data
│      └── kitti-360
│             ├── ImageSets
│             ├── KITTI-360
│             │      ├── calibration
│             │      ├── data_2d_raw
│             │      ├── data_2d_semantics
│             │      ├── data_3d_boxes
│             │      └── data_poses
│             ├── train_val
│             │      ├── calib
│             │      ├── label
│             │      └── images_2
│             ├── training
│             │      ├── calib
│             │      ├── label
│             │      └── images_2
│             └── testing
│                    ├── calib
│                    ├── label
│                    └── images_2
│ ...

4.4. Prepare infos for KITTI-360 dataset.

python scripts/gen_info_kitti.py --data_root data/kitti-360

5. Waymo Dataset

5.1. Download Waymo dataset from official website.

5.2. Convert the dataset to KITTI format.

Set up environment

# Decompress the Waymo zip files into their corresponding directories
ls *.tar | xargs -i tar xvf {} -C your_target_dir
# Set up environment
conda create -n py36_waymo_tf python=3.7
conda activate py36_waymo_tf
conda install cudatoolkit=11.3 -c pytorch
# Newer versions of tf are not in conda. tf>=2.4.0 is compatible with conda.
pip install tensorflow-gpu==2.4
conda install pandas
pip3 install waymo-open-dataset-tf-2-4-0 --user

Parse Waymo dataset

ln -s [waymo root] ./data/waymo/raw_data
conda activate py36_waymo_tf
python scripts/data_converter/converter.py --load_dir data/waymo --save_dir data/waymo/parse_data --split training   --num_proc 10
python scripts/data_converter/converter.py --load_dir data/waymo --save_dir data/waymo/parse_data --split validation   --num_proc 10

Convert Waymo dataset parsed to KITTI format

python scripts/data_converter/waymo2kitti.py --source-root data/waymo/parse_data --target-root data/waymo-kitti

BEVHeight_Plus
├── data
│   ├── waymo
│   │   ├── ImageSets
│   │   │   ├── train_tfrecord.txt
│   │   │   ├── val_tfrecord.txt
│   │   ├── raw_data
│   │   │   ├── training
│   │   │   │   ├── segment-10017090168044687777_6380_000_6400_000_with_camera_labels.tfrecord
│   │   │   ├── validation
│   │   │   │   ├── segment-10203656353524179475_7625_000_7645_000_with_camera_labels.tfrecord
│   │   ├── parse_data
│   │   │   ├── training_org
│   │   │   │   ├── segment-id
│   │   │   ├── validation_org
│   │   │   │   ├── segment-id
│   │   ├── waymo_train_org.txt
│   │   ├── waymo_val_org.txt
│   ├── waymo-kitti
│   │   ├── ImageSets
│   │   ├── training
│   │   ├── validation

5.3. Prepare infos for Waymo dataset.

python scripts/gen_info_kitti.py --data_root data/waymo-kitti

6. Visualize the dataset in KITTI format

python scripts/data_converter/visual_tools.py --data_root data/waymo-kitti --demo_dir ./demo