Towards Global Localization using Multi-Modal Object-Instance Re-Identification

Aneesh Chavan¹, Vaibhav Agrawal^*1, Vineeth Bhat^*1,
Sarthak Chittawar^*1, Siddharth Srivastava³, Chetan Arora², K Madhava Krishna^1
1Robotics Research Centre, IIIT Hyderabad, ²IIT Delhi, ³Typeface Inc.
^*equal contribution

Accepted at Advances in Robotics, AIR 2025 (Oral)

Code | Paper | Datasets | Project Page

Setup

Clone all submodules

git submodule update --init

Setup conda environment

conda env create -f environment.yml
conda activate dator

Setup additional modules

Please clone the repo recursively to clone all the submodules as well.

RAM Model

cd object_memory/recognize-anything
pip install -e .

Grounding Dino and SAM

cd object_memory/Grounded-Segment-Anything
export AM_I_DOCKER=False
export BUILD_WITH_CUDA=True
export CUDA_HOME=/usr/local/cuda-11.8 # export CUDA_HOME=/path/to/cuda-11.3/ for others
python -m pip install -e segment_anything
pip install --no-build-isolation -e GroundingDINO

NOTE: Update the environment YAML before merging any PR. Remove the prefix property from the YAML file as well.

Download weights

bash bash_scripts/download_ram_sam_weights.sh 

• DATOR checkpoints are available at Google Drive Link, please change the directory path on line 102 in utils/embeddings.py to your download location.

Overall Documentation

Full Localisation Run

python tum_localisation_trial.py -t {run_name} --data-path {data_path} --map-pcd-cache-path {map_pcd_cache_path} --memory-load-path {memory_load_path} --embeddings {dino/clip/vit/dator}
python real_localisation_trial.py -t {run_name} --data-path {data_path} --map-pcd-cache-path {map_pcd_cache_path} --memory-load-path {memory_load_path} --embeddings {dino/clip/vit/dator}

DATOR ReID training

• Similar to the TransReID training setup.