RGB-D-Map-Segmentation-and-Pose-Estimation

Map point cloud segmentation using RGB-D sensor. The mapping is using rtabmap and do the segmentation at the same time. This project work using yolov5 as an object detector, using filtering to update extracted point cloud in the map frame and estimate the position from the object to the map. The code was designed to work out of the box with the Spot Robot body camera.

• Tested under Ubuntu 18.04 and ROS Melodic
• Require and

Getting Started

1.1. Setup the workspace

clone the repository to your ros workspace

cd ~/ros_ws/src/
git clone https://github.com/sugnite/RGB-D-Map-Segmentation-and-Pose-Estimation

Install the requirements for It's recommanded to create a virtual environement for both python versions (rgbd_sem_py3 and rgbd_sem_py2 in this case)

(rgbd_sem_py3)$ pip3 install -U -r requirements3.txt

Install requirements for

(rgbd_sem_py2)$ pip install -U -r requirements.txt

1.2. Setup Yolov5

Download the yolov5 for ROS package in your workspace directory. Follow the instructions to setup the package.

git clone https://github.com/sugnite/yolov5_ros

Download the weight of the project (require gdown).

pip install down
cd ~/ros_ws/src/yolov5_ros/src/yolov5
gdown https://drive.google.com/uc?id=1YIUekSGq6SvQ7KzKNgbcB-D4Yt8-SDLP

1.3. Setup RTAB-Map

Install rtab-map package for the mapping process

sudo apt install ros-melodic-rtabmap-ros

1.4. Install Spot Package

Install Spot packages from clearpath robotics

(rgbd_sem_py3)$ pip3 install bosdyn-client bosdyn-mission bosdyn-api bosdyn-core
git clone https://github.com/clearpathrobotics/spot_ros

1.5. Make your workspace

Make your workspace executable

cd ~/ros_ws/
catkin_make
source devel/setup.bash

1.6. Download the rosbag for tests (optional)

Download the testing rosbag

cd ~/ros_ws/src/RGB-D-Map-Segmentation-and-Pose-Estimation/rgbd_map_segmentation_and_pose_estimation/bags
gdown https://drive.google.com/uc?id=1tAV1GYfxH25EosJLpR4FSFkXNUuFxjuB

Run the program

The program work with 2 files to start in both python2 an python3

2.1. Run the Demo bag

Setup sim parameters

First Terminal

roscore

Second Terminal

rosparam set use_sim_time true
roscd rgbd_map_segmentation_and_pose_estimation/bags 
rosbag play --clock spot_demo_bag.bag

The next launch file will start the object detection network and the 3D filtering of the extracted point cloud

Third Terminal |

(rgbd_sem_py3)$ roslaunch rgbd_map_segmentation_and_pose_estimation semantic_classifier.launch

The last file will start the point cloud extraction and 3D pose estimation of the segmented objects

Fourth Terminal |

(rgbd_sem_py2)$ roslaunch rgbd_map_segmentation_and_pose_estimation second_node_semantic_classifier.launch