Apply point cloud from realsense into OpenCV Surface matching algorithm

[anewworl]

So i have try to export the point cloud from the RealSense camera (I am using model D435i). I base on this link and try to replicate it by using point cloud from D435i https://docs.opencv.org/4.x/d9/d25/group__surface__matching.html and this is the reference i get from the opencv-contrib python here is the link to the document https://github.com/opencv/opencv_contrib/blob/4.x/modules/surface_matching/samples/ppf_load_match.py

import cv2 as cv

N = 2
modelname = "parasaurolophus_6700"
scenename = "rs1_normals"

detector = cv.ppf_match_3d_PPF3DDetector(0.025, 0.05)

print('Loading model...')
pc = cv.ppf_match_3d.loadPLYSimple("data/%s.ply" % modelname, 1)


print('Training...')
detector.trainModel(pc)

print('Loading scene...')
pcTest = cv.ppf_match_3d.loadPLYSimple("data/%s.ply" % scenename, 1)

print('Matching...')
results = detector.match(pcTest, 1.0/40.0, 0.05)

print('Performing ICP...')
icp = cv.ppf_match_3d_ICP(100)
_, results = icp.registerModelToScene(pc, pcTest, results[:N])

print("Poses: ")
for i, result in enumerate(results):
    #result.printPose()
    print("\n-- Pose to Model Index %d: NumVotes = %d, Residual = %f\n%s\n" % (result.modelIndex, result.numVotes, result.residual, result.pose))
    if i == 0:
        pct = cv.ppf_match_3d.transformPCPose(pc, result.pose)
        cv.ppf_match_3d.writePLY(pct, "%sPCTrans.ply" % modelname)

but i stuck at the step where i try to load the .ply data file export from the RealSense is not in correct format (https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py). Had anyone do this before can you have me with this problem pls.

[MartyG-RealSense]

Hi @anewworl I have not seen an example of a RealSense ply file being imported directly into an OpenCV script. My research indicates that imported ply data is typically converted into an OpenCV mat, like in the discussion at the link below.

https://stackoverflow.com/questions/29435273/pointcloud-data-to-mat-data

[anewworl]

Hi @MartyG-RealSense, After reading some article I had done some research and can finally do it but it come to another problem is that the point cloud from the realsense is kinda not meet my expectation, down here is some of the result and image in 2D (RGB, Depth), Point Cloud in Realsense:

Real world image

Left Image is file CAD, Right image is point cloud from realsense

Matching Reuslt

So i am try to do is detect pose(roll. pitch and yaw) of the free-object if you know any better solution just let me know.
Thansk for reading.

[MartyG-RealSense]

On the RealSense Viewer depth image, the background is likely missing because the Viewer has the threshold post-processing filter enabled by default, which restricts the maximum distance of the depth that can be displayed to 4 meters from the camera. Going to Stereo Module > Post-Processing, expanding open the list of post-processing filters and disabling the Threshold filter will enable depth up to 10 meters away to be included on the depth image.

The amount of detail on the depth image may also increase and the number of holes / gaps decrease if you maximize the value of the Laser Power setting to '360'. This setting can be found under Stereo Module > Controls.

Calculating roll, pitch and yaw with pyrealsense2 code by accessing the camera's IMU component is discussed at #4391

[anewworl]

oh no I mean here is the roll pitch yaw of an object.

[MartyG-RealSense]

If #4391 is not suitable for your requirements for calculating roll-pitch-yaw with Python code then the official OpenCV documentation does provide a Python pose estimation tutorial that makes use of an OpenCV SolvePNP algorithm to obtain pose.

https://docs.opencv.org/4.x/d7/d53/tutorial_py_pose.html

If ROS is an option for you then you could consider the DOPE method. Such a system uses an RGB image to recognize the object as a box and then calculates its 6 Degrees of Freedom (6DOF) pose. A couple of RealSense-compatible example projects of this type are in the links below.

https://github.com/pauloabelha/Deep_Object_Pose
https://github.com/avinashsen707/AUBOi5-D435-ROS-DOPE

Another way to obtain the pose of an object with a 400 Series camera is to attach an ArUco image tag to the object.

https://github.com/Jphartogi/ipa_marker_detection

However, if you have a Python solution already that can obtain the pose of an object and it works well then I would recommend continuing with it.