[QUESTION] Should the labels and from uHuamans2 office dataset be like this?

[nesdust]

Hi and thanks for the interesting software and technology you have created.

Have been trying it out the past week and can't figure out if the labels and the would look like this (see the attached picture). The only reference I have seen are from your paper and YouTube presentation, neither of which shows the semantic labels overlapping the mesh. Could this be something that is wrong with our local config?

[nesdust]

E.g. 'Sofa' seems to be mapped to desktop, keyboard and so on. Also table/desk doesn't seem to be included at all.

[nathanhhughes]

Thanks for your interest in our work! Yes, the labels for each object are expected to be drawn at that height (see our youtube video here associated with the newer journal version of our paper).

The label names themselves are incorrect; they were hard-coded to be from the label space we use for our 2D semantic segmentation networks (which is the only label space we use for the linked video). I had thought I had included the non-hardcoded version in the latest release (but obviously didn't). I just pushed the fixes to this and Hydra-ROS, so now the label names should be pulled from the correct label space. Sorry about that!

[nesdust]

Hi! Thank you very much for the quick answer and fix, very appreciated!

A follow up question related to the semantic labeling though. How is it decided which objects that are tracked and labeled, e.g. the table (blue) is not included in the scene graph?

[nathanhhughes]

There is a configuration parameter that controls which semantic labels are considered objects (I think ~/objects/labels for this version of the code, though it might also be ~/frontend/object/labels). For the office scene, it is set here.

[nesdust]

Hi again! Works great. And it is now possible to choose which bounding boxes to create. There seems to be some problems with the tables, where separation of the middle ones seems hard to achieve. So far mostly tried different configurations settings in the hydra/config/uhumans2/reconstruction_config.yaml (https://github.com/MIT-SPARK/Hydra/blob/main/config/uhumans2/reconstruction_config.yaml) but with little success. Is this the right place to look and do you have some guidance which parameters that could be of interest?

Thank you again for all you help!

[nathanhhughes]

The euclidean clustering that is used to detect objects tends to undersegment nearby objects of the same class (i.e., the tables / desks that you are referring to). The parameters that control that process are here; cluster_tolerance is probably has the most influence for undersegmentation (the lower, the less undersegmentation). However, there is definitely a limit to how accurately Hydra will segment nearby object instances (this is a fundamentla limitation of how Hydra detects objects that we're trying to address in some ongoing work).

[nesdust]

Perfect, then I'm at the right track! Yet again thank you very much.

After familiarizing myself with the code I'm trying to test it out with my own scanner (realsense d435i). Right now i'm able to record a bag file with the topics (that i think are the most relevant):

• \camera\color\ image_raw,
• \camera\ imu,
• \camera\infra1\ image_rect_raw and
• \camera\infra2\ image_rect_raw

Is there some updated guide on how to make hydra work with either a real time scanner input or for a prerecorded bagfile? And lastly if not, do you have any general tips for getting it to work?

[nathanhhughes]

You'll probably also want /camera/depth/image_rect_raw instead of trying to do dense stereo yourself from the infrared images (we typically also run depth_image_proc/register instead of having the D455 do registration to the RGB image because it's been less stable in the past to do it directly with the realsense library). #1 still remains the best guide for how to set up Hydra with your own sensor, though note that some of the instructions are a little outdated (there is no longer a separate hydra_topology node, and the changes to Kimera-VIO that are required to do visual loop-closures are released). Finally, you may want to take a look at this launch file (we use the D435i as input for that config). semantic_recolor still isn't released yet, but the launch file that is included looks somewhat similar to this (albeit with 2D semanatic segmentation)

[skarthikbe96]

Hello nathanhhughes,

Thanks for great work, I am using realsense d435i. I am able to install and run Hydra latest version with uHuman dataset. I am able to run a 2D semantic segmentation external that provide /semantic_pointcloud topic . Can you please provide an example launch file ( older version example ) that will run the latest version of Hydra without requiring the semantic_recolor part. Thanks in advance

[nathanhhughes]

Hi @skarthikbe96, thanks for your interest in our work! We plan to release several changes soon (in which I plan to include an example semantic segmentation node in python / drop the usage of semantic recolor). In the meantime, the example launch file that you point to should be straightforward to adapt (but I will not be able to adapt it myself due to time constraints). I'm going to close this issue, but feel free to open up a new one if you have further questions!