Not able to build Moveit2 docker image

[MinahilRaza]

I have been trying to run demos on my machine. The spaceros image builds successfully but I get the following error for the moveit2 one

[Bckempa]

I'll see if I can replicate when we do our release validation builds.

@mkhansenbot, might this be related to #142?

[eholum]

@MinahilRaza it looks like the actual failure in interactive_markers was cut off in your image build. Do you happen to have more of the build logs available?

As @Bckempa said, we're currently doing our release build validations and will see if we run into any problems on our end.

[roboPanda69]

Hi I also faced the same issue during my image build. But I got the error not for only interactive_markers but different package (i.e., whenever I was running a new build process I was getting the same error but now for different package). The main error I got, was the error as c++: fatal error: Killed signal terminated program cc1plus`.

The method which worked for me was we have to increase the ram limit in the docker. These are the steps to do the same -

1. Open Docker Desktop
2. Go to Settings
3. Go for Resources and Advance
4. Increase the slider of RAM limit. For me default was set at 3.9 GB. I would suggest to change for 9.8 GB or greater. (I had seen some process taking straight 8 GB RAM, and these might cause problems)

Also in the Dockerfile inside the Moveit2 folder, I had to replace the lines from -
# Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'

to these lines as -
# Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --executor sequential --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'

i.e., add the line to --executor sequential to colcon build which would reduce number of parallel jobs, hence less RAM usage.

Then it worked perfectly, although it takes time to build.

[roboPanda69]

@MinahilRaza it looks like the actual failure in interactive_markers was cut off in your image build. Do you happen to have more of the build logs available?

As @Bckempa said, we're currently doing our release build validations and will see if we run into any problems on our end.

@eholum @Bckempa During build validation could we check if the build is working for systems having less RAM. I checked my RAM usage during the build while this line was running from the Dockerfile -
# Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'

and I noticed that even if I set the RAM limit of Docker to be 15.5 GB (I have a total of 16 GB RAM) it was not able to build the image (Even whole of the RAM was being used only for the build process).

Also, could I make a pull request with the solution that I proposed in the above thread ?

[asimonov]

Hi I also faced the same issue during my image build. But I got the error not for only interactive_markers but different package (i.e., whenever I was running a new build process I was getting the same error but now for different package). The main error I got, was the error as c++: fatal error: Killed signal terminated program cc1plus`.

The method which worked for me was we have to increase the ram limit in the docker. These are the steps to do the same -

1. Open Docker Desktop

2. Go to Settings

3. Go for Resources and Advance

4. Increase the slider of RAM limit. For me default was set at 3.9 GB. I would suggest to change for 9.8 GB or greater. (I had seen some process taking straight 8 GB RAM, and these might cause problems)

Also in the Dockerfile inside the Moveit2 folder, I had to replace the lines from - # Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'

to these lines as - # Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --executor sequential --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'

i.e., add the line to --executor sequential to colcon build which would reduce number of parallel jobs, hence less RAM usage.

Then it worked perfectly, although it takes time to build.

I am not in favour or reducing parallelism of the build. it would be taking much longer for people with more powerful machines and CI...

I am using an x86 laptop with 15GB and 16 cores and these docker builds do not fail for me. could it be that you have higher number of CPUs for less RAM? or could this be related to swapfile sizing?

[asimonov]

I'll see if I can replicate when we do our release validation builds.

@mkhansenbot, might this be related to #142?

I think #142 was about nav2 docker. this one is about moveit2

[Bckempa]

@eholum @Bckempa During build validation could we check if the build is working for systems having less RAM.

OOM-ing during build is a known issue with Moveit2, this might be more appropriate discussion on the upstream project directly. Until then I've been using ZRAM to build consistently on my machines

[eholum]

Until then I've been using ZRAM to build consistently on my machines

Cannot emphasize the utility of ZRAM enough.

I am not in favour or reducing parallelism of the build. it would be taking much longer for people with more powerful machines and CI...

I agree! One other work around I tend to use is just to drastically increase the size of my swapfile.

[roboPanda69]

Hi I also faced the same issue during my image build. But I got the error not for only interactive_markers but different package (i.e., whenever I was running a new build process I was getting the same error but now for different package). The main error I got, was the error as c++: fatal error: Killed signal terminated program cc1plus`.
The method which worked for me was we have to increase the ram limit in the docker. These are the steps to do the same -

1. Open Docker Desktop

2. Go to Settings

3. Go for Resources and Advance

4. Increase the slider of RAM limit. For me default was set at 3.9 GB. I would suggest to change for 9.8 GB or greater. (I had seen some process taking straight 8 GB RAM, and these might cause problems)

Also in the Dockerfile inside the Moveit2 folder, I had to replace the lines from - # Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'
to these lines as - # Build MoveIt2 RUN /bin/bash -c 'source ${SPACEROS_DIR}/install/setup.bash \ && colcon build --executor sequential --cmake-args -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON --event-handlers desktop_notification- status-'
i.e., add the line to --executor sequential to colcon build which would reduce number of parallel jobs, hence less RAM usage.
Then it worked perfectly, although it takes time to build.

I am not in favour or reducing parallelism of the build. it would be taking much longer for people with more powerful machines and CI...

I am using an x86 laptop with 15GB and 16 cores and these docker builds do not fail for me. could it be that you have higher number of CPUs for less RAM? or could this be related to swapfile sizing?

Hi, you are right. I tried building images by increasing swap file size (also the swap file limit in Docker) and it worked perfectly with no errors.

[MinahilRaza]

Increasing the swap file size solved this issue for me. It worked perfectly without errors