Add docker buildx bake and support for cmake

[danvergara]

Replaces #669 and rewrites #673

Changes

• This PR introduces docker buildx bake command which is a new feature that lets you define your build configuration using a declarative file
• This PR rewrites 673 because it could break <=28.1 images as it is today, due to the removal of the autosgen.sh file in favor of cmake to build Bitcoin.
• This PR adds a new Dockerfile to support the new way to compile Bitcoin and a couple of new “base” targets to the docker-bake.hcl file that both inherit from the maintained-base target
• The second base stage is for >28.1 and uses the new Dockerfile.dev file. The name can be changed to anything else, just the first option that came to mind
• The BUILD_ARGS field was modified to support new parameters to build Bitcoin using cmake

[willcl-ark]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

[deadmanoz]

Nice work @danvergara 👏 💪

There were a few suggestions from myself, Onyekachukwu-Nweke and macgyver13 (can't tag them) on the original PR that you might want to consider incorporating. #669

These suggestions reduce target duplication and make the docker-bake.hcl more HCL-idiomatic (if there is such a thing).

[pinheadmz]

Looks good so far, one comment below. I'm going to run a bunch of builds on a remote server today in the background and will report back how it all goes.

[pinheadmz]

does this need amd and arm platforms?

[pinheadmz]

Oh I suppose not because it inherits. Looks like they get built in parallel too which is awesome:

[danvergara]

Yeah, it only rewrites what's needed for the cmake build, other than that, it inherits from the maintained-base target.

[danvergara]

Thanks @deadmanoz. Let me address those suggestions later today after work.

[pinheadmz]

should also mention --load or --push ?

Or actually, see src/warnet/image_build.py I'm not sure if these explicit docker buildx commands are necessary because of that abstract command...?

[danvergara]

Added some a few paragraphs on how to use those options.

[pinheadmz]

From this branch on Ubuntu/x86 I was able to build two autotools images (v0-16-1 and inv-5k) but failed on a cmake build (master) with an OOM, which is really wild since I have 24 GB RAM on this server... but I also have 32 CPUs, so I actually wonder if -j$(nproc) is doing too much ?

docker info looks like it has all the resources of the machine:

...

 CPUs: 32
 Total Memory: 22.91GiB
....

Also weird because I have built bitcoin with cmake outside of docker on this server just fine with -j 32

Here's the build error:

2616.2 c++: fatal error: Killed signal terminated program cc1plus
2616.2 compilation terminated.
2616.3 gmake[2]: *** [src/CMakeFiles/bitcoin_node.dir/build.make:331: src/CMakeFiles/bitcoin_node.dir/init.cpp.o] Error 1
2616.3 gmake[2]: *** Waiting for unfinished jobs....
2739.1 c++: fatal error: Killed signal terminated program cc1plus
2739.1 compilation terminated.
2739.1 gmake[2]: *** [src/CMakeFiles/bitcoin_node.dir/build.make:471: src/CMakeFiles/bitcoin_node.dir/net_processing.cpp.o] Error 1
3226.5 gmake[1]: *** [CMakeFiles/Makefile2:690: src/CMakeFiles/bitcoin_node.dir/all] Error 2
3586.5 [ 78%] Linking CXX static library libbitcoin_common.a
3603.8 [ 78%] Built target bitcoin_common
3603.8 gmake: *** [Makefile:136: all] Error 2
------
WARNING: No output specified for bitcoin-master target(s) with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile.dev:34
--------------------
  33 |
  34 | >>> RUN set -ex \
  35 | >>>     && cd /build \
  36 | >>>     && git clone --depth 1 "https://github.com/${REPO}" \
  37 | >>>     && cd bitcoin \
  38 | >>>     && git fetch --depth 1 origin "$COMMIT_SHA" \
  39 | >>>     && git checkout "$COMMIT_SHA" \
  40 | >>>     && git apply /tmp/isroutable.patch \
  41 | >>>     && git apply /tmp/addrman.patch \
  42 | >>>     && sed -i s:sys/fcntl.h:fcntl.h: src/compat/compat.h \
  43 | >>>     && cmake -B build \
  44 | >>>     -DCMAKE_INSTALL_PREFIX=${BITCOIN_PREFIX} \
  45 | >>>     ${BUILD_ARGS} \
  46 | >>>     && cmake --build build -j$(nproc) \
  47 | >>>     && cmake --install build \
  48 | >>>     && strip ${BITCOIN_PREFIX}/bin/bitcoin-cli \
  49 | >>>     && strip ${BITCOIN_PREFIX}/bin/bitcoind \
  50 | >>>     && rm -f ${BITCOIN_PREFIX}/lib/libbitcoinconsensus.a \
  51 | >>>     && rm -f ${BITCOIN_PREFIX}/lib/libbitcoinconsensus.so.0.0.0
  52 |
--------------------
ERROR: failed to solve: ResourceExhausted: process "/dev/.buildkit_qemu_emulator /bin/sh -c set -ex
     && cd /build     && git clone --depth 1 \"https://github.com/${REPO}\"     && cd bitcoin     && git fetch --depth 1 origin \"$COMMIT_SHA\"
     && git checkout \"$COMMIT_SHA\"     && git apply /tmp/isroutable.patch     && git apply /tmp/addrman.patch     && sed -i s:sys/fcntl.h:fcntl.h: src/compat/compat.h
     && cmake -B build     -DCMAKE_INSTALL_PREFIX=${BITCOIN_PREFIX}     ${BUILD_ARGS}     && cmake --build build -j$(nproc)
     && cmake --install build     && strip ${BITCOIN_PREFIX}/bin/bitcoin-cli     && strip ${BITCOIN_PREFIX}/bin/bitcoind     && rm -f ${BITCOIN_PREFIX}/lib/libbitcoinconsensus.a
     && rm -f ${BITCOIN_PREFIX}/lib/libbitcoinconsensus.so.0.0.0" did not complete successfully: cannot allocate memory

[pinheadmz]

Oh right, multiple builds in parallel... that does explain the resource consumption. Is there a work around? Or maybe do the builds in series instead?

[danvergara]

@pinheadmz have you built multiple targets at the same time for arm64? I think build in series is the way to go. I have pipelines that cross-build images at work (from Ubuntu/x86 to linux/arm64) that take up to 60 minutes.

Btw, do you know why the test suite failed? I did not touch other that the documentation in the latest commit

[pinheadmz]

Ill restart CI, looks like a flaky test. I haven't tried building multiple targets on this server outside of guix builds which are run in parallel and take hours.

[danvergara]

@pinheadmz any luck in series builds?

[danvergara]

Tonight, I built the master target for every platform available. I only did it for the arm64 but not for amd64 and arm7.
Fedora 41 with 32 GB of RAM. It took around an hour to complete. cc @pinheadmz

[pinheadmz]

machine is busy on a guix build for right now but I'll try again later. Do you know what the dockerx flags are to force serial builds?

[danvergara]

My bad, seems like there's no forced serial builds, only parallelism management and it is done via Buildkit configuration.

https://docs.docker.com/build/buildkit/configure/#max-parallelism

Another way might using be multiple docker enabled remote builder nodes:

https://www.ivonet.nl/2022/04/10/docker-multi-platform-build-with-buildx/

[pinheadmz]

Ok i set parallelism to = 1 and re-created the builder. That seems to be working now. So I'll have some locally baked warnet images to test here later today then if all goes well this is RFM

[pinheadmz]

ACK -- OK great thank you!