Skip to content

Commit 81e3aa1

Browse files
per1234per1234
committed
Make Linux build in a container for compatibility with older distros
Background ---------- The Linux build of Arduino IDE is dynamically linked against the libstdc++ and glibc shared libraries. This results in it having a dependency on the version of the libraries that happens to be present in the environment it is built in. Although newer versions of the shared libraries are compatible with executables linked against an older version, the reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example: ``` Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/per/Downloads/arduino-ide_nightly-20231006_Linux_64bit/resources/app/lib/backend/native/nsfw.node) ``` or: ``` Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/per/Downloads/arduino-ide_2.0.5-snapshot-90b1f67_Linux_64bit/resources/app/node_modules/nsfw/build/Release/nsfw.node) ``` We were originally able to achieve our targeted range of Linux distro compatibility by running the Linux build in the ubuntu-18.04 GitHub Actions hosted runner machine. Unfortunately GitHub stopped offering that runner machine. This meant we were forced to update to using the ubuntu-20.04 runner machine instead, which caused the loss of compatibility of the automatically generated builds with previously supported distro versions (e.g., Ubuntu 18.04). Since that time, the release builds have been produced manually, which is inefficient and prone to human error. Update ------ The identified solution to restoring fully automated builds with the target range of Linux distro compatibility is to run the build in a Docker container that provides the suitable environment. This means a combination of an older distro version with the modern versions of the development tool dependencies of the build. Such a combination was achieved by creating a bespoke image based on the ubuntu:18.04 image. The Dockerfile is hosted in the Arduino IDE repository in order to allow it to be maintained in parallel with the code and infrastructure. Image Publishing ---------------- A "Push Container Images" GitHub Actions continuous delivery workflow is added to push updated images to the GitHub Container registry when a commit that modifies relevant files is pushed to the main branch. This means the image does not have formally versioned tags and the IDE build uses the container that results from the configuration at the tip of the main branch at the time of the build. I think that is a reasonable approach in this use case where the image is targeted to a single application rather than intended to be used by multiple projects. Container Validation -------------------- The build workflow is configured to trigger on completion of that push workflow in order to provide validation of the IDE build using the updated container as well as the resulting tester builds of the IDE. A "Check Containers" GitHub Actions continuous integration workflow is added to provide basic validation for changes to the Dockerfile. It will automatically build the image and run the container on any push or pull request that modifies relevant files. Container Workflow Design ------------------------- With the goal of reusability, the image data is contained in a job matrix in the workflow to allow them to accommodate any number of arbitrary images. Build Workflow Configuration ---------------------------- A container property is added to the build job matrix data. If the container.image property is set to null, GitHub Actions will run the job directly in the runner environment instead of in a container. This allows us to produce the builds using either a container or a bare runner machine as is appropriate for each target. Unfortunately the latest v4 version of the actions/checkout action used to checkout the repository into the job environment has a dependency on a higher version of glibc than is provided by the Linux container. For this reason, the workflow is configured to use actions/checkout@v3 for the Linux build job. We will likely receive pull requests from Dependabot offering to update this outdated action dependency for the v4 and at each subsequent major version release of the action (which are not terribly frequent). We must decline the bump of the action in that specific step, but accept the bumps of all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this should not be too bothersome.
1 parent a8e63c8 commit 81e3aa1

File tree

4 files changed

+271
-2
lines changed

4 files changed

+271
-2
lines changed

Diff for: ‎.github/workflows/assets/linux.Dockerfile

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# The Arduino IDE Linux build workflow job runs in this container.
2+
# syntax=docker/dockerfile:1
3+
4+
FROM ubuntu:18.04
5+
6+
# See: https://unofficial-builds.nodejs.org/download/release/
7+
ARG node_version="18.17.1"
8+
9+
RUN \
10+
apt-get \
11+
--yes \
12+
update
13+
14+
# This is required to get add-apt-repository
15+
RUN \
16+
apt-get \
17+
--yes \
18+
install \
19+
"software-properties-common=0.96.24.32.22"
20+
21+
# Install Git
22+
# The PPA is required to get a modern version of Git. The version in the Ubuntu 18.04 package repository is 2.17.1,
23+
# while action/checkout@v3 requires 2.18 or higher.
24+
RUN \
25+
add-apt-repository \
26+
--yes \
27+
"ppa:git-core/ppa" && \
28+
apt-get \
29+
--yes \
30+
update && \
31+
\
32+
apt-get \
33+
--yes \
34+
install \
35+
"git=1:2.42.0-0ppa1~ubuntu18.04.1" && \
36+
\
37+
apt-get \
38+
--yes \
39+
purge \
40+
"software-properties-common"
41+
42+
# The repository path must be added to safe.directory, otherwise any Git operations on it would fail with a
43+
# "dubious ownership" error. actions/checkout configures this, but it is not applied to containers.
44+
RUN \
45+
git config \
46+
--add \
47+
--global \
48+
"safe.directory" "/__w/arduino-ide/arduino-ide"
49+
ENV \
50+
GIT_CONFIG_GLOBAL="/root/.gitconfig"
51+
52+
# Install Python
53+
# The Python installed by actions/setup-python has dependency on a higher version of glibc than available in the
54+
# ubuntu:18.04 container.
55+
RUN \
56+
apt-get \
57+
--yes \
58+
install \
59+
"python3.8-minimal=3.8.0-3ubuntu1~18.04.2" && \
60+
\
61+
ln \
62+
--symbolic \
63+
--force \
64+
"$(which python3.8)" \
65+
"/usr/bin/python3"
66+
67+
# Install Theia's package dependencies
68+
# These are pre-installed in the GitHub Actions hosted runner machines.
69+
RUN \
70+
apt-get \
71+
--yes \
72+
install \
73+
"libsecret-1-dev=0.18.6-1" \
74+
"libx11-dev=2:1.6.4-3ubuntu0.4" \
75+
"libxkbfile-dev=1:1.0.9-2"
76+
77+
# Install Node.js
78+
# It is necessary to use the "unofficial" linux-x64-glibc-217 build because the official Node.js 18.x is dynamically
79+
# linked against glibc 2.28, while Ubuntu 18.04 has glibc 2.27.
80+
ARG node_installation_path="/tmp/node-installation"
81+
ARG artifact_name="node-v${node_version}-linux-x64-glibc-217"
82+
RUN \
83+
mkdir "$node_installation_path" && \
84+
cd "$node_installation_path" && \
85+
\
86+
apt-get \
87+
--yes \
88+
install \
89+
"wget=1.19.4-1ubuntu2.2" && \
90+
\
91+
archive_name="${artifact_name}.tar.xz" && \
92+
wget \
93+
"https://unofficial-builds.nodejs.org/download/release/v${node_version}/${archive_name}" && \
94+
\
95+
apt-get \
96+
--yes \
97+
purge \
98+
"wget" && \
99+
\
100+
tar \
101+
--file="$archive_name" \
102+
--extract && \
103+
rm "$archive_name"
104+
ENV PATH="${PATH}:${node_installation_path}/${artifact_name}/bin"
105+
106+
# Install Yarn
107+
# Yarn is pre-installed in the GitHub Actions hosted runner machines.
108+
RUN \
109+
npm \
110+
install \
111+
--global \
112+
"[email protected]"

Diff for: ‎.github/workflows/build.yml

+40-2
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ on:
3535
- '*.md'
3636
schedule:
3737
- cron: '0 3 * * *' # run every day at 3AM (https://docs.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule)
38+
workflow_run:
39+
workflows:
40+
- Push Container Images
41+
branches:
42+
- main
43+
types:
44+
- completed
3845

3946
env:
4047
# See vars.GO_VERSION field of https://github.com/arduino/arduino-cli/blob/master/DistTasks.yml
@@ -49,6 +56,12 @@ env:
4956
# Human identifier for the job.
5057
name: Windows
5158
runs-on: windows-2019
59+
# The value is a string representing a JSON document.
60+
# Setting image value to null causes the job to run directly in the runner machine instead of in a container.
61+
container: |
62+
{
63+
\"image\": null
64+
}
5265
# Name of the secret that contains the certificate.
5366
certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX
5467
# Name of the secret that contains the certificate password.
@@ -68,7 +81,11 @@ env:
6881
name: Windows_X86-64_zip
6982
- config:
7083
name: Linux
71-
runs-on: ubuntu-20.04
84+
runs-on: ubuntu-latest
85+
container: |
86+
{
87+
\"image\": \"ghcr.io/arduino/arduino-ide/linux:main\"
88+
}
7289
mergeable-channel-file: 'false'
7390
artifacts:
7491
- path: '*Linux_64bit.zip'
@@ -78,6 +95,10 @@ env:
7895
- config:
7996
name: macOS x86
8097
runs-on: macos-latest
98+
container: |
99+
{
100+
\"image\": null
101+
}
81102
# APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from:
82103
# https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate
83104
certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
@@ -93,6 +114,10 @@ env:
93114
- config:
94115
name: macOS ARM
95116
runs-on: macos-latest-xlarge
117+
container: |
118+
{
119+
\"image\": null
120+
}
96121
certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
97122
certificate-password-secret: KEYCHAIN_PASSWORD
98123
certificate-extension: p12
@@ -255,20 +280,34 @@ jobs:
255280
matrix:
256281
config: ${{ fromJson(needs.select-targets.outputs.build-matrix) }}
257282
runs-on: ${{ matrix.config.runs-on }}
283+
container: ${{ fromJSON(matrix.config.container) }}
284+
defaults:
285+
run:
286+
# Avoid problems caused by different default shell for container jobs (sh) vs non-container jobs (bash).
287+
shell: bash
288+
258289
timeout-minutes: 90
259290

260291
steps:
261292
- name: Checkout
293+
if: fromJSON(matrix.config.container).image == null
262294
uses: actions/checkout@v4
263295

296+
- name: Checkout
297+
# actions/checkout@v4 has dependency on a higher version of glibc than available in the Linux container.
298+
if: fromJSON(matrix.config.container).image != null
299+
uses: actions/checkout@v3
300+
264301
- name: Install Node.js
302+
if: fromJSON(matrix.config.container).image == null
265303
uses: actions/setup-node@v3
266304
with:
267305
node-version: ${{ env.NODE_VERSION }}
268306
registry-url: 'https://registry.npmjs.org'
269307
cache: 'yarn'
270308

271309
- name: Install Python 3.x
310+
if: fromJSON(matrix.config.container).image == null
272311
uses: actions/setup-python@v4
273312
with:
274313
python-version: '3.x'
@@ -285,7 +324,6 @@ jobs:
285324
version: 3.x
286325

287326
- name: Package
288-
shell: bash
289327
env:
290328
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
291329
AC_USERNAME: ${{ secrets.AC_USERNAME }}

Diff for: ‎.github/workflows/check-containers.yml

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: Check Containers
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- ".github/workflows/check-containers.ya?ml"
7+
- "**.Dockerfile"
8+
- "**/Dockerfile"
9+
push:
10+
paths:
11+
- ".github/workflows/check-containers.ya?ml"
12+
- "**.Dockerfile"
13+
- "**/Dockerfile"
14+
repository_dispatch:
15+
schedule:
16+
# Run periodically to catch breakage caused by external changes.
17+
- cron: "0 7 * * MON"
18+
workflow_dispatch:
19+
20+
jobs:
21+
run:
22+
name: Run (${{ matrix.image.path }})
23+
runs-on: ubuntu-latest
24+
permissions: {}
25+
services:
26+
registry:
27+
image: registry:2
28+
ports:
29+
- 5000:5000
30+
31+
env:
32+
IMAGE_NAME: name/app:latest
33+
REGISTRY: localhost:5000
34+
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
image:
39+
- path: .github/workflows/assets/linux.Dockerfile
40+
41+
steps:
42+
- name: Checkout repository
43+
uses: actions/checkout@v4
44+
45+
- name: Build and push to local registry
46+
uses: docker/build-push-action@v5
47+
with:
48+
context: .
49+
file: ${{ matrix.image.path }}
50+
push: true
51+
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
52+
53+
- name: Run container
54+
run: |
55+
docker \
56+
run \
57+
--rm \
58+
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

Diff for: ‎.github/workflows/push-container-images.yml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
name: Push Container Images
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- ".github/workflows/push-container-images.ya?ml"
7+
push:
8+
paths:
9+
- ".github/workflows/push-container-images.ya?ml"
10+
- "**.Dockerfile"
11+
- "**/Dockerfile"
12+
repository_dispatch:
13+
schedule:
14+
# Run periodically to catch breakage caused by external changes.
15+
- cron: "0 8 * * MON"
16+
workflow_dispatch:
17+
18+
jobs:
19+
push:
20+
name: Push (${{ matrix.image.name }})
21+
if: github.repository == 'arduino/arduino-ide'
22+
runs-on: ubuntu-latest
23+
permissions:
24+
contents: read
25+
packages: write
26+
27+
strategy:
28+
fail-fast: false
29+
matrix:
30+
image:
31+
- path: .github/workflows/assets/linux.Dockerfile
32+
name: ${{ github.repository }}/linux
33+
registry: ghcr.io
34+
35+
steps:
36+
- name: Checkout repository
37+
uses: actions/checkout@v4
38+
39+
- name: Log in to the Container registry
40+
uses: docker/login-action@v3
41+
with:
42+
password: ${{ secrets.GITHUB_TOKEN }}
43+
registry: ${{ matrix.image.registry }}
44+
username: ${{ github.repository_owner }}
45+
46+
- name: Extract metadata for image
47+
id: metadata
48+
uses: docker/metadata-action@v5
49+
with:
50+
images: ${{ matrix.image.registry }}/${{ matrix.image.name }}
51+
52+
- name: Build and push image
53+
uses: docker/build-push-action@v5
54+
with:
55+
context: .
56+
file: ${{ matrix.image.path }}
57+
labels: ${{ steps.metadata.outputs.labels }}
58+
# Workflow is triggered on relevant events for the sake of a "dry run" validation but image is only pushed to
59+
# registry on commit to the main branch.
60+
push: ${{ github.ref == 'refs/heads/main' }}
61+
tags: ${{ steps.metadata.outputs.tags }}

0 commit comments

Comments
 (0)