Skip to content

Commit 0ad9b97

Browse files
per1234per1234
committed
Make Linux build in a container for compatibility with older distros
Background ---------- The Linux build of Arduino IDE is dynamically linked against the libstdc++ and glibc shared libraries. This results in it having a dependency on the version of the libraries that happens to be present in the environment it is built in. Although newer versions of the shared libraries are compatible with executables linked against an older version, the reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example: ``` Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/per/Downloads/arduino-ide_nightly-20231006_Linux_64bit/resources/app/lib/backend/native/nsfw.node) ``` or: ``` Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/per/Downloads/arduino-ide_2.0.5-snapshot-90b1f67_Linux_64bit/resources/app/node_modules/nsfw/build/Release/nsfw.node) ``` We were originally able to achieve our targeted range of Linux distro compatibility by running the Linux build in the ubuntu-18.04 GitHub Actions hosted runner machine. Unfortunately GitHub stopped offering that runner machine. This meant we were forced to update to using the ubuntu-20.04 runner machine instead, which caused the loss of compatibility of the automatically generated builds with previously supported distro versions (e.g., Ubuntu 18.04). Since that time, the release builds have been produced manually, which is inefficient and prone to human error. Update ------ The identified solution to restoring fully automated builds with the target range of Linux distro compatibility is to run the build in a Docker container that provides the suitable environment. This means a combination of an older distro version with the modern versions of the development tool dependencies of the build. Such a combination was achieved by creating a bespoke image based on the ubuntu:18.04 image. The Dockerfile is hosted in the Arduino IDE repository in order to allow it to be maintained in parallel with the code and infrastructure. Image Publishing ---------------- A "Push Container Images" GitHub Actions continuous delivery workflow is added to push updated images to the GitHub Container registry when a commit that modifies relevant files is pushed to the main branch. This means the image does not have formally versioned tags and the IDE build uses the container that results from the configuration at the tip of the main branch at the time of the build. I think that is a reasonable approach in this use case where the image is targeted to a single application rather than intended to be used by multiple projects. Container Validation -------------------- The build workflow is configured to trigger on completion of that push workflow in order to provide validation of the IDE build using the updated container as well as the resulting tester builds of the IDE. A "Check Containers" GitHub Actions continuous integration workflow is added to provide basic validation for changes to the Dockerfile. It will automatically build the image and run the container on any push or pull request that modifies relevant files. Container Workflow Design ------------------------- With the goal of reusability, the image data is contained in a job matrix in the workflow to allow them to accommodate any number of arbitrary images. Build Workflow Configuration ---------------------------- A container property is added to the build job matrix data. If the container.image property is set to null, GitHub Actions will run the job directly in the runner environment instead of in a container. This allows us to produce the builds using either a container or a bare runner machine as is appropriate for each target. Unfortunately the latest v4 version of the actions/checkout action used to checkout the repository into the job environment has a dependency on a higher version of glibc than is provided by the Linux container. For this reason, the workflow is configured to use actions/checkout@v3 for the Linux build job. We will likely receive pull requests from Dependabot offering to update this outdated action dependency for the v4 and at each subsequent major version release of the action (which are not terribly frequent). We must decline the bump of the action in that specific step, but accept the bumps of all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this should not be too bothersome.
1 parent a8e63c8 commit 0ad9b97

File tree

4 files changed

+274
-2
lines changed

4 files changed

+274
-2
lines changed

Diff for: ‎.github/workflows/assets/linux.Dockerfile

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# The Arduino IDE Linux build workflow job runs in this container.
2+
# syntax=docker/dockerfile:1
3+
4+
FROM ubuntu:18.04
5+
6+
# See: https://unofficial-builds.nodejs.org/download/release/
7+
ARG node_version="18.17.1"
8+
9+
RUN \
10+
apt-get \
11+
--yes \
12+
update
13+
14+
# This is required to get add-apt-repository
15+
RUN \
16+
apt-get \
17+
--yes \
18+
install \
19+
"software-properties-common=0.96.24.32.22"
20+
21+
# Install Git
22+
# The PPA is required to get a modern version of Git. The version in the Ubuntu 18.04 package repository is 2.17.1,
23+
# while action/checkout@v3 requires 2.18 or higher.
24+
RUN \
25+
add-apt-repository \
26+
--yes \
27+
"ppa:git-core/ppa" && \
28+
apt-get \
29+
--yes \
30+
update && \
31+
\
32+
apt-get \
33+
--yes \
34+
install \
35+
"git=1:2.42.0-0ppa1~ubuntu18.04.1" && \
36+
\
37+
apt-get \
38+
--yes \
39+
purge \
40+
"software-properties-common"
41+
42+
# The repository path must be added to safe.directory, otherwise any Git operations on it would fail with a
43+
# "dubious ownership" error. actions/checkout configures this, but it is not applied to containers.
44+
RUN \
45+
git config \
46+
--add \
47+
--global \
48+
"safe.directory" "/__w/arduino-ide/arduino-ide"
49+
ENV \
50+
GIT_CONFIG_GLOBAL="/root/.gitconfig"
51+
52+
# Install Python
53+
# The Python installed by actions/setup-python has dependency on a higher version of glibc than available in the
54+
# ubuntu:18.04 container.
55+
RUN \
56+
apt-get \
57+
--yes \
58+
install \
59+
"python3.8-minimal=3.8.0-3ubuntu1~18.04.2" && \
60+
\
61+
ln \
62+
--symbolic \
63+
--force \
64+
"$(which python3.8)" \
65+
"/usr/bin/python3"
66+
67+
# Install Theia's package dependencies
68+
# These are pre-installed in the GitHub Actions hosted runner machines.
69+
RUN \
70+
apt-get \
71+
--yes \
72+
install \
73+
"libsecret-1-dev=0.18.6-1" \
74+
"libx11-dev=2:1.6.4-3ubuntu0.4" \
75+
"libxkbfile-dev=1:1.0.9-2"
76+
77+
# Install Node.js
78+
# It is necessary to use the "unofficial" linux-x64-glibc-217 build because the official Node.js 18.x is dynamically
79+
# linked against glibc 2.28, while Ubuntu 18.04 has glibc 2.27.
80+
ARG node_installation_path="/tmp/node-installation"
81+
ARG artifact_name="node-v${node_version}-linux-x64-glibc-217"
82+
RUN \
83+
mkdir "$node_installation_path" && \
84+
cd "$node_installation_path" && \
85+
\
86+
apt-get \
87+
--yes \
88+
install \
89+
"wget=1.19.4-1ubuntu2.2" && \
90+
\
91+
archive_name="${artifact_name}.tar.xz" && \
92+
wget \
93+
"https://unofficial-builds.nodejs.org/download/release/v${node_version}/${archive_name}" && \
94+
\
95+
apt-get \
96+
--yes \
97+
purge \
98+
"wget" && \
99+
\
100+
tar \
101+
--file="$archive_name" \
102+
--extract && \
103+
rm "$archive_name"
104+
ENV PATH="${PATH}:${node_installation_path}/${artifact_name}/bin"
105+
106+
# Install Yarn
107+
# Yarn is pre-installed in the GitHub Actions hosted runner machines.
108+
RUN \
109+
npm \
110+
install \
111+
--global \
112+
"[email protected]"

Diff for: ‎.github/workflows/build.yml

+34-2
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ on:
3535
- '*.md'
3636
schedule:
3737
- cron: '0 3 * * *' # run every day at 3AM (https://docs.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule)
38+
workflow_run:
39+
workflows:
40+
- Push Container Images
41+
branches:
42+
- main
43+
types:
44+
- completed
3845

3946
env:
4047
# See vars.GO_VERSION field of https://github.com/arduino/arduino-cli/blob/master/DistTasks.yml
@@ -49,6 +56,10 @@ env:
4956
# Human identifier for the job.
5057
name: Windows
5158
runs-on: windows-2019
59+
# The value is a string representing a JSON document.
60+
# Setting this to null causes the job to run directly in the runner machine instead of in a container.
61+
container: |
62+
null
5263
# Name of the secret that contains the certificate.
5364
certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX
5465
# Name of the secret that contains the certificate password.
@@ -68,7 +79,11 @@ env:
6879
name: Windows_X86-64_zip
6980
- config:
7081
name: Linux
71-
runs-on: ubuntu-20.04
82+
runs-on: ubuntu-latest
83+
container: |
84+
{
85+
\"image\": \"ghcr.io/arduino/arduino-ide/linux:main\"
86+
}
7287
mergeable-channel-file: 'false'
7388
artifacts:
7489
- path: '*Linux_64bit.zip'
@@ -78,6 +93,8 @@ env:
7893
- config:
7994
name: macOS x86
8095
runs-on: macos-latest
96+
container: |
97+
null
8198
# APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from:
8299
# https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate
83100
certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
@@ -93,6 +110,8 @@ env:
93110
- config:
94111
name: macOS ARM
95112
runs-on: macos-latest-xlarge
113+
container: |
114+
null
96115
certificate-secret: APPLE_SIGNING_CERTIFICATE_P12
97116
certificate-password-secret: KEYCHAIN_PASSWORD
98117
certificate-extension: p12
@@ -255,20 +274,34 @@ jobs:
255274
matrix:
256275
config: ${{ fromJson(needs.select-targets.outputs.build-matrix) }}
257276
runs-on: ${{ matrix.config.runs-on }}
277+
container: ${{ fromJSON(matrix.config.container) }}
278+
defaults:
279+
run:
280+
# Avoid problems caused by different default shell for container jobs (sh) vs non-container jobs (bash).
281+
shell: bash
282+
258283
timeout-minutes: 90
259284

260285
steps:
261286
- name: Checkout
287+
if: fromJSON(matrix.config.container).image == null
262288
uses: actions/checkout@v4
263289

290+
- name: Checkout
291+
# actions/checkout@v4 has dependency on a higher version of glibc than available in the Linux container.
292+
if: fromJSON(matrix.config.container).image != null
293+
uses: actions/checkout@v3
294+
264295
- name: Install Node.js
296+
if: fromJSON(matrix.config.container).image == null
265297
uses: actions/setup-node@v3
266298
with:
267299
node-version: ${{ env.NODE_VERSION }}
268300
registry-url: 'https://registry.npmjs.org'
269301
cache: 'yarn'
270302

271303
- name: Install Python 3.x
304+
if: fromJSON(matrix.config.container).image == null
272305
uses: actions/setup-python@v4
273306
with:
274307
python-version: '3.x'
@@ -285,7 +318,6 @@ jobs:
285318
version: 3.x
286319

287320
- name: Package
288-
shell: bash
289321
env:
290322
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
291323
AC_USERNAME: ${{ secrets.AC_USERNAME }}

Diff for: ‎.github/workflows/check-containers.yml

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: Check Containers
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- ".github/workflows/check-containers.ya?ml"
7+
- "**.Dockerfile"
8+
- "**/Dockerfile"
9+
push:
10+
paths:
11+
- ".github/workflows/check-containers.ya?ml"
12+
- "**.Dockerfile"
13+
- "**/Dockerfile"
14+
repository_dispatch:
15+
schedule:
16+
# Run periodically to catch breakage caused by external changes.
17+
- cron: "0 7 * * MON"
18+
workflow_dispatch:
19+
20+
jobs:
21+
run:
22+
name: Run (${{ matrix.image.path }})
23+
runs-on: ubuntu-latest
24+
permissions: {}
25+
services:
26+
registry:
27+
image: registry:2
28+
ports:
29+
- 5000:5000
30+
31+
env:
32+
IMAGE_NAME: name/app:latest
33+
REGISTRY: localhost:5000
34+
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
image:
39+
- path: .github/workflows/assets/linux.Dockerfile
40+
41+
steps:
42+
- name: Checkout repository
43+
uses: actions/checkout@v4
44+
45+
- name: Build and push to local registry
46+
uses: docker/build-push-action@v5
47+
with:
48+
context: .
49+
file: ${{ matrix.image.path }}
50+
push: true
51+
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
52+
53+
- name: Run container
54+
run: |
55+
docker \
56+
run \
57+
--rm \
58+
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

Diff for: ‎.github/workflows/push-container-images.yml

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
name: Push Container Images
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- ".github/workflows/push-container-images.ya?ml"
7+
push:
8+
paths:
9+
- ".github/workflows/push-container-images.ya?ml"
10+
- "**.Dockerfile"
11+
- "**/Dockerfile"
12+
repository_dispatch:
13+
schedule:
14+
# Run periodically to catch breakage caused by external changes.
15+
- cron: "0 8 * * MON"
16+
workflow_dispatch:
17+
18+
jobs:
19+
push:
20+
name: Push (${{ matrix.image.name }})
21+
# Only run the job when GITHUB_TOKEN has the privileges required for Container registry login.
22+
if: >
23+
(
24+
github.event_name != 'pull_request' &&
25+
github.repository == 'arduino/arduino-ide'
26+
) ||
27+
(
28+
github.event_name == 'pull_request' &&
29+
github.event.pull_request.head.repo.full_name == 'arduino/arduino-ide'
30+
)
31+
runs-on: ubuntu-latest
32+
permissions:
33+
contents: read
34+
packages: write
35+
36+
strategy:
37+
fail-fast: false
38+
matrix:
39+
image:
40+
- path: .github/workflows/assets/linux.Dockerfile
41+
name: ${{ github.repository }}/linux
42+
registry: ghcr.io
43+
44+
steps:
45+
- name: Checkout repository
46+
uses: actions/checkout@v4
47+
48+
- name: Log in to the Container registry
49+
uses: docker/login-action@v3
50+
with:
51+
password: ${{ secrets.GITHUB_TOKEN }}
52+
registry: ${{ matrix.image.registry }}
53+
username: ${{ github.repository_owner }}
54+
55+
- name: Extract metadata for image
56+
id: metadata
57+
uses: docker/metadata-action@v5
58+
with:
59+
images: ${{ matrix.image.registry }}/${{ matrix.image.name }}
60+
61+
- name: Build and push image
62+
uses: docker/build-push-action@v5
63+
with:
64+
context: .
65+
file: ${{ matrix.image.path }}
66+
labels: ${{ steps.metadata.outputs.labels }}
67+
# Workflow is triggered on relevant events for the sake of a "dry run" validation but image is only pushed to
68+
# registry on commit to the main branch.
69+
push: ${{ github.ref == 'refs/heads/main' }}
70+
tags: ${{ steps.metadata.outputs.tags }}

0 commit comments

Comments
 (0)