Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerfile GHA don't reuse base images #1978

Closed
bbockelm opened this issue Feb 1, 2025 · 2 comments · Fixed by #1972
Closed

Dockerfile GHA don't reuse base images #1978

bbockelm opened this issue Feb 1, 2025 · 2 comments · Fixed by #1972
Assignees
Labels
critical High priority for next release infrastructure GitHub Actions, Release management, and CI internal Internal code improvements, not user-facing
Milestone

Comments

@bbockelm
Copy link
Collaborator

bbockelm commented Feb 1, 2025

I noticed (at least for PR builds) that the build-server-images matrix rebuilds the base image each time.

That is, we currently build the base image and, if successful, we rebuild the base image 8 additional times (one for each derivative of the base image) in parallel. That's around an hour of GitHub action minutes per PR ... which is not so great.

Either pass the images between the build steps or let's remove the parallelism (so we're only building twice!).

@bbockelm bbockelm added critical High priority for next release infrastructure GitHub Actions, Release management, and CI internal Internal code improvements, not user-facing labels Feb 1, 2025
@bbockelm bbockelm added this to the v7.14 milestone Feb 1, 2025
@brianaydemir
Copy link
Contributor

brianaydemir commented Feb 5, 2025

Using this 7.13.0-rc.1 action run as a point of reference:

  • The build-base-image step is successfully caching the final-stage image.

  • The build-base-image step is failing to cache the Pelican binaries because it doesn't depend on pelican-build.

As a result, when the action goes through the build matrix, it ends up building the Pelican binary eight times.

Actually, final-stage does depend on pelican-build. The problem is that pelican-build depends on the entire GitHub workspace (via COPY . .), which includes the .git directory, which I believe contains files with embedded timestamps. Because each job contains its own checkout of the repository, none of them can use the build cache for the Pelican binary.

@brianaydemir brianaydemir linked a pull request Feb 5, 2025 that will close this issue
@brianaydemir
Copy link
Contributor

brianaydemir commented Feb 6, 2025

The other issue is that sometimes, the attempt to cache image layers straight up fails, as in this run:

Image Image

As a result, everything in the matrix ends up getting built from scratch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
critical High priority for next release infrastructure GitHub Actions, Release management, and CI internal Internal code improvements, not user-facing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants