Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use caches and staged builds in docker images and use buildkit in the CI #69

Merged
merged 4 commits into from
Nov 22, 2021

Conversation

ml-evs
Copy link
Member

@ml-evs ml-evs commented Nov 19, 2021

This PR adds staged builds to the docker images for the API and app, so that hot-reloading dev servers can also be used locally with Docker-based development environments.

It also:

  • adds several optimizations for building docker images locally (package manager caches).
  • tries to fix some of the issues mentioned in Speed up CI tests #44, by using BuildKit to accelerate docker-compose builds. The total CI time is now roughly halved (6 mins from 11 mins, though billable time is still 10 mins so most of the work is being done by the parallel l builds).

I tried for hours to get layer caching to work between runs, but tools either support buildkit or docker-compose (not both) and even loading from the cache itself is a bit slow (see https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md). When we go open source, we could use GitHub's free container registry as a cache.

@ml-evs ml-evs force-pushed the ml-evs/docker_changes branch 6 times, most recently from 610ebbf to 66fa80a Compare November 20, 2021 14:07
@ml-evs ml-evs changed the title Use staged builds in docker images Use caches and staged builds in docker images and use buildkit in the CI Nov 20, 2021
@ml-evs ml-evs marked this pull request as ready for review November 20, 2021 14:18
@ml-evs ml-evs requested a review from jdbocarsly as a code owner November 20, 2021 14:18
@ml-evs ml-evs mentioned this pull request Nov 20, 2021
@ml-evs ml-evs force-pushed the ml-evs/docker_changes branch from 57d72f9 to 66fa80a Compare November 20, 2021 17:59
- Add 'dev' and 'prod' profiles for use with docker-compose
- Use package manager caches for pip and yarn within layers for faster
local builds
- Only install minimal deps for production builds
@ml-evs ml-evs force-pushed the ml-evs/docker_changes branch from 33299bf to 07a1b73 Compare November 22, 2021 10:47
@ml-evs ml-evs force-pushed the ml-evs/docker_changes branch from 07a1b73 to 396f82a Compare November 22, 2021 10:59
@ml-evs
Copy link
Member Author

ml-evs commented Nov 22, 2021

Not sure how much you can usefully review this @jdbocarsly, so feel free to take a look but I will merge tonight unless I hear otherwise.

@ml-evs ml-evs merged commit cea00cb into main Nov 22, 2021
@ml-evs ml-evs deleted the ml-evs/docker_changes branch October 5, 2022 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant