Michael Herman This article takes a look at how to speed up your Docker-based builds on CircleCI, GitLab CI, and GitHub Actions with Docker layer Caching and BuildKit.
Contents
Docker Layer Caching
Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let's take a look at a quick example.
Dockerfile:
# pull base image FROM python:3.12.2-slim # install netcat RUN apt-get update && \ apt-get -y install netcat && \ apt-get clean # set working directory WORKDIR /usr/src/app # install requirements COPY ./requirements.txt . RUN pip install -r requirements.txt # add app COPY . . # run server CMD gunicorn -b 0.0.0.0:5000 manage:app You can find the full source code for this project in the docker-ci-cache repo on GitHub.
The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:
[+] Building 0.4s (12/12) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 37B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 35B 0.0s => [internal] load metadata for docker.io/library/python:3.12.2-slim 0.3s => [internal] load build context 0.0s => => transferring context: 555B 0.0s => [1/7] FROM docker.io/library/python:3.12.2-slim@sha256:bdefda2b80c5b4d993ef83d2445d81b2b894bf627b62bd7b0f01244de2b6a 0.0s => CACHED [2/7] RUN apt-get update && apt-get -y install netcat && apt-get clean 0.0s => CACHED [3/7] WORKDIR /usr/src/app 0.0s => CACHED [4/7] COPY ./requirements.txt . 0.0s => CACHED [5/7] RUN pip install -r requirements.txt 0.0s => CACHED [6/7] COPY project . 0.0s => CACHED [7/7] COPY manage.py . 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:2b8b7c5a6d1b77d5bcd689ab265b0281ad531bd2e34729cff82285f5abdcb59f 0.0s => => naming to docker.io/library/cache 0.0s Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:
=> [6/7] COPY project . => [7/7] COPY manage.py . To avoid invalidating the cache:
- Start your Dockerfile with commands that are less likely to change
- Place commands that are more likely to change (like
COPY . .) as late as possible - Add only the necessary files (use a .dockerignore file)
For more tips and best practices, check out the Docker Best Practices for Python Developers article.
BuildKit
If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder, in place of the traditional image builder back-end inside the Docker engine. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.
Example:
$ docker pull mjhea0/docker-ci-cache:latest $ docker build --tag mjhea0/docker-ci-cache:latest . With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.
To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1. Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.
Example:
export DOCKER_BUILDKIT=1 # Build and cache image $ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE=1 . # Build image from remote cache $ docker build --cache-from mjhea0/docker-ci-cache:latest . CI Environments
Since CI platforms provide a fresh environment for every build, you'll need to use a remote image registry as the source of the cache for BuildKit's layer caching.
Steps:
-
Log in to the image registry (like Docker Hub, Elastic Container Registry (ECR), and Quay, to name a few).
It's worth noting that both GitLab and GitHub have their own registries for use within your repositories (both public and private) on their platforms -- GitLab Container Registry and GitHub Packages, respectively.
-
Use Docker build's
--cache-fromoption to use the existing image as the cache source. - Push the new image to the registry if the build is successful.
Let's look at how to do this on CircleCI, GitLab CI, and GitHub Actions, using both single and multi-stage Docker builds with and without Docker Compose. Each of the examples use Docker Hub as the image registry with REGISTRY_USER and REGISTRY_PASS set as variables in the CI builds in order to push to and pull from the registry.
Make sure to set
REGISTRY_USERandREGISTRY_PASSas environment variables in the build environment:
Single-stage Builds
CircleCI:
# _config-examples/single-stage/circle.yml version: 2.1 jobs: build: machine: image: ubuntu-2204:2024.01.1 environment: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 steps: - checkout - run: name: Log in to docker hub command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run: name: Build from dockerfile command: | docker build \ --cache-from $CACHE_IMAGE:latest \ --tag $CACHE_IMAGE:latest \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run: name: Push to docker hub command: docker push $CACHE_IMAGE:latest GitLab CI:
# _config-examples/single-stage/.gitlab-ci.yml image: docker:stable services: - docker:dind variables: DOCKER_DRIVER: overlay2 CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 stages: - build docker-build: stage: build before_script: - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script: - docker build --cache-from $CACHE_IMAGE:latest --tag $CACHE_IMAGE:latest --file ./Dockerfile --build-arg BUILDKIT_INLINE_CACHE=1 "." after_script: - docker push $CACHE_IMAGE:latest GitHub Actions:
# _config-examples/single-stage/github.yml name: Docker Build on: [push] env: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 jobs: build: name: Build Docker Image runs-on: ubuntu-latest steps: - name: Checkout master uses: actions/checkout@v4 - name: Log in to docker hub run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name: Build from dockerfile run: | docker build \ --cache-from $CACHE_IMAGE:latest \ --tag $CACHE_IMAGE:latest \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name: Push to docker hub run: docker push $CACHE_IMAGE:latest Compose
If you're using Docker Compose, you can add the cache_from option to the compose file, which maps back to the docker build --cache-from <image> command when you run docker-compose build.
Example:
version: '3.8' services: web: build: context: . cache_from: - mjhea0/docker-ci-cache:latest image: mjhea0/docker-ci-cache:latest To take advantage of BuildKit, make sure you're using a version of Docker Compose >= 1.25.0. To enable BuildKit, set the DOCKER_BUILDKIT and COMPOSE_DOCKER_CLI_BUILD environment variables to 1. Then, again, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.
CircleCI:
# _config-examples/single-stage/compose/circle.yml version: 2.1 jobs: build: machine: image: ubuntu-2204:2024.01.1 environment: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 steps: - checkout - run: name: Log in to docker hub command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run: name: Build images command: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 - run: name: Push to docker hub command: docker push $CACHE_IMAGE:latest GitLab CI:
# _config-examples/single-stage/compose/.gitlab-ci.yml image: docker/compose:latest services: - docker:dind variables: DOCKER_DRIVER: overlay2 CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 stages: - build docker-build: stage: build before_script: - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script: - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 after_script: - docker push $CACHE_IMAGE:latest GitHub Actions:
# _config-examples/single-stage/compose/github.yml name: Docker Build on: [push] env: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 jobs: build: name: Build Docker Image runs-on: ubuntu-latest steps: - name: Checkout master uses: actions/checkout@v4 - name: Log in to docker hub run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name: Build Docker images run: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 - name: Push to docker hub run: docker push $CACHE_IMAGE:latest Multi-stage Builds
With the multi-stage build pattern, you'll have to apply the same workflow (build, then push) for each intermediate stage since those images are discarded before the final image is created. The --target option can be used to build each stage of the multi-stage build separately.
Dockerfile.multi:
# base FROM python:3.12.2 as base COPY ./requirements.txt / RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt # stage FROM python:3.12.2-slim RUN apt-get update && \ apt-get -y install netcat && \ apt-get clean WORKDIR /usr/src/app COPY --from=base /wheels /wheels COPY --from=base requirements.txt . RUN pip install --no-cache /wheels/* COPY . /usr/src/app CMD gunicorn -b 0.0.0.0:5000 manage:app CircleCI:
# _config-examples/multi-stage/circle.yml version: 2.1 jobs: build: machine: image: ubuntu-2204:2024.01.1 environment: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 steps: - checkout - run: name: Log in to docker hub command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run: name: Build base from dockerfile command: | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run: name: Build stage from dockerfile command: | docker build \ --cache-from $CACHE_IMAGE:base \ --cache-from $CACHE_IMAGE:stage \ --tag $CACHE_IMAGE:stage \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run: name: Push base image to docker hub command: docker push $CACHE_IMAGE:base - run: name: Push stage image to docker hub command: docker push $CACHE_IMAGE:stage GitLab CI:
# _config-examples/multi-stage/.gitlab-ci.yml image: docker:stable services: - docker:dind variables: DOCKER_DRIVER: overlay2 CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 stages: - build docker-build: stage: build before_script: - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script: - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker build --cache-from $CACHE_IMAGE:base --cache-from $CACHE_IMAGE:stage --tag $CACHE_IMAGE:stage --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." after_script: - docker push $CACHE_IMAGE:stage GitHub Actions:
# _config-examples/multi-stage/github.yml name: Docker Build on: [push] env: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 jobs: build: name: Build Docker Image runs-on: ubuntu-latest steps: - name: Checkout master uses: actions/checkout@v4 - name: Log in to docker hub run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name: Build base from dockerfile run: | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name: Build stage from dockerfile run: | docker build \ --cache-from $CACHE_IMAGE:base \ --cache-from $CACHE_IMAGE:stage \ --tag $CACHE_IMAGE:stage \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name: Push base image to docker hub run: docker push $CACHE_IMAGE:base - name: Push stage image to docker hub run: docker push $CACHE_IMAGE:stage Compose
Example compose file:
version: '3.8' services: web: build: context: . cache_from: - mjhea0/docker-ci-cache:stage image: mjhea0/docker-ci-cache:stage CircleCI:
# _config-examples/multi-stage/compose/circle.yml version: 2.1 jobs: build: machine: image: ubuntu-2204:2024.01.1 environment: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 steps: - checkout - run: name: Log in to docker hub command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run: name: Build base from dockerfile command: | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run: name: Build Docker images command: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 - run: name: Push base image to docker hub command: docker push $CACHE_IMAGE:base - run: name: Push stage image to docker hub command: docker push $CACHE_IMAGE:stage GitLab CI:
# _config-examples/multi-stage/compose/.gitlab-ci.yml image: docker/compose:latest services: - docker:dind variables: DOCKER_DRIVER: overlay CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 stages: - build docker-build: stage: build before_script: - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script: - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 after_script: - docker push $CACHE_IMAGE:base - docker push $CACHE_IMAGE:stage GitHub Actions:
# _config-examples/multi-stage/compose/github.yml name: Docker Build on: [push] env: CACHE_IMAGE: mjhea0/docker-ci-cache DOCKER_BUILDKIT: 1 COMPOSE_DOCKER_CLI_BUILD: 1 jobs: build: name: Build Docker Image runs-on: ubuntu-latest steps: - name: Checkout master uses: actions/checkout@v4 - name: Log in to docker hub run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name: Build base from dockerfile run: | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name: Build images run: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 - name: Push base image to docker hub run: docker push $CACHE_IMAGE:base - name: Push stage image to docker hub run: docker push $CACHE_IMAGE:stage Conclusion
The caching strategies outlined in this article should work well for single-stage builds and multi-stage builds with two or three stages.
Each stage added to a build step requires a new build and push along with the addition of the --cache-from options for each parent stage. Thus, each new stage will add more clutter, making the CI file increasingly more difficult to read. Fortunately, BuildKit supports multi-stage builds with Docker layer caching built using a single stage. Review the following articles for more info on such advanced BuildKit patterns:
- Advanced Dockerfiles: Faster Builds and Smaller Images Using BuildKit and Multistage Builds
- Docker build cache sharing on multi-hosts with BuildKit and buildx
- Speed up multi-stage Docker builds in CI/CD with Buildkit’s registry cache
Finally, it's important to note that while caching may speed up your CI builds, you should re-build your images without cache from time to time in order to download the latest OS patches and security updates. For more on this, review this thread.
--
The code can be found in the docker-ci-cache repo:
Cheers!