Speed Up Your CI With Docker (Github Actions)

How to share your Docker image across your jobs with GitHub Actions

This Article will use GitHub Actions as an example; if you are interested in doing the same with Travis-CI, please follow this Article.

The purpose is to create a Docker image from our source code and run our tests through that image. That way, we can speed up our test suite by running it in parallel. But how do we share the image created instead of creating it for each job? That is what we are looking at in this Article.

Github Actions is super popular at that time. If I like it a lot, I have to say it still has room for improvement; for instance, the lack of YAML functionality, like the fact we can’t anchor an alias to a node, makes the configuration lengthy and prone to errors. But as the project advances at an incredible pace, I have no doubt things will improve in no time.

UPDATE: Well, unfortunately, it seems that is not going to happen, check out this short article to see how to work around the lack of YAML Anchors.

We are going to explore two ways to achieve our goal here, Cache Registery and Artifact.

Using GitHub Actions Cache

The complete working file is here.

Github Actions give us access to a cache registry in this section. We are going to use it.

The configuration of GitHub Actions takes place in the file .github/workflows/main.yml. You can name the file whatever you want, and you can add as much as a workflow you need/want under the workflows directory. For the sake of simplicity, we will stick with the default name.

name: Docker Cached Image
on: [push,pull_request]
env:
  SERVICE_NAME: dummy_project

We declare our first section responsible for creating and storing the Docker Image we will use later. We are going step by step:

First of all, we need to check out the code.

jobs:
  image-build-and-cache:
  runs-on: ubuntu-latest
  steps:
      - name: Check Out Code
        uses: actions/checkout@v2

To use the Cache, we need to declare it.

...
env:
  IMAGE_CACHE_DIR: /tmp/cache/docker-image
  IMAGE_CACHE_KEY: cache-image
jobs:
  ...
  steps:
    ...
    - name: Create Cache Registry
        id: cache-docker-images
        uses: actions/cache@v2
        with:
          path: ${{ env.IMAGE_CACHE_DIR }}
          key: ${{ runner.os }}-docker-images-${{ steps.branch-commits.outputs.last_sha }}

We define the key of our Cache depending on the last commit. That way, we ensure our Cache will expire each time we check out a new code. Indeed we want our Cache for the lifetime of the CI builds. To get that SHA, we have several options, Github provides useful environment variables like $GITHUB_SHA, but we will use one of the last features of the new workflows instead for the sake of learning.

name: Docker Cached Image
...
jobs:
  image-build-and-cache:
  ...
  outputs:
    branch: ${{ steps.branch-commits.outputs.branch }}
    sha: ${{ steps.branch-commits.outputs.sha_short }}
  steps:
    ...
    - name: Get Current Branch and Commit
      id: branch-commits
      shell: bash
      run: |
        echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
        echo "::set-output name=sha_short::$(git rev-parse --short HEAD)"

We just declare our outputs for further use.

Now is the time to build and store our Docker Image.

...
env:
  IMAGE_CACHE_DIR: /tmp/cache/docker-image
  APP_IMAGE_TAG: dummy_project/app-ci:latest
  RUBY_VERSION: 3.0.0
jobs:
  ...
  steps:
    ...
    - name: Build Docker Image from Source Code
      if: steps.cache-docker-images.outputs.cache-hit != 'true'
      uses: docker/build-push-action@v2
      with:
        build-args: |
          ARG_RUBY_VERSION=${{ env.RUBY_VERSION }}
        context: .
        file: dockerfiles/Dockerfile-ci
        tags: ${{ env.APP_IMAGE_TAG }}
        outputs: type=docker,dest=${{ env.IMAGE_CACHE_DIR }}/image-app.tar

The build push action is going to create the Docker image and push the compressed image into the Cache; you can see the last command outputs: as an equivalent of:

docker save --output ${{ env.IMAGE_CACHE_DIR }}/image-app.tar ${{ env.SERVICE_NAME }}/app-ci:latest

We are now ready to run the other jobs using the just-created image.

name: Docker Cached Image
...
jobs:
  image-build-and-cache:
   ...
    steps:
      ...
  test-app:
    needs: image-build-and-cache
    runs-on: ubuntu-latest
    steps:
      - name: Declare Cache Registry
        id: cache-docker-images
        uses: actions/cache@v2
        with:
          path: ${{ env.IMAGE_CACHE_DIR }}
          key: ${{ runner.os }}-docker-images-${{ needs.image-build-and-cache.outputs.sha }}

The first thing to notice here is that the second job requires the first one, as you can see with the line:

needs: image-build-and-cache

That will ensure the first job is done before that job starts.

We need to declare our Cache before using it. Unfortunately, without the help of the definition of commons section, this will end up in an old fashion copy past of what we already defined in the first job.

Now we can reach the cache-store and load the previously created image.

name: Docker Cached Image
...
jobs:
  image-build-and-cache:
  ...
  test-app:
    needs: image-build-and-cache
    ...
    steps:
      ...
      - name: Docker load
        run: |
          docker load --input ${{ env.IMAGE_CACHE_DIR }}/image-app.tar

And that is it! We are now able to run whatever we need through our Docker container:

name: Docker Cached Image
...
jobs:
  image-build-and-cache:
  ...
  test-app:
    needs: image-build-and-cache
    ...
    steps:
      ...
      - name: Run Test
        run: |
          docker run --rm \
            --env DB_HOST=${{ env.SERVICE_NAME }}-db \
            --env DB_PASSWORD=${{ env.DB_PASSWORD }} \
            --env DB_USERNAME=root \
            --network=${{ env.SERVICE_NAME }}-bridge-docker-network \
            ${{ env.SERVICE_NAME }}/app-ci:${{ env.VERSION }} /bin/bash -c "bin/rails test"

You might have noticed the use of the keyword :if when building the Docker Image. Why are we doing that? Well, it is not that useful if you have only one workflow. However, if you run multiple workflows, once one creates the image and caches it, the remaining workflows will use the Cache and not build the image repeatedly.

We can’t expire the Cache manually, but GitHub will do it for us, check out the politic of eviction.

Using GitHub Artifacts

The complete working file is here.

As an alternative to using the Cache, we can use Artifacts. For that, you need to respective change the two declarations of the Cache by those:

name: Docker Cached Image
....
jobs:
  image-build-and-cache:
  ...
  steps:
    ...
    - name: Upload docker image
      uses: actions/upload-artifact@v2
      with:
        name: image-app.tar
        path: /tmp/image-app.tar
        retention-days: 1
    ...
  test-app:
    needs: image-build-and-cache
    ...
    steps:
      ...
        
      - name: Download docker image
        uses: actions/download-artifact@v2
        with:
          name: image-app.tar
          path: /tmp
      - name: Load docker image
        ...
      - name: Run Test
        ...

That’s it, and we can now run countless jobs against our built image.

Complete example here.