Speed Up Your CI With Docker (Travis-CI)
How to share your Docker image across your jobs with Travis-CI
If you choose to run all your CI with containers, you might want to share your built image across your jobs to save time.
This Article will use Travis-CI as an example; if you are interested in doing the same with GitHub Actions, please follow this Article.
The purpose is to create a Docker image from our source code and run our tests through that image. That way, we can speed up our test suite by running it in parallel. But how share the image created instead of creating it for each job? That is what we are looking at in this Article.
The complete working file is here.
Travis-CI comes with a handful of virtual environments to help us run our tests with the needed tools and languages set up for us, but we can choose the minimal one as we only need Docker. For more details, check out the official documentation.
In the main section of our `.travis.yml`, we start like that:
dist: focal # Ubuntu 20.04 LTSlanguage: minimal
We are not interested in building anything in the minimal virtual environment. We can skip those steps:
git:
clone: falsebefore_install: skip
install: skip
But we want to get docker services up and running.
services:
- dockerThe only steps we want to get in the main section will be the before and the after script to start and stop the services needed to execute the tests. Let say, for instance. We need a database and key-value data stores.
env:
global:
- DATABASE_PASSWORD=db-password-123
- APP_NAME=acmebefore_script:
- docker network create ${APP_NAME}-bridge-docker-network - |
echo "Start Mysql" && \
docker run --rm --detach \
--name ${APP_NAME}-db \
--env MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD} \
--network=${APP_NAME}-bridge-docker-network \
mysql:latest - |
echo "Start Redis" && \
docker run --rm --detach \
--name ${APP_NAME}-redis \
--network=${APP_NAME}-bridge-docker-network \
redis:latestafter_script:
- |
echo "Kill Docker Containers" && \
docker kill $(docker ps -q)
An important thing to note here, to let the containers talk to each other; they need to be attached to the same Network.
env:
global:
- APP_NAME=my-appbefore_script:
- docker network create ${APP_NAME}-bridge-docker-network
Now it is time to build our image, so in this section, we will create our first stage. In that stage, we need to clone our source code, so we need to reactivate git-clone, and we don’t want to start the services yet as we are doing nothing with them, so that we will skip them.
jobs:
include:
- stage: Build Docker images
git:
clone: true
depth: 1
before_script: skip
after_script: skip script:
- |
echo "Build Docker Image from Source Code" && \
docker build \
. \
--build-arg ARG_RUBY_VERSION=${ARG_RUBY_VERSION} \
--tag ${APP_NAME}/app-ci:latest \
-f dockerfiles/Dockerfile-ci
Now we have our image acme/app-ci:latest, we still have two important things to do, first saving the image as an archive and saving that image in a Travis-CI workspace. Workspace is a BETA feature of Travis-CI which should have been around for a long time, IMHO.
To save the Docker image as an archive we use docker save.
The complete stage looks like that:
jobs:
include:
- stage: Build Docker images
git:
clone: true
depth: 1
before_script: skip
after_script: skip script:
- |
echo "Build Docker Image from Source Code" && \
docker build \
. \
--build-arg ARG_RUBY_VERSION=${ARG_RUBY_VERSION} \
--tag ${APP_NAME}/app-ci:latest \
-f dockerfiles/Dockerfile-ci - docker save --output ${TRAVIS_BUILD_ID}-app-ci.tar ${APP_NAME}/app-ci:latest workspaces:
create:
name: docker-build-images
paths:
- ${TRAVIS_BUILD_ID}-${APP_NAME}-app-ci.tar
Once we named the workspace we will be able to call it later.
We are going to create our second stage. That stage doesn’t need to get the source code, so we don’t need to turn git-clone on.
jobs:
include:
- stage: Build Docker images
# ... removed for clarity sake ... - stage: Application Correctness
script:
- docker exec ${APP_NAME}-app-ci bash -c "bin/rails test"
workspaces:
use:
- docker-build-images
Here we use shell bash, but you can use minimal sh if you feel so. We use a Rails project to provide a concrete example, but that Article can be applied to whatever language you are using.
Wait a minute! There is an important piece missing here! Do you spot it already? Yeah, we don’t have our Docker Image up and running, so we can’t call it! First, we need to start the container and set up the test env, meaning the database here. Let fix that! So go back to the main section in the before_script step.
env:
global:
# ... removed for clarity ... - RAILS_LOG_TO_STDOUT=true
- REDIS_URL=redis://acme-redis:6379/1before_script:
- docker load --input ${TRAVIS_BUILD_ID}-app-ci.tar # ... removed for clarity ... - |
echo "Run Docker Container" && \
docker run --detach \
--name ${APP_NAME}-app-ci \
--env DATABASE_HOST=${APP_NAME}-db \
--env DATABASE_PASSWORD=${DATABASE_PASSWORD} \
--env DATABASE_USERNAME=root \
--env REDIS_URL=${REDIS_URL}
--network=${APP_NAME}-bridge-docker-network \
-t ${APP_NAME}/app-ci:${VERSION} /bin/bash - |
echo "Load SQL Structure" && \
docker exec ${APP_NAME}-app-ci bash -c "bin/rails db:setup"
There are a couple of things worth commenting on here.
First of all, to access our created image, we need to load it! We use the workspace docker-build-images to make our image accessible from the filesystem; Travis-CI does an archive of the previously created workspace and uncompresses it in the current context.
If we are not familiar with Docker, it is worth noting that Docker let reach another container by its name if they are in the same Network, that is why the Redis URL shows acme-redis as URL, and we call the database host like acme-db.
The second important thing here is that we run the container of our application with -t option to allocate a pseudo-tty and ask to run the shell /bin/bash that way, our container will stay up and running for us.
Complete working example here.
