Building Docker with Jenkins: Top Four Annoyances & Their Fixes

If you’re using Jenkins to build and test Docker images, we have a new Jenkins plugin that’s going to change your life. Instead of building images on Jenkins itself (or on a slave) you can check a box and have it built (and tested) on a dedicated CloudShare environment, using the CloudShare Docker-Machine plugin.

Why would you want that? Allow me to count the ways in which life is annoying without it (visually illustrated by a software engineer of limited artistic talent).

We make the use of Jenkins easier for you, Contact our expert to get CloudShare now!

Building Docker on Jenkins is Annoying

So, you have Jenkins building your Docker images? Great. That means you’ve either run into the following problems, or are going to, sooner or later.

Annoyance #1: Docker eats up your disk space

Build enough images or run enough docker-based tests, and your disk is bound to fill up – pretty quickly, too. It is common for CI setups to build images for every commit in a branch, and on a busy Jenkins node this can mean hundreds of GBs a day of new images, dangling image layers, exited containers, orphaned volumes, etc.

And it’s not just disk-space, mind you. It’s easy to end up with long-running containers that somehow did not get killed when the build finished, and continue to eat up RAM and CPU.

Typically, you combat this with periodical docker-gc (or the new prune command), but even that is not a panacea. You have to be careful not to prune away images that you actually do need long-term. So you need black lists, white lists… it’s not trivial.

Annoyance #2: Conflicting published ports

Once you have more than one Jenkins job or if have enabled concurrent builds for the same job, you soon realize that your trusty docker-compose.yml, which has served you so well for local testing on your developer machine, breaks when running in Jenkins. After all, you can’t have more than one container bound to a particular host port, and when you build multiple branches of the same project, this often happens.

Typically this means you get rid of all the port publishing in your Compose/test scripts – which is a shame, because that could be really useful when testing locally. Otherwise, you start adding flags and templating your Compose files in the spirit of “don’t publish ports when running during CI”. It’s not pretty.

Annoyance #3: Docker-ception

Are you running Jenkins, itself, as a container, which in turn builds and runs other containers? After all, why not? What’s good for the goose is good for the Jenkins. But, that won’t work unless you mount the Docker socket onto the Jenkins container:

-v /var/run/docker.sock:/var/run/docker.sock

Hardly the safest of practices. Docker-ception is particularly risky when done in a web-accessible container like Jenkins. Even that’s not enough in some cases and you find you have to run Jenkins with some extra privileges. Yikes.

Oh, and if you’re periodically GCing/pruning your Docker leftovers, be careful to exclude the Jenkins container itself…Yeah, Jenkins GCing itself, it happened to [ahem] a friend of mine. Embarrassing.

Annoyance #4: SSHing into Jenkins

Sooner or later, your build breaks and you need to debug something that only happens during CI. We’ve all been there. We’ve all SSHed into Jenkins, paused the job and “execed” into some running container in order to debug something that never seems to happen your dev-machine. It’s not a good thing when every developer can, and occasionally does, SSH into the CI machine.

Combine that with the rigorous pruning you find yourself doing more and more frequently to free up disk space, and you either lose the containers that capture what you’re trying to debug or have to pause (and remember to resume) the all-important GC process.

Okay, okay, it’s annoying. Get to the solution.

While I could go on listing annoyances, let’s see how using Docker-Machine, and particularly Docker-Machine on CloudShare, makes them go away.

With the plugin installed, you get this awesome little checkbox in your job configuration:

When checked, Jenkins will automatically provision a dedicated environment/VM on CloudShare and set up the right Docker-Machine environment variables, so any Docker command you issue runs remotely on the VM.

By the way, if you’re into pipeline, that works, too. Anything you wrap with the cloudshareDockerMachine DSL extension happens on the VM, instead of locally.

cloudshareDockerMachine {
  // docker stuff here. e.g. docker-compose run ...
}

What You Get

1. No more pruning/GC.

The images you build and the containers you run reside in a dedicated VM in a dedicated environment on CloudShare. Your Jenkins node will not get clogged up.

CloudShare Bonus: Auto-expire

These docker-machines get expired (deleted) after three days (by default, it’s configurable), so even they don’t get a chance to get clogged up with old data.

And you don’t have to remember to delete them. They’ll just go away after a while and will be re-created if you run the job again, as needed.

2. Binding to host ports

You can publish host ports to your heart’s content. Different jobs will use different VMs, so there won’t be any collisions. Hurray for having our cake and eating it, too!

3. No Docker-ception

With the plugin enabled, Jenkins uses Docker-Machine to run containers remotely and securely over TCP. So there’s no need to mount the docker socket onto Jenkins. A lot less risky.

4. Isolation

Caught an interesting bug during CI? You don’t need to SSH into Jenkins to investigate it. You can just SSH into your VM, and there, in perfect isolation, debug for as long as you want without disturbing other jobs.

CloudShare Bonus: Snapshots

Caught a bug and want to save it for later? You can snapshot your docker-machine VM, along with its running processes, and look into it later.

5. Don’t need slaves

If all your builds are essentially docker-based, there are fewer reasons to set up Jenkins slaves. You get parallelization out of the box, since every Job runs on its own VM. Jenkins is just orchestrating; the real work happens on the docker-machines.

CloudShare Bonus: Auto-suspend

CloudShare docker-machines are actually cheaper than slave nodes, since they get auto-suspended when the job is idle. We have extended our auto-suspension mechanism to sense when a docker-machine is actually being used. Once no containers are running and nothing is transmitted over the Docker port (TCP:2376) for 15 minutes, CloudShare auto-suspends the environment. It will automatically resume when the job runs again.

We make the use of Jenkins easier for you, Contact our expert to get CloudShare now!

Get It

You can install our Docker-Machine Jenkins plugin through the Jenkins update manager. Give it a try. It makes the whole Jenkins-Docker experience a lot less frustrating.

Read our blog post about Docker 1.13