Create the smallest possible Docker container

Adriaan de Jonge

When you are playing around with Docker, you quickly notice that you are downloading large numbers of megabytes as you use preconfigured containers. A simple Ubuntu container easily exceeds 200MB and as software is installed on top of it, the size increases. In some use cases, you do not need everything that comes with Ubuntu. For example, if you want to run a simple web server, written in Go, there is no need for any tool around that at all.

I have been searching for the smallest possible container to start with and found this one:

docker pull scratch

The scratch image is perfect. Literally perfect! It is elegant, small and fast. It does not contain any bugs, security leaks, slow code or technical debt. And that is because it is basically empty. Except for a bit of metadata added by Docker. In fact, you could have created this scratch image yourself with this command as described in the Docker documentation:

tar cv --files-from /dev/null | docker import - scratch

 

So that is it, the smallest possible Docker image. End of blog post!

... or is there something more we can say about this? For example, how do you use the scratch base image? It turns out this brings some challenges of its own.

Creating content for the scratch image

What can we run on an empty base image? An executable without dependencies. Do you have executables without dependencies?

I used to write code in Python, Java and JavaScript. Each of these languages/platforms require a runtime installed. Recently, I started looking into the Go (or GoLang if you prefer) platform. And it seems (spoiler alert) like Go is statically  linked. So I tried compiling a simple web server saying Hello World and running it within the scratch container. Here is the code for the Hello World web server:

package main

import (
	"fmt"
	"net/http"
)

func helloHandler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintln(w, "Hello World from Go in minimal Docker container")
}

func main() {
	http.HandleFunc("/", helloHandler)

	fmt.Println("Started, serving at 8080")
	err := http.ListenAndServe(":8080", nil)
	if err != nil {
		panic("ListenAndServe: " + err.Error())
	}
}

 

Obviously, I cannot compile my webserver inside the scratch container as there is no Go compiler in it. And as I am working on a Mac, I also cannot compile a Linux binary just like that. (Actually, it is possible to cross-compile GoLang sources to different platforms, but that is material for another blog post)

So I first need a Docker container with a Go compiler. Let's start simple:

docker run -ti google/golang /bin/bash

 

Inside this container, I can build the Go web server, which I have committed in a GitHub repository:

go get github.com/adriaandejonge/helloworld

 

The go get command is a variant of the go build command that allows fetching and building remote dependencies. You can start the resulting executable with:

$GOPATH/bin/helloworld

 

This works. But it is not what we want. We need the hello world container to run inside the scratch container. So, in fact, we need a Dockerfile saying:

FROM scratch
ADD bin/helloworld /helloworld
CMD ["/helloworld"]

 

and then start that. Unfortunately, the way we started the google/golang container, there is no way to build this Dockerfile. So first, we need a way to access Docker from within the container.

Calling Docker from within Docker

When you use Docker, sooner or later you run into the need to control Docker from within Docker. There are multiple ways to accomplish this. You could use recursion and run Docker inside Docker. However, that seems overly complex and again leads to large containers. You can also provide access to the Docker server outside the instance with a few additional command line options:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti google/golang /bin/bash

 

Before you continue, please rerun the Go compiler, as Docker forgot our previous compilation during the restart:

go get github.com/adriaandejonge/helloworld

 

When starting the container, the -v flag creates a volume inside the Docker container and allows you to provide a file from the Docker machine as input. The /var/run/docker.sock is the Unix socket that allows access to the Docker server. The $(which docker) part is a clever way to provide the path for the docker executable inside the container without hardcoding it. However, be careful when you use this command on an Apple when using boot2docker. If the docker executable is installed in a different location than it is installed in boot2docker's virtual machine, this results in a mismatch. It will be the executable inside the boot2docker virtual server that gets inserted into the container. So you may want to replace $(which docker) with /usr/local/bin/docker which is hardcoded. Similarly, if you run a different system, there is a chance that the /var/run/docker.sock has a different location and you need to adjust it accordingly.

Now you can use the Dockerfile inside the google/golang container in the $GOPATH directory, which points to /gopath in this example. Actually, I already checked this Dockerfile into GitHub. So you can copy it from the Go build directory to the desired location like this:

cp $GOPATH/src/github.com/adriaandejonge/helloworld/Dockerfile $GOPATH

 

You need to copy this as the compiled binary is now located in $GOPATH/bin and it is not possible to include files from parent directories when building a Dockerfile. So after copying, the next step is:

docker build -t adejonge/helloworld $GOPATH

 

And if all goes, well, Docker responds with something like:

Successfully built 6ff3fd5a381d

 

Which allows you to run the container:

docker run -ti --name hellobroken adejonge/helloworld

 

But unfortunately, now Docker responds with:

2014/07/02 17:06:48 no such file or directory

 

So what is going on? We have a statically linked executable inside a scratch container. Did we make a mistake?

As it turns out, Go does not statically link libraries. Or at least not all libraries. Under Linux, we can see the dynamically linked libraries for an executable with the ldd command:

ldd $GOPATH/bin/helloworld 

 

Which responds with:

linux-vdso.so.1 => (0x00007fff039fe000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f61df30f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f61def84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f61df530000)

 

So before we can run the Hello World webserver, we need to tell the Go compiler to actually do static linking.

Creating statically linked executables in Go

In order to create statically linked executables, we need to tell Go to use the cgo compiler rather than the go compiler. The command to do so is:

CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld

 

The CGO_ENABLED environment variable tells Go to use the cgo compiler rather than the go compiler. The -a flag tells Go to rebuild all dependencies. Otherwise you still end up with dynamically linked dependencies. And finally the -ldflags '-s' flag is a nice extra. It reduces the file size of the resulting executable by roughly 50%. You can also do this without the cgo compiler. The size reduction is a result from removing debug information.

Just to be sure, rerun the ldd command.

ldd $GOPATH/bin/helloworld 

 

It should now respond with:

not a dynamic executable

 

You can also rerun the steps for creating the Docker container around the executable from scratch:

docker build -t adejonge/helloworld $GOPATH

 

And if all goes well, Docker responds with something like:

Successfully built 6ff3fd5a381d

 

Which allows you to run the container:

docker run -ti --name helloworld adejonge/helloworld

 

And this time it should respond with:

Started, serving at 8080

 

Until so far, there were many manual steps and there is a lot of room for error. Let's exit from the google/golang container and continue from the surrounding machine:

<Press Ctrl-C>
exit

 

You can check the existence or absence of containers and images with:

docker ps -a
docker images -a

 

And you can do some cleaning of Docker with:

docker rm -f helloworld
docker rmi -f adejonge/helloworld

 

Creating a Docker container that creates a Docker container

The steps we took so far, we can also record in a Dockerfile and have Docker do the work for us:

FROM google/golang
RUN CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld
RUN cp /gopath/src/github.com/adriaandejonge/helloworld/Dockerfile /gopath
CMD docker build -t adejonge/helloworld gopath

 

I checked this Dockerfile into a separate GitHub repository called adriaandejonge/hellobuild. It can be built with this command:

docker build -t adejonge/hellobuild github.com/adriaandejonge/hellobuild

 

Providing the  -t flag names the image as adejonge/hellobuild and implicitly tags it as latest. These names make it easier for you to remove the image later on. Next,  you can create a container from this image while providing the flags that you have seen earlier in this post:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti --name hellobuild adejonge/hellobuild

 

Providing the --name hellobuild flag makes it easier to remove the container after running. In fact, you can do so right away, because after running this command, you already created the adejonge/helloworld image:

docker rm -f hellobuild
docker rmi -f adejonge/hellobuild

 

And now you can start a new container named helloworld based on the adejonge/helloworld image as you have done before:

docker run -ti --name helloworld adejonge/helloworld

 

Because all these steps are run from the same command line, without opening a bash shell inside a Docker container, you can add these steps to a bash script and run it automatically. For your convenience, I have added these bash scripts to the hellobuild GitHub repository.

Also, if you want to try the smallest possible Docker container running a Hello World web server without following all the steps described in this blog post, you can also use the pre-built image that I checked into the Docker Hub repository:

docker pull adejonge/helloworld

 

With docker images -a you can see that the size is 3.6MB. Of course, you can make it even smaller if you manage to create an executable that is smaller than the web server in Go that I wrote. In C or Assembly you may be able to do so. However, you can never make it smaller than the scratch image.

Comments (21)

  1. Mark van Holsteijn - Reply

    July 5, 2014 at 10:07 am

    Creating images and executables with such a small footprint is ideal for speed of development, deployment and runtime resource consumption. Most virtual environment I know overcommit on cpu, but not on memory. This way it is possible to cram far more applications on a machine. This approach has a huge potential for cost savings in companies where whole virtual machines running dozens of processes are dedicated to fullfill a single role as web or application server.

  2. Arnout Engelen - Reply

    July 7, 2014 at 4:32 pm

    Very cool stuff.

    This does again beg the question whether it still makes sense to deploy applications on general-purpose operating systems: much of their complexity comes from enforcing process boundaries - and perhaps we shouldn't care about those so much anymore.

    A fascinating project in that direction is MirageOS (http://www.openmirage.org/). On a more humorous note: https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript

  3. Andrew Phillips - Reply

    July 7, 2014 at 9:05 pm

    > much of their complexity comes from enforcing process boundaries - and perhaps we
    > shouldn't care about those so much anymore.

    @Arnout: A related question worth asking, in my opinion: perhaps we shouldn't care, but what is the downside? Yes, there is a performance cost (as discussed in "The Birth and Death..."), but is that performance cost big enough to merit the investment of moving away from it?

    Sure, if somebody comes up with the tooling to make that easy, many people will look at it (see Docker), but would it be worth moving from e.g. RHEL to CoreOS to run Docker in an existing organization? I'll be interested to see numbers around that.

    @Adriaan: cool experiment and great post!

  4. zoobab - Reply

    July 9, 2014 at 10:10 pm

    I adapted openwrt for docker, size is 5mb. Plenty of packages available.

  5. Geoff - Reply

    July 10, 2014 at 12:05 am

    Excellent write-up, scratch and the docker-within-docker are great things for the Docker enthusiast to know. Thanks for putting this out!

  6. Todd Sampson - Reply

    July 10, 2014 at 9:04 pm

    Thanks for the post! The dynamic linking had been causing me issues in my own small footprint builds. This cleaned them right up. Much appreciated.

  7. […] a quick aside, Adriaan de Jonge recently published an article titled ‘Create The Smallest Possible Docker Container’ in which he describes how to create an image that literally contains nothing but a statically […]

  8. Anonymous - Reply

    July 31, 2014 at 11:49 am

    Great post.
    What do you think about doing it with minimal linux live? http://minimal.linux-bg.org/

  9. Kevin - Reply

    August 4, 2014 at 6:37 am

    One question regards the Dockerfile with " From Scratch..." , why it has to copy to /Gopath so it can be build within a Docker container . I did try put dockerfile in like /gopath/src/ and using /gopath/bin/helloworld instead bin/helloworld and I got can't find file exception .

    Is any reason docker build can only find relative path from /gopath instead of absolute path ?

  10. R. Toma - Reply

    August 21, 2014 at 3:41 pm

    Reading this post I had a deja-vu. After scratching my mind I remembered reading this blog:
    https://medium.com/@kelseyhightower/optimizing-docker-images-for-static-binaries-b5696e26eb07

    Great minds think alike...

  11. […] Адриан де Йонг (Adriaan de Jonge) недавно опубликовал статью «Создание самого маленького возможного контейнера Docker», в которой он описал как собрать образ, не содержащий […]

  12. Aaron Greenlee - Reply

    September 10, 2014 at 4:16 am

    Thanks for this post! Have you made any changes since it was posted?

    Also, I was wondering if you could share the entire docker file so I can see you process in a single view?

    Again, thank you for sharing. I am looking at reducing my docker image size and this post was exactly what I hoped to find: docker and go!

  13. wyi - Reply

    September 15, 2014 at 6:14 pm

    Thanks for the awesome work!

    Is there anyway to getting a little more than scratch? I want /bin/sh in addition, because when I pass command line parameters to helloworld using CMD, Docker complains that it cannot find /bin/sh.

  14. Sheldon Hearn - Reply

    October 2, 2014 at 7:00 pm

    At first, this looked very cool. But on rereading it, I wonder what the point is. If you can create scratch with docker import, why not just build the static binary wherever you like (no build container), and tar it into docker import?

    • Adriaan de Jonge - Reply

      October 2, 2014 at 8:45 pm

      Hi Sheldon, thank you for your feedback. Actually, the answer to your question is in the article: "And as I am working on a Mac, I also cannot compile a Linux binary just like that. (Actually, it is possible to cross-compile GoLang sources to different platforms, but that is material for another blog post)".

      Another answer: I use Docker containers the same way that other teams use a Continuous Integration server. If the software compiles in an empty, neutral server, I can be sure the compilation is reproducible. If it compiles only on my own machine, I am not.

  15. Sheldon Hearn - Reply

    October 6, 2014 at 9:41 am

    Ah, I hadn't considered the value of the predictable build environment. Thanks. :-)

  16. Fulldude - Reply

    October 25, 2014 at 1:18 pm

    How World i Run it in the scratch Container as user Nobody?

  17. binaryphile - Reply

    November 3, 2014 at 12:10 am

    Rather than going to the trouble of exposing docker from the host to within the container, wouldn't a simpler solution have been to mount the local directory as a volume and simply copy the helloworld file outside the container? Then you could have built the dockerfile from your host.

Add a Comment