Minimal containers using Nix
This was written years ago but never published.
Recently I learned that you can create your own distroless docker images with Nix!
"Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shells, or any other programs you would expect to find in a standard Linux distribution.
I'm not going to go into details about why one should do this, but in short: smaller images with less bloat lead to a smaller attack surface and probably increased security.
I tried two different ways to build distroless images with nix:
- Use nix package manager in a dockerfile
- Use
nix-build
to build a docker image
Below is how I went about it.
Use nix package manager in dockerfile
This is an example of a container that will only contain nginx, its dependencies, and user-specific content/config:
FROM nixos/nix AS build
# Install nginx
RUN mkdir -p /output/store
RUN nix-env --profile /output/profile -i nginx
RUN cp -va $(nix-store -qR /output/profile) /output/store
# Create empty directories needed by nginx
RUN mkdir -p /to_add/var/log/nginx \
/to_add/var/cache/nginx \
/to_add/var/conf/ \
/to_add/var/www
# Create user and group for nginx
RUN addgroup --system nginx
RUN adduser --system -G nginx --uid 31337 nginx
# Make sure nginx can write to required directories
RUN chown -R 31337 /to_add/
FROM scratch
# Copy over nginx files and dependencies
COPY --from=build /output/store /nix/store
COPY --from=build /output/profile/ /usr/local/
COPY --from=build /to_add /
# Copy required user information
COPY --from=build /etc/passwd /etc/passwd
COPY --from=build /etc/group /etc/group
# Add user specific content and config
COPY ./index.html /var/www/
COPY ./nginx.conf /var/conf/nginx.conf
EXPOSE 80
CMD ["nginx", "-p", "/var/"]
This is a multi-stage build where we first use the nixos/nix
docker image to install nginx
and add a user/group for nginx to use. In the second step we copy the build outputs (ie. nginx and other required folders and files) from the previous steps and start the service.
Build and run the image with
docker build -t nginx-nix:1 .
docker run nginx-nix:1
Build docker image with nix-build
Not only is nix a package manager, but it's also a declarative language. We can use this to create a minimal nginx container, with nothing but the nginx dependencies and configs.
Example docker-nginx.nix
:
let
pkgs = import <nixpkgs> {};
# Add an index file
nginxWebRoot = pkgs.writeTextDir "index.html" (builtins.readFile ./index.html);
# Port nginx listens to
nginxPort = "80";
# Create nginx config
nginxConf = pkgs.writeText "nginx.conf" ''
user nginx nginx;
daemon off;
error_log /dev/stdout info;
pid /dev/null;
events {}
http {
access_log /dev/stdout;
server {
listen ${nginxPort};
index index.html;
location / {
root ${nginxWebRoot};
}
}
}
'';
in
pkgs.dockerTools.buildImage {
# Name of the container
name = "nix-nginx";
# Install nginx
contents = pkgs.nginx;
# Create directories required by nginx
extraCommands = ''
mkdir -p var/log/nginx
mkdir -p var/cache/nginx
'';
# Create the nginx user
runAsRoot = ''
#!${pkgs.stdenv.shell}
${pkgs.dockerTools.shadowSetup}
groupadd --system nginx
useradd --system --gid nginx nginx
'';
# Start the service and expose the port
config = {
Cmd = [ "nginx" "-c" nginxConf ];
ExposedPorts = {
"${nginxPort}/tcp" = {};
};
};
}
You might think this looks complicated and hard to read, and you'd be right. It is, but such is life!
Build nix docker image with:
nix-build docker-nginx.nix
Load image into docker:
docker load < result
result
is a symlink that is created in the directory you run nix-build
. It points to the actual build output in the nix store.
Run the image with:
docker run <whatever image id is printed by the previous step>
Compare to other nginx images
Let's look at the most interesting one: the official nginx Alpine image. Alpine is known for creating small docker images (largely thanks to their glibc alternative musl.)
Image size
Let's take a look at the docker images sizes
$ docker images "*nginx*" --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"
REPOSITORY TAG SIZE
nginx-nix 1 58.2MB
nginx alpine 23.5MB
nginx latest 142MB
nix-nginx vxfix25i4qh1vy32r9cvnbdsaz2r2vix 62.4MB
Just looking at the size of the image we can see that the Alpine image is about 20 MB while our nix-based images are around 60 MB. So in this case Alpine is the winner, the latest official nginx image the loser, and our images somewhere in between.
It is possible to use musl with Nix as well, I imagine the size would be more or less equal in that case.
Image content
Size isn't everything so let's take a look at the filesystem and what's in it. We can dump the images to tar files and inspect them manually with:
docker save <image> > image.tar
But it's pretty tedious going through all layers manually like that. There is a tool called dive which makes it much easier to explore a docker image.
We can also get a flattened view of the filesystem by exporting a container with:
docker export $(docker create <image>:<tag>) | tar -tv | less
This is probably the easiest way to get an overview.
Regardless of which method is used, you'll see that the Alpine image contains many files we might not need or want, things that potentially could be a security problem. It comes bundled with tools such as sh
, wget
, and the Alpine package manager apk
. The nix image will have no such tools - only nginx and its dependencies.
Debugging a distroless image
Having tools in a container might sometimes be useful for different debugging. Just exec into it and start the diagnostics. With a distroless/minimized image, it's not that easy. What we can do instead is attach a sidecar container:
docker run \
--rm \
-it \
--pid=container:<container id> \
--net=container:<container id> \
--cap-add sys_admin \
alpine \
sh
If you're on Kubernetes and can't SSH to nodes to run that command, you might1 be able to use an ephemeral container like this instead:
kubectl debug \
-it <pod> \
--image=alpine \
--target=<pod>
Read more
Some links I found useful:
- Nix example containers
- Write-up of using Nix to build containers: I was wrong about Nix
- Quest for Minimal Docker Images - part 3 - Includes a section about Nix. Generally a good series of posts
If you have it enabled, see feature gates↩