Mellow Root

Minimal containers using Nix

This was written years ago but never published.

Recently I learned that you can create your own distroless docker images with Nix!

"Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shells, or any other programs you would expect to find in a standard Linux distribution.

I'm not going to go into details about why one should do this, but in short: smaller images with less bloat lead to a smaller attack surface and probably increased security.

I tried two different ways to build distroless images with nix:

Below is how I went about it.

Use nix package manager in dockerfile

This is an example of a container that will only contain nginx, its dependencies, and user-specific content/config:

FROM nixos/nix AS build

# Install nginx
RUN mkdir -p /output/store
RUN nix-env --profile /output/profile -i nginx
RUN cp -va $(nix-store -qR /output/profile) /output/store

# Create empty directories needed by nginx
RUN mkdir -p /to_add/var/log/nginx \
             /to_add/var/cache/nginx \
             /to_add/var/conf/ \
             /to_add/var/www

# Create user and group for nginx
RUN addgroup --system nginx
RUN adduser --system -G nginx --uid 31337 nginx

# Make sure nginx can write to required directories
RUN chown -R 31337 /to_add/

FROM scratch

# Copy over nginx files and dependencies
COPY --from=build /output/store /nix/store
COPY --from=build /output/profile/ /usr/local/
COPY --from=build /to_add /

# Copy required user information
COPY --from=build /etc/passwd /etc/passwd
COPY --from=build /etc/group /etc/group

# Add user specific content and config
COPY ./index.html /var/www/
COPY ./nginx.conf /var/conf/nginx.conf
EXPOSE 80
CMD ["nginx", "-p", "/var/"]

This is a multi-stage build where we first use the nixos/nix docker image to install nginx and add a user/group for nginx to use. In the second step we copy the build outputs (ie. nginx and other required folders and files) from the previous steps and start the service.

Build and run the image with

docker build -t nginx-nix:1 .
docker run nginx-nix:1

Build docker image with nix-build

Not only is nix a package manager, but it's also a declarative language. We can use this to create a minimal nginx container, with nothing but the nginx dependencies and configs.

Example docker-nginx.nix:

let
  pkgs = import <nixpkgs> {};

  # Add an index file
  nginxWebRoot = pkgs.writeTextDir "index.html" (builtins.readFile ./index.html);

  # Port nginx listens to
  nginxPort = "80";

  # Create nginx config
  nginxConf = pkgs.writeText "nginx.conf" ''
    user nginx nginx;
    daemon off;
    error_log /dev/stdout info;
    pid /dev/null;
    events {}
    http {
      access_log /dev/stdout;
      server {
        listen ${nginxPort};
        index index.html;
        location / {
          root ${nginxWebRoot};
        }
      }
   }
  '';
in
  pkgs.dockerTools.buildImage {
    # Name of the container
    name = "nix-nginx";

    # Install nginx
    contents = pkgs.nginx;

    # Create directories required by nginx
    extraCommands = ''
      mkdir -p var/log/nginx
      mkdir -p var/cache/nginx
    '';

    # Create the nginx user
    runAsRoot = ''
      #!${pkgs.stdenv.shell}
      ${pkgs.dockerTools.shadowSetup}
      groupadd --system nginx
      useradd --system --gid nginx nginx
    '';

    # Start the service and expose the port
    config = {
      Cmd = [ "nginx" "-c" nginxConf ]; 
      ExposedPorts = {
          "${nginxPort}/tcp" = {};
      };
    };
  }

You might think this looks complicated and hard to read, and you'd be right. It is, but such is life!

Build nix docker image with:

nix-build docker-nginx.nix

Load image into docker:

docker load < result

result is a symlink that is created in the directory you run nix-build. It points to the actual build output in the nix store.

Run the image with:

docker run <whatever image id is printed by the previous step>

Compare to other nginx images

Let's look at the most interesting one: the official nginx Alpine image. Alpine is known for creating small docker images (largely thanks to their glibc alternative musl.)

Image size

Let's take a look at the docker images sizes

$ docker images "*nginx*" --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

REPOSITORY   TAG                                SIZE
nginx-nix    1                                  58.2MB
nginx        alpine                             23.5MB
nginx        latest                             142MB
nix-nginx    vxfix25i4qh1vy32r9cvnbdsaz2r2vix   62.4MB

Just looking at the size of the image we can see that the Alpine image is about 20 MB while our nix-based images are around 60 MB. So in this case Alpine is the winner, the latest official nginx image the loser, and our images somewhere in between.

It is possible to use musl with Nix as well, I imagine the size would be more or less equal in that case.

Image content

Size isn't everything so let's take a look at the filesystem and what's in it. We can dump the images to tar files and inspect them manually with:

docker save <image> > image.tar

But it's pretty tedious going through all layers manually like that. There is a tool called dive which makes it much easier to explore a docker image.

We can also get a flattened view of the filesystem by exporting a container with:

docker export $(docker create <image>:<tag>) | tar -tv | less

This is probably the easiest way to get an overview.

Regardless of which method is used, you'll see that the Alpine image contains many files we might not need or want, things that potentially could be a security problem. It comes bundled with tools such as sh, wget, and the Alpine package manager apk. The nix image will have no such tools - only nginx and its dependencies.

Debugging a distroless image

Having tools in a container might sometimes be useful for different debugging. Just exec into it and start the diagnostics. With a distroless/minimized image, it's not that easy. What we can do instead is attach a sidecar container:

docker run \
  --rm \
  -it \
  --pid=container:<container id> \
  --net=container:<container id> \
  --cap-add sys_admin \
  alpine \
  sh

If you're on Kubernetes and can't SSH to nodes to run that command, you might1 be able to use an ephemeral container like this instead:

kubectl debug \
    -it <pod> \
    --image=alpine \
    --target=<pod>

Read more

Some links I found useful:

  1. If you have it enabled, see feature gates↩

#TIL #containers #nix #security