Jun 16, 2016 update: explained reasons for using Docker with QEMU.

For one of my previous assignments I had to automate building ARM-based device firmware and over-the-air updates. While the recovery partition of a device was relatively small and simple and could be built using Buildroot, the root partition was based on Debian 8 “Jessie”.

Let’s make a firmware for a hypothetical ARM-based HAL 9000.

The goal is to describe the process overall and not to delve into device specifics. You still need to figure out the boot sequence, configure and build the kernel, set up partitioning and write custom setup scripts that work for your device.

TL;DR: scripts for the lazy are in the end.

## Prerequisites

I assume you’re using some flavor of Linux, can figure out when to use elevated privileges and you have these tools at your disposal:

The reason why I used Docker and QEMU User Emulation instead of just scripting in chroot was that some software needed to be cross compiled and it had complex dependencies. So compiling it natively, thanks to QEMU dynamic binary translation, in the same environment (i.e. OS, kernel and packages) where it’s going to run was the easiest.

## Making a minimal Debian ARM image using debootstrap and chroot

According to debootstraps’s website it is:

a tool which will install a Debian base system into a subdirectory of another, already installed system.

And that’s just what we’re going to do:

#### deboostrap Stage 2

Enable QEMU user emulation by copying qemu-arm-static to run ARM binaries:

Now start stage 2 of deboostrap:

This will take a while but in the end you’ll have a directory with a minimal Debian system, sans your device’s kernel, modules and software. At the time of this writing, I ended up with 266MB rootfs.

#### Importing minimal Debian rootfs into Docker

Making rootfs archive:

Importing rootfs into Docker:

Now you have a Docker Debian image that serves as a base for future firmwares. I set up a job in Jenkins to rebuild it once a week, so device firmwares are always based on the latest software.

## Installing device-specific kernel, modules and software inside Docker

I use the base image from previous steps:

#### Rootfs overlay

All static OS configuration files and scripts are in rootfs_overlay directory that is applied to the base image before any other scripts are run.

A good candidate for the overlay is fstab file from /etc/fstab that list device-specific partitions. Another one is a dpkg filter to be placed in /etc/dpkg/dpkg.cfg.d/01_filter to prevent some unnecessary files from being installed. There’s a good article on Ubuntu Wiki about reducing disk footprint.

#### Kernel and modules

This is also a good place to install the pre-built device-specific kernel and modules. Keep in mind when configuring and building a kernel for your device that some OS features (e.g. iptables, webcam support, etc.) require specific modules to be enabled. These modules can take up significant amount of space.

#### Installing software

I put all the software setup into a few scripts files, e.g. setup_some_feature.sh. First, since we cannot reply to any questions when installing packages we need to set:

Then we run apt-get install with -y --no-install-recommends -o Dpkg::Options::='--force-confold' arguments. force-confold is needed in case we want to prevent configurations files carried over from the overlay in previous steps from being overwritten. If you are building software inside Docker, you will need to install and setup the development environment first. Here’s a likely set of packages that you might need when building software:

If your device has Wi-Fi or networking capabilities that you manage from scripts, you might need one or more of the following packages:

• crda
• isc-dhcp-client
• iw
• wireless-tools
• wpasupplicant

In case you have software that needs elevated privileges that is configured to do so via a /etc/sudoers.d/* file, you will need to install the sudo package.

The relevant part of the Dockerfile to run the setup scripts:

#### Cleanup

I put pre-image-authoring cleanup into author.sh. First, let’s get rid of any packages that might be left over from the build environment and some other misc packages that are not needed in the final image, remove unused dependencies and clean Apt cache:

We can go a step further and remove Apt utilities as well, since we’re not going to install packages on a running system.

Don’t forget to disable the root login:

Now, let’s remove some static files, caches, histories, logs and unused locales. Again, Ubuntu Wiki has a good rundown of the process.

The relevant part of the Dockerfile to run the author script:

The final step before making a firmware image is to remove temporary scripts and QEMU:

Now it’s time to build the firmware image using the Dockerfile:

Since Docker stores all individual layers, you can structure your Dockerfile so that most-frequently changed scripts and commands are at the bottom. That way you can dramatically speed up your builds.

## Making a firmware image from a Docker image

First, let’s make a sparse image file, format it as ext4 and mount it. According to Wikipedia a sparse file is:

…a type of computer file that attempts to use file system space more efficiently when the file itself is mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space.

It’s only possible to export from a Docker container, so we need to make a temporary one, export and then remote it:

Now we can unmount and pack the rootfs image:

## That’s it!

The firmware for our HAL 9000 is ready for flashing. You can flash it using:

You may take it a step further and create a signed image using cpio and openssl that you can flash using swupdate. Or make an incremental over-the-air update.

To make this fully-automated you can hook it up to your CI of choice and have a scheduled job (for the base image) or source repository triggers (for the firmware).

Feel free to ask questions in comments below or drop me an email if you need some advice.

## TL;DR

build.sh - you’ll have to read the post for the comments :p

setup_some_feature.sh - you really have to write your own.

author.sh - this is a non-exhaustive sample. You will have different things to clean up in your firmware.

Dockerfile

/etc/dpkg/dpkg.cfg.d/01_filter - from overlay