Jun 16, 2016 update: explained reasons for using Docker with QEMU.

For one of my previous assignments I had to automate building ARM-based device firmware and over-the-air updates. While the recovery partition of a device was relatively small and simple and could be built using Buildroot, the root partition was based on Debian 8 “Jessie”.

Let’s make a firmware for a hypothetical ARM-based HAL 9000.

The goal is to describe the process overall and not to delve into device specifics. You still need to figure out the boot sequence, configure and build the kernel, set up partitioning and write custom setup scripts that work for your device.

TL;DR: scripts for the lazy are in the end.

Prerequisites

I assume you’re using some flavor of Linux, can figure out when to use elevated privileges and you have these tools at your disposal:

The reason why I used Docker and QEMU User Emulation instead of just scripting in chroot was that some software needed to be cross compiled and it had complex dependencies. So compiling it natively, thanks to QEMU dynamic binary translation, in the same environment (i.e. OS, kernel and packages) where it’s going to run was the easiest.

Making a minimal Debian ARM image using debootstrap and chroot

According to debootstraps’s website it is:

a tool which will install a Debian base system into a subdirectory of another, already installed system.

And that’s just what we’re going to do:

deboostrap Stage 1

1
2
3
4
rootfs_dir=rootfs
# Debian release to install. We want 8.x, hence Jessie.
distro=jessie
debootstrap --arch=armhf --foreign $distro $rootfs_dir

deboostrap Stage 2

Enable QEMU user emulation by copying qemu-arm-static to run ARM binaries:

1
2
qas=$(which qemu-arm-static)
cp $qas $rootfs_dir$qas

Now start stage 2 of deboostrap:

1
chroot $rootfs_dir /debootstrap/debootstrap --second-stage

This will take a while but in the end you’ll have a directory with a minimal Debian system, sans your device’s kernel, modules and software. At the time of this writing, I ended up with 266MB rootfs.

Importing minimal Debian rootfs into Docker

Making rootfs archive:

1
2
3
4
5
pushd . > /dev/null
cd $rootfs_dir
tar -czf ../rootfs.tar.gz .
popd > /dev/null
rm -rf $rootfs_dir

Importing rootfs into Docker:

1
2
tag='georgesapkin/hal9000:base'
docker import rootfs.tar.gz $tag

Now you have a Docker Debian image that serves as a base for future firmwares. I set up a job in Jenkins to rebuild it once a week, so device firmwares are always based on the latest software.

Installing device-specific kernel, modules and software inside Docker

I use the base image from previous steps:

1
FROM georgesapkin/hal9000:base

Rootfs overlay

All static OS configuration files and scripts are in rootfs_overlay directory that is applied to the base image before any other scripts are run.

1
COPY rootfs_overlay /

A good candidate for the overlay is fstab file from /etc/fstab that list device-specific partitions. Another one is a dpkg filter to be placed in /etc/dpkg/dpkg.cfg.d/01_filter to prevent some unnecessary files from being installed. There’s a good article on Ubuntu Wiki about reducing disk footprint.

Kernel and modules

This is also a good place to install the pre-built device-specific kernel and modules. Keep in mind when configuring and building a kernel for your device that some OS features (e.g. iptables, webcam support, etc.) require specific modules to be enabled. These modules can take up significant amount of space.

1
2
3
COPY kernel/boot         /
COPY kernel/lib/firmware /lib/
COPY kernel/lib/modules  /lib/

Installing software

I put all the software setup into a few scripts files, e.g. setup_some_feature.sh. First, since we cannot reply to any questions when installing packages we need to set:

1
export DEBIAN_FRONTEND=noninteractive

Then we run apt-get install with -y --no-install-recommends -o Dpkg::Options::='--force-confold' arguments. force-confold is needed in case we want to prevent configurations files carried over from the overlay in previous steps from being overwritten. If you are building software inside Docker, you will need to install and setup the development environment first. Here’s a likely set of packages that you might need when building software:

1
2
3
4
5
6
7
8
apt-get install -y \
    --no-install-recommends \
    -o Dpkg::Options::='--force-confold' \
    build-essential \
    ca-certificates \
    curl \
    git \
    python

If your device has Wi-Fi or networking capabilities that you manage from scripts, you might need one or more of the following packages:

  • crda
  • isc-dhcp-client
  • iw
  • wireless-tools
  • wpasupplicant

In case you have software that needs elevated privileges that is configured to do so via a /etc/sudoers.d/* file, you will need to install the sudo package.

The relevant part of the Dockerfile to run the setup scripts:

1
2
COPY setup_some_feature.sh /scripts/
RUN /scripts/setup_some_feature.sh

Cleanup

I put pre-image-authoring cleanup into author.sh. First, let’s get rid of any packages that might be left over from the build environment and some other misc packages that are not needed in the final image, remove unused dependencies and clean Apt cache:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apt-get remove -y \
    binutils \
    build-essential \
    cpp \
    cpp-4.9 \
    dpkg-dev \
    g++ \
    g++-4.9 \
    gcc \
    gcc-4.9 \
    git \
    git-man \
    libc6-dev \
    libc-dev-bin \
    libgcc-4.9-dev \
    linux-libc-dev \
    libstdc++-4.9-dev \
    make \
    manpages \
    nano \
    wget

apt-get autoremove -y
apt-get -q clean

We can go a step further and remove Apt utilities as well, since we’re not going to install packages on a running system.

Don’t forget to disable the root login:

1
usermod -L root

Now, let’s remove some static files, caches, histories, logs and unused locales. Again, Ubuntu Wiki has a good rundown of the process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
rm -rf \
    /usr/share/applications \
    /usr/share/apps \
    /usr/share/doc \
    /usr/share/games \
    /usr/share/groff \
    /usr/share/icons \
    /usr/share/info \
    /usr/share/linda \
    /usr/share/lintian \
    /usr/share/man \
    /usr/share/pixmaps \
    /var/cache/* \
    /var/lib/apt/lists/* \
    /var/log/* \
        2> /dev/null || true

find /usr/share/locale -mindepth 1 -maxdepth 1 ! -name 'en*' \
    ! -name 'locale.alias' | xargs rm -rf 2> /dev/null || true

rm -rf ~/.bash_history 2> /dev/null || true

The relevant part of the Dockerfile to run the author script:

1
2
COPY author.sh /scripts/
RUN /scripts/author.sh

The final step before making a firmware image is to remove temporary scripts and QEMU:

1
2
RUN rm /usr/bin/qemu-arm-static
RUN rm -rf /scripts

Now it’s time to build the firmware image using the Dockerfile:

1
2
image_tag=georgesapkin/hal9000:$(date -I)
docker build --tag="$image_tag" -f Dockerfile .

Since Docker stores all individual layers, you can structure your Dockerfile so that most-frequently changed scripts and commands are at the bottom. That way you can dramatically speed up your builds.

Making a firmware image from a Docker image

First, let’s make a sparse image file, format it as ext4 and mount it. According to Wikipedia a sparse file is:

…a type of computer file that attempts to use file system space more efficiently when the file itself is mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space.

1
2
3
4
5
6
7
8
size=500M
rootfs_image=hal9000-$(date -I).img
dd if=/dev/zero of=$rootfs_image bs=1 count=0 seek=$size status=none

mkfs.ext4 -b 4096 -F $rootfs_image

mkdir -p $rootfs_dir
mount -o loop $rootfs_image $rootfs_dir

It’s only possible to export from a Docker container, so we need to make a temporary one, export and then remote it:

1
2
3
4
container_name=hal9000-$(date -I)
docker run --name $container_name $tag true
docker export $container_name | tar -xf - -C $rootfs_dir/
docker rm $container_name

Now we can unmount and pack the rootfs image:

1
2
umount $rootfs_dir
gzip < $rootfs_image > $rootfs_image.gz

That’s it!

The firmware for our HAL 9000 is ready for flashing. You can flash it using:

1
gunzip -c rootfs.img.gz | dd of=/dev/your_device_root_partition bs=64K

You may take it a step further and create a signed image using cpio and openssl that you can flash using swupdate. Or make an incremental over-the-air update.

To make this fully-automated you can hook it up to your CI of choice and have a scheduled job (for the base image) or source repository triggers (for the firmware).

Feel free to ask questions in comments below or drop me an email if you need some advice.

TL;DR

build.sh - you’ll have to read the post for the comments :p

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#!/bin/bash
set -euo pipefail

rootfs_dir=rootfs
distro=jessie
debootstrap --arch=armhf --foreign $distro $rootfs_dir

qas=$(which qemu-arm-static)
cp $qas $rootfs_dir$qas

chroot $rootfs_dir /debootstrap/debootstrap --second-stage

pushd . > /dev/null
cd $rootfs_dir
tar -czf ../rootfs.tar.gz .
popd > /dev/null
rm -rf $rootfs_dir

tag='georgesapkin/hal9000:base'
docker import rootfs.tar.gz $tag

date=$(date -I)
image_tag=georgesapkin/hal9000:$date
docker build --tag="$image_tag" -f Dockerfile .

size=500M
rootfs_image=hal9000-$date.img
dd if=/dev/zero of=$rootfs_image bs=1 count=0 seek=$size status=none

mkfs.ext4 -b 4096 -F $rootfs_image

mkdir -p $rootfs_dir
mount -o loop $rootfs_image $rootfs_dir

container_name=hal9000-$date
docker run --name $container_name $tag true
docker export $container_name | tar -xf - -C $rootfs_dir/
docker rm $container_name

umount $rootfs_dir
gzip < $rootfs_image > $rootfs_image.gz

setup_some_feature.sh - you really have to write your own.

author.sh - this is a non-exhaustive sample. You will have different things to clean up in your firmware.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
#!/bin/bash
set -euo pipefail

apt-get remove -y \
    binutils \
    build-essential \
    cpp \
    cpp-4.9 \
    dpkg-dev \
    g++ \
    g++-4.9 \
    gcc \
    gcc-4.9 \
    git \
    git-man \
    libc6-dev \
    libc-dev-bin \
    libgcc-4.9-dev \
    linux-libc-dev \
    libstdc++-4.9-dev \
    make \
    manpages \
    nano \
    wget

apt-get autoremove -y
apt-get -q clean

usermod -L root

rm -rf \
    /usr/share/applications \
    /usr/share/apps \
    /usr/share/doc \
    /usr/share/games \
    /usr/share/groff \
    /usr/share/icons \
    /usr/share/info \
    /usr/share/linda \
    /usr/share/lintian \
    /usr/share/man \
    /usr/share/pixmaps \
    /var/cache/* \
    /var/lib/apt/lists/* \
    /var/log/* \
        2> /dev/null || true

find /usr/share/locale -mindepth 1 -maxdepth 1 ! -name 'en*' \
    ! -name 'locale.alias' | xargs rm -rf 2> /dev/null || true

rm -rf ~/.bash_history 2> /dev/null || true

Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
FROM georgesapkin/hal9000:base

COPY rootfs_overlay /

COPY kernel/boot         /
COPY kernel/lib/firmware /lib/
COPY kernel/lib/modules  /lib/

COPY setup_some_feature_1.sh /scripts/
RUN /scripts/setup_some_feature_1.sh

COPY setup_some_feature_2.sh /scripts/
RUN /scripts/setup_some_feature_2.sh

COPY author.sh /scripts/
RUN /scripts/author.sh

RUN rm /usr/bin/qemu-arm-static
RUN rm -rf /scripts

/etc/dpkg/dpkg.cfg.d/01_filter - from overlay

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
path-exclude=/usr/share/applications/*
path-exclude=/usr/share/apps/*
path-exclude=/usr/share/doc/*
path-exclude=/usr/share/groff/*
path-exclude=/usr/share/i18n/charmaps/*
path-include=/usr/share/i18n/charmaps/UTF-8.gz
path-exclude=/usr/share/i18n/locales/*
path-include=/usr/share/i18n/locales/en_GB
path-include=/usr/share/i18n/locales/en_US
path-include=/usr/share/i18n/locales/i18n
path-include=/usr/share/i18n/locales/iso14651_t1
path-include=/usr/share/i18n/locales/iso14651_t1_common
path-include=/usr/share/i18n/locales/translit_*
path-exclude=/usr/share/icons/*
path-exclude=/usr/share/info/*
path-exclude=/usr/share/linda/*
path-exclude=/usr/share/lintian/*
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/locale.alias
path-include=/usr/share/locale/en_GB*
path-include=/usr/share/locale/en_US*
path-exclude=/usr/share/man/*
path-exclude=/usr/share/pixmaps/*