Making a Debian-based ARM device firmware
Jun 16, 2016 update: explained reasons for using Docker with QEMU.
For one of my previous assignments I had to automate building ARM-based device firmware and over-the-air updates. While the recovery partition of a device was relatively small and simple and could be built using Buildroot, the root partition was based on Debian 8 “Jessie”.
Let’s make a firmware for a hypothetical ARM-based HAL 9000.
The goal is to describe the process overall and not to delve into device specifics. You still need to figure out the boot sequence, configure and build the kernel, set up partitioning and write custom setup scripts that work for your device.
TL;DR: scripts for the lazy are in the end.
I assume you’re using some flavor of Linux, can figure out when to use elevated privileges and you have these tools at your disposal:
- Statically-compiled QEMU User Emulation, aka qemu-user-static.
The reason why I used Docker and QEMU User Emulation instead of just scripting in chroot was that some software needed to be cross compiled and it had complex dependencies. So compiling it natively, thanks to QEMU dynamic binary translation, in the same environment (i.e. OS, kernel and packages) where it’s going to run was the easiest.
According to debootstraps’s website it is:
a tool which will install a Debian base system into a subdirectory of another, already installed system.
And that’s just what we’re going to do:
Enable QEMU user emulation by copying
qemu-arm-static to run ARM binaries:
Now start stage 2 of deboostrap:
chroot $rootfs_dir /debootstrap/debootstrap --second-stage
This will take a while but in the end you’ll have a directory with a minimal Debian system, sans your device’s kernel, modules and software. At the time of this writing, I ended up with 266MB rootfs.
Making rootfs archive:
pushd . > /dev/null
Importing rootfs into Docker:
Now you have a Docker Debian image that serves as a base for future firmwares. I set up a job in Jenkins to rebuild it once a week, so device firmwares are always based on the latest software.
I use the base image from previous steps:
All static OS configuration files and scripts are in
rootfs_overlay directory that is applied to the base image before any other scripts are run.
COPY rootfs_overlay /
A good candidate for the overlay is fstab file from
/etc/fstab that list device-specific partitions. Another one is a dpkg filter to be placed in
/etc/dpkg/dpkg.cfg.d/01_filter to prevent some unnecessary files from being installed. There’s a good article on Ubuntu Wiki about reducing disk footprint.
This is also a good place to install the pre-built device-specific kernel and modules. Keep in mind when configuring and building a kernel for your device that some OS features (e.g. iptables, webcam support, etc.) require specific modules to be enabled. These modules can take up significant amount of space.
COPY kernel/boot /
I put all the software setup into a few scripts files, e.g.
setup_some_feature.sh. First, since we cannot reply to any questions when installing packages we need to set:
Then we run
apt-get install with
-y --no-install-recommends -o Dpkg::Options::='--force-confold' arguments.
force-confold is needed in case we want to prevent configurations files carried over from the overlay in previous steps from being overwritten. If you are building software inside Docker, you will need to install and setup the development environment first. Here’s a likely set of packages that you might need when building software:
apt-get install -y \
If your device has Wi-Fi or networking capabilities that you manage from scripts, you might need one or more of the following packages:
In case you have software that needs elevated privileges that is configured to do so via a
/etc/sudoers.d/* file, you will need to install the
The relevant part of the Dockerfile to run the setup scripts:
COPY setup_some_feature.sh /scripts/
I put pre-image-authoring cleanup into
author.sh. First, let’s get rid of any packages that might be left over from the build environment and some other misc packages that are not needed in the final image, remove unused dependencies and clean Apt cache:
apt-get remove -y \
We can go a step further and remove Apt utilities as well, since we’re not going to install packages on a running system.
Don’t forget to disable the root login:
usermod -L root
Now, let’s remove some static files, caches, histories, logs and unused locales. Again, Ubuntu Wiki has a good rundown of the process.
rm -rf \
The relevant part of the Dockerfile to run the author script:
COPY author.sh /scripts/
The final step before making a firmware image is to remove temporary scripts and QEMU:
RUN rm /usr/bin/qemu-arm-static
Now it’s time to build the firmware image using the Dockerfile:
Since Docker stores all individual layers, you can structure your Dockerfile so that most-frequently changed scripts and commands are at the bottom. That way you can dramatically speed up your builds.
First, let’s make a sparse image file, format it as ext4 and mount it. According to Wikipedia a sparse file is:
…a type of computer file that attempts to use file system space more efficiently when the file itself is mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space.
It’s only possible to export from a Docker container, so we need to make a temporary one, export and then remote it:
Now we can unmount and pack the rootfs image:
The firmware for our HAL 9000 is ready for flashing. You can flash it using:
gunzip -c rootfs.img.gz | dd of=/dev/your_device_root_partition bs=64K
You may take it a step further and create a signed image using
openssl that you can flash using
swupdate. Or make an incremental over-the-air update.
To make this fully-automated you can hook it up to your CI of choice and have a scheduled job (for the base image) or source repository triggers (for the firmware).
Feel free to ask questions in comments below or drop me an email if you need some advice.
build.sh - you’ll have to read the post for the comments :p
setup_some_feature.sh - you really have to write your own.
author.sh - this is a non-exhaustive sample. You will have different things to clean up in your firmware.