This blog post, written by Szilárd Dömötör, is the second post in a series of blog posts on transforming the Raspberry Pi into a security enhanced IoT platform. The first post explained how to build and install the default OP-TEE implementation for the Raspberry Pi 3. This one describes how you can build your own custom Linux system (with OP-TEE) using the Buildroot environment.

Introduction

When using OP-TEE on the Raspberry Pi 3 the default root file system (rootfs) is generated with a simple initramfs build script: gen-rootfs. This is usable for testing OP-TEE and its functionalities, but for more complex applications, greater customizability is needed. Adding Linux packages this way is not practical, as updating the packages and OP-TEE trusted applications and delivering those updates becomes cumbersome after some time. One might prefer to use a traditional Linux distribution like Raspbian, but in those mainline distributions OP-TEE support is missing or incomplete unfortunately.

This is when Buildroot and Yocto Project comes in the picture. With these solutions one can build a completely customizable embedded Linux system. These build systems provide the rootfs, toolchains, kernel, bootloader and a great number of installable packages. Everything is compiled from scratch using cross compilation. Both are very actively developed and maintained projects, and they are widely used in the industry by companies like Intel, Juniper, Xilinx, Texas Instruments etc. Both have great documentation and online training courses. Both are Free Software: Buildroot is released under GNU GPL version 2 (or later), Yocto is a mix of MIT and GPLv2.

We chose Buildroot, because it is simpler to use and understand than Yocto and has a strong focus on simplicity. It re-uses existing technologies, such as kconfig and make. By default it generates small images in a minimalist approach. Also, it is free of corporate management and has an open community. Here is a more in-depth comparison of the two projects.

The main product of Buildroot is the root filesystem image. We replaced the above mentioned default rootfs with the one generated by Buildroot and the kernel too; OP-TEE, U-Boot images and configuration are from the OP-TEE build output. You can follow the tutorial below to achieve this.

Note: In the meantime, the OP-TEE build project 6 also started using Buildroot for building, but at the time of this writing, RPi3 still uses gen-rootfs.

Getting Buildroot

These are the packages required for Buildroot:

Build tools: which, sed, make (version 3.81 or any later), binutils, build-essential (only for Debian based systems), gcc (version 2.95 or any later), g++ (version 2.95 or any later), bash, patch, gzip, bzip2, perl (version 5.8.7 or any later), tar, cpio, python (version 2.6 or any later), unzip, rsync, file (must be in /usr/bin/file), bc

Source fetching tools: wget

For optional packages, check this page.

Buildroot can be downloaded from buildroot.org as a tarball, or alternatively cloned from their git:

$ git clone git://git.buildroot.net/buildroot
$ cd buildroot

Check out the tag for the preferred version, for example:

$ git checkout tags/2018.02 -b 2018.02

Note: Buildroot releases are made every 3 months, in February, May, August and November. Long Time Support is available: every YYYY.02 release is maintained for one year, with security, build and bug fixes.

Configuration

An optional out of tree build can be used to build different Buildroot configurations from the same source tree. This way you don’t need to rebuild everything to use a previous config. This can be useful for testing more then 1 or 2 configurations, but is not necessary since the config below should work quite good.

Example:

$ mkdir ../build-glibc
$ cd ../build-glibc
$ make -C ../buildroot O=$(pwd) raspberrypi3_64_defconfig

From now on O and C can be omitted. The commands below should be run in the out of tree directory.

The default build configuration for the RPi3 is provided in Buildroot. It contains the necessary settings to make a Linux system for the RPi3. Using the 64 bit version is important, because OP-TEE requires a 64 bit kernel. You can edit configs/raspberrypi3_64_defconfig directly, or make a new defconfig in configs/raspberrypi3_64_custom_defconfig:

BR2_aarch64=y
BR2_ARM_FPU_VFPV4=y
BR2_JLEVEL=0
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_LINARO_AARCH64=y
BR2_SYSTEM_DHCP="eth0"
BR2_ROOTFS_OVERLAY="overlay"
BR2_ROOTFS_POST_BUILD_SCRIPT="board/raspberrypi3-64/post-build.sh"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_GIT=y
BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/linaro-swg/linux.git"
BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="rpi3-optee-4.6"
BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi3"
BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="rpi3.conf"
BR2_LINUX_KERNEL_DTS_SUPPORT=y
BR2_LINUX_KERNEL_INTREE_DTS_NAME="broadcom/bcm2710-rpi-3-b"
BR2_PACKAGE_RPI_FIRMWARE=y
# BR2_PACKAGE_RPI_FIRMWARE_INSTALL_DTB_OVERLAYS is not set
# Required tools to create the SD image
BR2_PACKAGE_HOST_DOSFSTOOLS=y
BR2_PACKAGE_HOST_E2FSPROGS=y
BR2_PACKAGE_HOST_GENIMAGE=y
BR2_PACKAGE_HOST_MTOOLS=y

Put this file in the configs directory, and start make to create the .config file which contains the previous settings and expands them (set other dependent flags):

$ make raspberrypi3_64_custom_defconfig

To edit the configuration use gconfig for Gnome, xconfig for Qt, menuconfig/nconfig for console menu.

$ make gconfig

Note: This only edits .config, to store the changes in the currently used defconfig too use make savedefconfig.

Explanation of the configuration

If you use the defconfig above, these settings are already applied. The locations of the settings in the menuconfig /gconfig window are in bullet points.

Toolchain

To reduce build time, and to be compatible with the kernel built with OP-TEE, we use the external toolchain provided by Linaro. With this setting, the toolchain will be downloaded, and not compiled like the internal toolchain.

• Set Toolchain -> Toolchain Type -> External Toolchain (BR2_TOOLCHAIN_EXTERNAL = y).
• Set Toolchain -> Toolchain -> Linaro AArch64 2017.11 (BR2_TOOLCHAIN_EXTERNAL_LINARO_AARCH64 = y). This is selected by default when you set the external toolchain option.

Kernel

We build the kernel with Buildroot too, since it is an integral part of the system and this way it can be configured just like Buildroot, from the same directory with linux-menuconfig. We use the same kernel that OP-TEE, see here and here.

• Set Kernel -> Linux Kernel -> URL of custom repository to https://github.com/linaro-swg/linux.git (BR2_LINUX_KERNEL_CUSTOM_REPO_URL = https://github.com/linaro-swg/linux.git).
• Set Kernel -> Linux Kernel -> Custom repository version to rpi3-optee-4.6 (BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION = rpi3-optee-4.6).

Additional configuration is also needed for the kernel. This can be done with config fragment files, like rpi3.conf located in the OP-TEE project. These fragments are merged with the main Linux configuration file.

• Set Kernel -> Linux Kernel -> Additional configuration fragment files to rpi3.conf (BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES = rpi3.conf).

rpi3.conf:

CONFIG_TEE=y
CONFIG_OPTEE=y

Note: Copy optee/build/kconfigs/rpi3.conf (available here) to buildroot folder if you not have done already.

We also have to set a Device Tree Source file, which is needed by Linux to describe non-discoverable hardware e.g., UART, I2C, some timers, etc. Check this site for a detailed explanation on the device tree.

• Set Kernel -> Linux Kernel -> Build a Device Tree Blob (DTB) -> Device Tree Source file names to broadcom/bcm2710-rpi-3-b (BR2_LINUX_KERNEL_INTREE_DTS_NAME = broadcom/bcm2710-rpi-3-b).

Customizing the rootfs with overlay

There are two ways to customize the resulting rootfs in Buildroot:

• Root filesystem overlays (BR2_ROOTFS_OVERLAY): a tree of files that is copied directly over the target filesystem after it has been built
• Post-build scripts (BR2_ROOTFS_POST_BUILD_SCRIPT): shell scripts called after Buildroot builds all the selected software, but before the rootfs images are assembled

We choose overlays for the sake of simplicity. The required files for OP-TEE should be placed in the overlay directory.

• Set System configuration -> Root filesystem overlay directories to overlay (BR2_ROOTFS_OVERLAY=”overlay”).

Output Images

To extract the rootfs to the SD card, a tarball should be produced. By default Buildroot generates an sdcard.img image file, which could be directly written to the SD card (done in the post-image script). This is unnecessary because using a tarball is more practical now.

• Delete System Configuration -> Custom scripts to run after creating filesystem images (BR2_ROOTFS_POST_IMAGE_SCRIPT = ”). This step is needed, because the post image script would create sdcard.img.
• Uncheck Filesystem Images -> ext2/3/4 root filesystem (BR2_TARGET_ROOTFS_EXT2 = n). This is only needed by the post image script, unnecessary to generate it now.
• Check Filesystem Images -> tar the root filesystem (BR2_TARGET_ROOTFS_TAR = y).

An uncompressed rootfs.tar and the kernel image Image will be created in the output/images directory.

Post-build script: are run before building the filesystem image, kernel and bootloader. Post-image script: can be used to perform some specific actions after all images have been created. See more about this here.

Other

The system hostname and banner can be set:

• System Configuration -> System hostname (BR2_TARGET_GENERIC_HOSTNAME).
• System Configuration -> System banner (BR2_TARGET_GENERIC_ISSUE).

To set the root password:

• System configuration -> Root password

Buildroot provides many packages which can be added to the build in Target Packages.

Recommendations:

• Networking applications -> dropbear: Small and simple SSH server.
• Text editors and viewers -> nano: If you don’t like vi.

Build & Installation

Finally, these are the steps for building and installation:

• Copy optee/build/kconfigs/rpi3.conf to buildroot folder.
• Populate the overlay folder:
  • copy xtest and tee-supplicant to overlay/bin/
  • copy libteec.so* to overlay/lib/
  • copy all *.ta files to overlay/lib/optee_armtz/

Or use this script to copy the files, just fill out the paths:

OVERLAY_PATH=#should be /buildroot/overlay
OPTEE_TEST_OUT_PATH=#should be /optee_test/out
OPTEE_APPS_PATH=#should be /optee_apps
OPTEE_CLIENT_EXPORT=#should be /optee_client/out
OPTEE_BUILD_PATH=#should be /build
mkdir -p OVERLAY_PATH/lib/optee_armtz
mkdir -pOVERLAY_PATH/bin
mkdir -p OVERLAY_PATH/etc/init.d
mkdir -pOVERLAY_PATH/data
mkdir -p OVERLAY_PATH/data/tee
cpOPTEE_TEST_OUT_PATH/xtest/xtest OVERLAY_PATH/bin/xtest
rsync -av --ignore-missing-argsOPTEE_APPS_PATH/out/ca/* OVERLAY_PATH/bin/
rsync -av --ignore-missing-argsOPTEE_APPS_PATH/out/ta/* OVERLAY_PATH/lib/optee_armtz/
findOPTEE_TEST_OUT_PATH/ta -name '*.ta' -exec cp {} OVERLAY_PATH/lib/optee_armtz/ \;
cpOPTEE_CLIENT_EXPORT/bin/tee-supplicant OVERLAY_PATH/bin/tee-supplicant
cpOPTEE_CLIENT_EXPORT/lib/libteec.so.1.0 OVERLAY_PATH/lib/libteec.so.1.0
ln -sf libteec.so.1.0OVERLAY_PATH/lib/libteec.so.1
ln -sf libteec.so.1 OVERLAY_PATH/lib/libteec.so
cpOPTEE_BUILD_PATH/init.d.optee OVERLAY_PATH/etc/init.d/optee
ln -sf opteeOVERLAY_PATH/etc/init.d/S09_optee

Note: This script also installs S09_optee in /etc/init.d to autostart tee-supplicant. For TA development, OPTEE_APPS_PATH/out is the TA and CA output location.

• Start the build with make. (For out of tree builds call it from the created build directory.)$ make
• Copy the Image to the FIT image directory, and rebuild the FIT image.
• For rootfs installation, simply wipe the rootfs partition of the SD card, and extract the rootfs.tar to it.
• For rootfs installation, simply wipe the rootfs partition of the SD card, and extract the rootfs.tar to it. Don’t forget to `sync` the mounted device before unmounting it otherwise the rootfs might get corrupted and the the Pi won’t boot.

The next post will explain how a verified boot process can be implemented on the Rapsberry Pi.

References

• https://github.com/linaro-swg/gen_rootfs
• https://github.com/OP-TEE/build/blob/master/docs/rpi3.md#41-root-file-system
• https://www.buildroot.org/
• https://www.yoctoproject.org/
• https://bootlin.com/pub/conferences/2016/elc/belloni-petazzoni-buildroot-oe/belloni-petazzoni-buildroot-oe.pdf
• https://github.com/OP-TEE/build/
• https://buildroot.org/downloads/manual/manual.html#requirement-optional
• https://github.com/linaro-swg/linux/tree/rpi3-optee-4.6
• https://github.com/OP-TEE/build/blob/master/kconfigs/rpi3.conf
• https://elinux.org/Device_Tree_What_It_Is
• https://buildroot.org/downloads/manual/manual.html#rootfs-custom
• https://github.com/OP-TEE/build/blob/master/docs/rpi3.md#57-other-root-filesystems-than-initramfs-based
• https://buildroot.org/downloads/manual/manual.html#_customization_emphasis_after_emphasis_the_images_have_been_created