A Journey to Boot Linux on Raspberry Pi

A Journey to Boot Linux on Raspberry Pi Jian-Hong Pan (StarNight) @ COSCUP 2020

Who am I 潘建宏 / Jian-Hong Pan (StarNight) Endless OS Foundation You can ﬁnd me at ● http://www.slideshare.net/chienhungpan/ ● GitHub: starnight ● Email: jian-hong [AT] endlessm.com starnight [AT] g.ncu.edu.tw

Outline ● History ● OSTree ● How does Raspberry Pi boot ● Device Tree ● Deploy Device Tree with OSTree ● Why U-Boot ● Wrap Up! Boot Endless OS on Raspberry Pi ● Demo ● Reference

History Endless OS Foundation focuses on enabling people to harness the power of computing everywhere. ● Develops a Linux-based operating system Endless OS as the payload and pre-loads a lot of knowledge content and apps. ● The preloaded content and apps help people self-learning, especially for the area without internet. ● Uses OSTree and ﬂatpak as the key technology for system deployment and update. ● Endless OS runs on general x86_64 computers, just like other Linux distributions. ● Raspberry Pi is the wide spread low cost ARM based computer in recent years. It is a good tool for education.

OSTree Description from OSTree Is both a shared library and suite of command line tools that combines a "git-like" model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration. ● Transactional upgrades and rollback for the system ● Replicating content incrementally over HTTP via GPG signatures and "pinned TLS" support ● Support for parallel installing more than just 2 bootable roots ● Binary history on the server side (and client) ● Introspectable shared library API for build and deployment systems ● Flexible support for multiple branches and repositories, supporting projects like flatpak which use libostree for applications, rather than hosts. Another reference: The Fundamentals of OSTree

OSTree Example $ ostree admin status * eos 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0 Version: 3.9.0~200702-230029 origin refspec: eos:os/eos/arm64/master GPG: Signature made Fri 03 Jul 2020 07:50:23 AM CST using RSA key ID 9E08D8DABA02FC46 GPG: Good signature from "EOS OSTree Signing Key 1 <maintainers@endlessm.com>" GPG: Key expires Sun 01 Jul 2029 12:18:11 AM CST GPG: Signature made Fri 03 Jul 2020 07:50:26 AM CST using RSA key ID FCF17B17F1F8E157 GPG: Good signature from "EOS Flatpak Signing Key 1 <maintainers@endlessm.com>" GPG: Key expires Wed 10 Jun 2026 01:47:38 AM CST eos 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0 (rollback) Version: 3.9.0~200621-230039 origin refspec: eos:os/eos/arm64/master GPG: Signature made Mon 22 Jun 2020 07:56:54 AM CST using RSA key ID 9E08D8DABA02FC46 GPG: Good signature from "EOS OSTree Signing Key 1 <maintainers@endlessm.com>" GPG: Key expires Sun 01 Jul 2029 12:18:11 AM CST GPG: Signature made Mon 22 Jun 2020 07:56:57 AM CST using RSA key ID FCF17B17F1F8E157 GPG: Good signature from "EOS Flatpak Signing Key 1 <maintainers@endlessm.com>" GPG: Key expires Wed 10 Jun 2026 01:47:38 AM CST

$ ls -l /ostree/deploy/eos/deploy/ total 16 drwxr-xr-x 14 root root 4096 Jun 22 08:08 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0 -rw-r--r-- 1 root root 41 Jun 22 08:08 1432c89ac02b1186ecbc0c3b6f1af2677aa09d4812cb97c4bd0fa46721342227.0.origin drwxr-xr-x 14 root root 4096 Jul 6 16:13 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0 -rw-r--r-- 1 root root 41 Jul 6 16:13 4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0.origin $ ls -l /ostree/deploy/eos/deploy/4c59454f03dbfe2c86858b0ade89caf36db3fba25233971484a4796a39e02d02.0 total 48 lrwxrwxrwx 5 root root 7 Jun 22 08:01 bin -> usr/bin drwxr-xr-x 2 root root 4096 Jan 1 1970 boot drwxr-xr-x 2 root root 4096 Jan 1 1970 dev drwxr-xr-x 115 root root 4096 Jul 6 17:07 etc lrwxrwxrwx 3 root root 9 Jun 22 08:01 home -> /var/home lrwxrwxrwx 5 root root 7 Jun 22 08:01 lib -> usr/lib lrwxrwxrwx 5 root root 9 Jun 22 08:01 lib64 -> usr/lib64 lrwxrwxrwx 3 root root 10 Jun 22 08:01 media -> /run/media drwxr-xr-x 2 root root 4096 Jan 1 1970 mnt lrwxrwxrwx 3 root root 8 Jun 22 08:01 opt -> /var/opt lrwxrwxrwx 3 root root 15 Jun 22 08:03 ostree -> /sysroot/ostree drwxr-xr-x 2 root root 4096 Jan 1 1970 proc lrwxrwxrwx 3 root root 13 Jun 22 08:06 root -> /var/roothome drwxr-xr-x 9 root root 4096 Jan 1 1970 run lrwxrwxrwx 5 root root 8 Jun 22 08:03 sbin -> usr/sbin drwxr-xr-x 2 root root 4096 Jan 1 1970 srv drwxr-xr-x 2 root root 4096 Jan 1 1970 sys drwxr-xr-x 2 root root 4096 Jan 1 1970 sysroot drwxrwxrwt 2 root root 4096 Jan 1 1970 tmp drwxr-xr-x 12 root root 4096 Jan 1 1970 usr drwxr-xr-x 11 root root 4096 Jan 1 1970 var Filesystem Hierarchy Standard (FHS)

Thanks to Carl’s talk yesterday: rootfs 淺談

Bootloader Configuration from OSTree $ cat /boot/loader/uEnv.txt kernel_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94 acea/vmlinuz-5.6.0-7-generic ramdisk_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f9 4acea/initramfs-5.6.0-7-generic.img fdt_ﬁle=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94acea/d evicetree-5.6.0-7-generic bootargs=rw splash plymouth.ignore-serial-consoles quiet loglevel=0 ostree=/ostree/boot.0/eos/e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94a cea/0 $ ls /boot/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071aeb1ff61cb914f94acea devicetree-5.6.0-7-generic initramfs-5.6.0-7-generic.img vmlinuz-5.6.0-7-generic

Start from Raspberry Pi 3 We started to enable Endless OS on Raspberry Pi 3 (RPi 3) ﬁrst. Key points: ● We want 64 bits system ● U-boot bootloader (RPi 3 & 4 ready) ● Linux mainline kernel including Device Tree ○ RPi 3 ready ○ RPi 4 boots up, but many parts are still under development ● OSTree ○ Generic for arm64 platforms

Boot Raspberry Pi 3 Understanding the Raspberry Pi Boot Process (Raspbian/Raspberry Pi OS (Ofﬁcial), Arch Linux ARM, Fedora ARM, Ubuntu) ● Understanding the Raspberry Pi Boot Process ● The boot folder ● Boot sequence 1st stage Boot from GPU with firmware in ROM and loads bootcode.bin in FAT 2nd stage Bootcode.bin loads start.elf ... 3rd stage Start.elf loads DT, kernel … and starts CPU 4th stage Kernel boots FAT boot partition EXT4 partition Raspbian for example

Device Tree Description from devicetree.org ● The devicetree is a data structure for describing hardware. ● Rather than hard coding every detail of a device into an operating system, many aspects of the hardware can be described in a data structure that is passed to the operating system at boot time. ● The devicetree is used by OpenFirmware (OF) Important events: ● Re: [GIT PULL] omap changes for v2.6.39 merge window on LKML ● Device trees I: Are we having fun yet? on LWN.net

Config Data Code Device Tree OF

drivers/bluetooth/hci_bcm.c static int bcm_of_probe(struct bcm_device *bdev) { device_property_read_u32(bdev->dev, "max-speed", &bdev->oper_speed); device_property_read_u8_array(bdev->dev, "brcm,bt-pcm-int-params", bdev->pcm_int_params, 5); bdev->irq = of_irq_get_byname(bdev->dev->of_node, "host-wakeup"); return 0; } … static const struct of_device_id bcm_bluetooth_of_match[] = { { .compatible = "brcm,bcm20702a1" }, { .compatible = "brcm,bcm4329-bt" }, { .compatible = "brcm,bcm4345c5" }, { .compatible = "brcm,bcm4330-bt" }, { .compatible = "brcm,bcm43438-bt", .data = &bcm43438_device_data }, { .compatible = "brcm,bcm43540-bt", .data = &bcm4354_device_data }, { .compatible = "brcm,bcm4335a0" }, { }, }; MODULE_DEVICE_TABLE(of, bcm_bluetooth_of_match); arch/arm/boot/dts/bcm2711-rpi-4-b.dts bluetooth { compatible = "brcm,bcm43438-bt"; max-speed = <2000000>; shutdown-gpios = <&expgpio 0 GPIO_ACTIVE_HIGH>; };

Device Tree Blobs are Deployed like $ ls /usr/lib/ﬁrmware/5.6.0-7-generic/device-tree/ actions amlogic broadcom intel qcom sprd allwinner apm cavium lg realtek synaptics altera arm freescale marvell renesas ti amd bitmain hisilicon mediatek rockchip xilinx $ ls /usr/lib/ﬁrmware/5.6.0-7-generic/device-tree/broadcom/ bcm2711-rpi-4-b.dtb bcm2837-rpi-3-b-plus.dtb stingray bcm2837-rpi-3-a-plus.dtb bcm2837-rpi-cm3-io3.dtb bcm2837-rpi-3-b.dtb northstar2

Single Device Tree? FIT for Generic arm64 OSTree OSTree can deploy only one device tree ﬁle originally, so we gather all of the boards’ DTBs into a single FIT-uImage as the deployment target. /{ description = "Device Tree List"; #address-cells = <1>; images { qcom_msm8994-angler-rev-101.dtb { description = "qcom_msm8994-angler-rev-101.dtb"; data = /incbin/("/boot/dtb/qcom/msm8994-angler-rev-101.dtb"); type = "flat_dt"; arch = "arm64"; compression = "none"; }; qcom_apq8096-db820c.dtb { ... }; }; Make U-Boot load the FIT-uImage, then pick the designated device tree in the FIT-uImage.

OSTree Supports Device Tree Directory Now ● OSTree can deploy only one device tree ﬁle originally fdt_ﬁle=/ostree/eos-8e2b71f01721f41d806e96d158390aa8a52c0f2ddb dfe2f6e7f043ce9ec3a2d7/devicetree-5.3.0-12-generic ● After opened the feature request ticket, OSTree can support Device Tree directory deployment now! commit 5f08649f51fe (“deploy: support devicetree directory”) fdtdir=/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021 c9af6c0d59fa96a21dae/dtb We have two ways to deploy the Device Trees now: 1. A single FIT-uimage → Endless OS chooses this way currently 2. Device Tree directory.

$ ls /boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/* /boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/amlogic: meson-a1-ad401.dtb meson-gxbb-p201.dtb meson-gxl-s905x-hwacom-amazetv.dtb meson-axg-s400.dtb meson-gxbb-vega-s95-meta.dtb meson-gxl-s905x-khadas-vim.dtb meson-g12a-sei510.dtb meson-gxbb-vega-s95-pro.dtb meson-gxl-s905x-libretech-cc.dtb meson-g12a-u200.dtb meson-gxbb-vega-s95-telos.dtb meson-gxl-s905x-nexbox-a95x.dtb meson-g12a-x96-max.dtb meson-gxbb-wetek-hub.dtb meson-gxl-s905x-p212.dtb meson-g12b-a311d-khadas-vim3.dtb meson-gxbb-wetek-play2.dtb meson-gxm-khadas-vim2.dtb meson-g12b-odroid-n2.dtb meson-gxl-s805x-libretech-ac.dtb meson-gxm-nexbox-a1.dtb meson-g12b-s922x-khadas-vim3.dtb meson-gxl-s805x-p241.dtb meson-gxm-q200.dtb meson-g12b-ugoos-am6.dtb meson-gxl-s905d-libretech-pc.dtb meson-gxm-q201.dtb meson-gxbb-kii-pro.dtb meson-gxl-s905d-p230.dtb meson-gxm-rbox-pro.dtb meson-gxbb-nanopi-k2.dtb meson-gxl-s905d-p231.dtb meson-gxm-s912-libretech-pc.dtb meson-gxbb-nexbox-a95x.dtb meson-gxl-s905d-phicomm-n1.dtb meson-gxm-vega-s96.dtb meson-gxbb-odroidc2.dtb meson-gxl-s905w-p281.dtb meson-sm1-khadas-vim3l.dtb meson-gxbb-p200.dtb meson-gxl-s905w-tx3-mini.dtb meson-sm1-sei610.dtb /boot/ostree/eos-8a4f394fb3b8d81f37e1fdb62376335955fc807dc021c9af6c0d59fa96a21dae/dtb/broadcom: bcm2711-rpi-4-b.dtb bcm2837-rpi-3-b.dtb bcm2837-rpi-cm3-io3.dtb stingray bcm2837-rpi-3-a-plus.dtb bcm2837-rpi-3-b-plus.dtb northstar2 Example: OSTree Deploys Device Tree Directory

4th stage Kernel boots Boot Raspberry Pi 3 with Endless OS Raspbian’s firmwares 4th stage U-Boot loads kernel, initramfs and flattened device tree (FDT) in OSTree and boots kernel with boot script Reference: ● Boot loaders in Raspbian ● The boot folder ● Boot options in config.txt Note: Start.elf loads kernel8.img, which is the entry file for arm64. Most of other distributions make it as the kernel directly. Endless OS makes it as U-Boot for some reasons. 1st stage Boot from GPU with firmware in ROM and loads bootcode.bin in FAT 2nd stage Bootcode.bin loads start.elf ... 3rd stage Start.elf loads DT, kernel … and starts CPU U-Boot

Why U-Boot? 1. Endless OS comes with OSTree which includes kernel, initramfs and device tree. They will be changed with different commits / revisions. 2. Because of OSTree, the booting will become failed to find the init daemon, if the kernel image has no related built-in modules and there is no initramfs. [ 2.939544] ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]--- [ 13.350160] WARNING: CPU: 0 PID: 10 at kernel/irq_work.c:102 irq_work_queue_on+0xe8/0x108 [ 13.360149] Modules linked in: [ 13.365009] CPU: 0 PID: 10 Comm: rcu_preempt Not tainted 4.19.76-v8+ #1 3. Raspberry Pi’s firmware loads the Device Tree and passes it to the booting file entry. However, it is not the Device Tree used by Linux mainline kernel. So, Endless OS needs U-Boot as the middle step to boot kernel with the initramfs and designated device tree. We prepared an U-Boot script to load device tree, initramfs and kernel, then boot with some arguments.

U-Boot Tells Us U-Boot> print ... board=rpi board_name=4 Model B ... boot_prefixes=/ /boot/ ... boot_targets=mmc0 mmc1 pxe dhcp ... cpu=armv8 ... fdt_addr_r=0x02600000 ... fdtfile=broadcom/bcm2711-rpi-4-b.dtb initrd_high=ffffffffffffffff kernel_addr_r=0x00080000 ... ramdisk_addr_r=0x02700000 ... scriptaddr=0x02400000

/boot/uEnv.txt Provided by OSTree kernel_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950 071aeb1ff61cb914f94acea/vmlinuz-5.6.0-7-generic ramdisk_image=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a9 50071aeb1ff61cb914f94acea/initramfs-5.6.0-7-generic.img fdt_ﬁle=/ostree/eos-e5ab3e8095af17c26f85814919ea5d55faa36a950071ae b1ff61cb914f94acea/devicetree-5.6.0-7-generic bootargs=rw splash plymouth.ignore-serial-consoles quiet loglevel=0 ostree=/ostree/boot.0/eos/e5ab3e8095af17c26f85814919ea5d55faa36a950 071aeb1ff61cb914f94acea/0

/boot/boot.scr platform_bootargs=cma=256M@512M echo "Load u-Boot environment ..." ext4load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} /boot/uEnv.txt env import -t ${scriptaddr} ${filesize} echo "Load device tree ..." fdt_load_addr_r=${ramdisk_addr_r} ext4load ${devtype} ${devnum}:${distro_bootpart} ${fdt_load_addr_r} /boot/${fdt_file} fdt addr ${fdt_load_addr_r} setexpr fdt_notation gsub / _ ${fdtfile} fdt get addr fdt_start_addr_r /images/${fdt_notation} data fdt get size fdt_size /images/${fdt_notation} data fdt move ${fdt_start_addr_r} ${fdt_addr_r} ${fdt_size} fdt addr ${fdt_addr_r} echo "Load kernel and unzip it ..." kernel_load_addr_r=${ramdisk_addr_r} ext4load ${devtype} ${devnum}:${distro_bootpart} ${kernel_load_addr_r} /boot/${kernel_image} unzip ${kernel_load_addr_r} ${kernel_addr_r} echo "Load RAM disk ..." ext4load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} /boot/${ramdisk_image} echo "Boot ..." setenv bootargs root=LABEL=ostree ${bootargs} ${platform_bootargs} booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r} Extract the DTB from FIT-uImage

Storage Layout ROM on SoC SD Boot Partition (FAT) Boot loaders (bootcode.bin, start.elf ...) u-boot SD OSTree Partition (ext4) ● /boot/ ○ boot.scr ○ uEnv.txt ○ ostree/ … / ■ <kerne> ■ <initramfs> ■ <devicetree> ● /ostree/… ● ... Power on

4th stage Kernel boots Boot Raspberry Pi 4 (new SoC) with Endless OS Raspbian’s ﬁrmwares 4th stage U-Boot loads kernel, initramfs and flattened device tree (FDT) in OSTree and boots kernel with boot script 1st stage Boot from GPU with firmware in ROM including bootcode.bin job in Second stage: loads start4x.elf ... in FAT 3rd stage Start4.elf loads DT, kernel … and starts CPU U-Boot

But U-Boot always boots Linux kernel failed on Raspberry Pi 4B !???

arch/arm/boot/dts/bcm2711-rpi-4-b.dts / { compatible = "raspberrypi,4-model-b", "brcm,bcm2711"; model = "Raspberry Pi 4 Model B"; chosen { /* 8250 auxiliary UART instead of pl011 */ stdout-path = "serial1:115200n8"; }; /* Will be ﬁlled by the bootloader */ memory@0 { device_type = "memory"; reg = <0 0 0>; }; ... WHAT ???!

[U-Boot] rpi4: fix dram bank initialization (Denied) ● Found the memory → reg must be fixed up for the DRAM bank in Raspberry Pi 4B’s Device Tree. Raspberry Pi 4B’s memory could be 1, 2, 4 and 8 GB. So, the reg should not be fixed values. The values must be filled by someone dynamically. ● Raspberry Pi’s boot firmware must know how to fix the reg’s values. Although, U-Boot does not know the values directly, it can be launched by Raspberry Pi’s firmware. And, Raspberry Pi’s firmware already puts the correct values in the memory stack during the execution of U-Boot. I sent the patch, but it is denied. ● Because, there is already a better and generic way which is good!!! ○ commit 7a3f15e7 ("dm: Fix default address cells return value") ○ commit 8076fc29 ("libfdt: Allow #size-cells of 0") ○ commit 0ba41ce1 ("libfdt: return correct value if #size-cells property is not present") ○ commit ce2dae3a ("libfdt: fdt_address_cells() and fdt_size_cells()")

Enable CONFIG_ARCH_FIXUP_FDT_MEMORY in U-Boot for Raspberry Pi 4B

Reference ● Raspberry Pi 4 Preview Release | Endless OS 3.7.7~beta1 ● Upstream Raspberry Pi 4 B support ● Raspberry Pi → Documentation → The boot folder ● RPi U-Boot ● OSTree ● Device Tree ● U-Boot

A Journey to Boot Linux on Raspberry Pi

A Journey to Boot Linux on Raspberry Pi

More Related Content

What's hot

Similar to A Journey to Boot Linux on Raspberry Pi

More from Jian-Hong Pan

Recently uploaded

In this document

A Journey to Boot Linux on Raspberry Pi