ARM Linux Kernel early startup code debugging

This post shows how to debug early (pre-decompression/pre-relocation) initialization code of an ARM (Aarch32) Linux kernel. Debugging kernel code is often not needed and anyway rather hard due to the interaction with real hardware and concurrency in play.  However, to watch, read and learn about early ARM initialization code, debugging can be really useful. Early Initialization is running without concurrency anyway, so this is not a problem in this case.

Before starting, I assume you have a working ARM cross compile environment, a compiled kernel and Qemu at hand. Make sure to compile the kernel with debug symbols (CONFIG_DEBUG_KERNEL=y). I use the following arguments to start Qemu:

$ /usr/bin/qemu-system-arm -s -S -M virt -smp 1 \
  -nographic -monitor none -serial stdio \
  -kernel arch/arm/boot/zImage \
  -initrd core-image-minimal-qemuarm.cpio_.gz \
  -append "console=ttyAMA0 earlycon earlyprintk"

Especially the arguments -s -S are notable here, since the former makes sure Qemu’s built-in debugger is available at port 1234 and the latter stops the machine. This now allows to connect to Qemu using gdb. I use the gdb from my ARM cross compiler toolchain. Once I have a gdb prompt, lets immediately enable gdb’s automatic disassembler on next line before connecting:

$ arm-buildroot-linux-gnueabihf-gdb
(gdb) set disassemble-next-line on
(gdb) show disassemble-next-line
Debugger's willingness to use disassemble-next-line is on.
(gdb) target remote :1234
Remote debugging using :1234
0x40000000 in ?? ()
=> 0x40000000: 00 00 a0 e3 mov r0, #0

Continue reading “ARM Linux Kernel early startup code debugging”

Control FTDI CBUS while TTY is open

For embedded projects I want to control the FTDI CBUS pins as GPIOs. The libFTDI1 library allows to program the CBUS pins as GPIOs by setting the CBUS function to CBUS_IOMODE. I created a Python script available at the python_ft232_cbus_config repository (must be run as root!). The script reprograms CBUS2/3 to IOMODE, while leaving CBUS0/1 at their default (TXLED/RXLED). It should be fairly easy to change the script for your needs.

Once the CBUS pins are programmed as IOMODE, they can be controlled through libFTDI1. There is a C example in the source repository at examples/bitbang_cbus.c. However, there is one downside when using the library: The FTDI kernel driver gets detached automatically. Since I am a screen user, this means that my open sessions get closed automatically too! Using DONT_DETACH_SIO_MODULE seems not to do the trick, the library returns with error code -5 (unable to claim device). It seems that controlling FTDI from user space inherently conflicts with the in-kernel FTDI driver.

Already a while ago I tried to fix this by adding GPIO support to the kernel driver. Unfortunately this did not work out/I lost interest. However, Peter had a valuable hint back then: Control Transfers should be possible without detaching the kernel driver. Continue reading “Control FTDI CBUS while TTY is open”

IPv6 renew issue with Fiber7 and OpenWrt

Since using Fiber7, I was having issues with IPv6 connectivity using OpenWrt. Debugging revealed that Init7’s IPv6 server sets the DHCPv6 Unicast option along with a unicast address, but does not respond when using unicast addressing to renew the lease 20 minutes later. The lease then expires and breaks IPv6 connectivity.

A DHCPv6 server can choose to announce the unicast option, but if announced, the server is supposed to reply to unicast requests. So this seems to be a misbehaving DHCPv6 server. Luckily, the DHCPv6 client shipped with OpenWrt 18.06 now comes with a workaround (odhcp6c: add noserverunicast config option for broken DHCPv6 servers). The option noserverunicast ignores any advertised server unicast addresses and continues using multicast. Init7’s DHCPv6 server seems to answer renew requests using multicast just fine, providing stable IPv6 connectivity.

To enable this workaround, add the following line to the IPv6 WAN interface:

config interface 'wan6'
        option noserverunicast '1'

Then restart your router. Continue reading “IPv6 renew issue with Fiber7 and OpenWrt”

iptable prevents nftables to be loaded

Since a while I am using nftables for my firewalling needs. My nftables.conf has some prerouting settings. After playing with docker, I had the issue that I was no longer able to reload my nftables:

/etc/nftables.conf:12:9-18: Error: Could not process rule: Device or resource busy
chain prerouting {

Also disabling the Docker service did not help. It seems that the kernel module iptable_nat needs to be removed, but this is currently in use:

# rmmod iptable_nat
rmmod: ERROR: Module iptable_nat is in use

There are some iptable rules/chains active which prevent the module from unloading. By clearing the iptable configuration, especially the nat table, it is possible to remove iptable_nat and then using nftables again.

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X




Ubuntu 18.04 LTS (Bionic Beaver) Server Installer differences

Ubuntu 18.04 LTS Server comes now in two flavors with different installers:

  • Ubuntu Server (new Ubuntu-specific Subiquity installer, ubuntu-18.04-live-server-amd64.iso)
  • Alternative Ubuntu Server installer (Debian installer, ubuntu-18.04-server-amd64.iso)

Canonical itself refers to the traditional installer for advanced networking and storage features. However, there are also other differences, this blog post looks into them.

Ubuntu 18.04 LTS Server Live (Subiquity)
Ubuntu 18.04 LTS Server (Debian Installer)

Continue reading “Ubuntu 18.04 LTS (Bionic Beaver) Server Installer differences”

Getting rid of “will be checked for errors at next reboot”

Some of my virtual servers running Ubuntu 16.04.4 showed a hint whenever logging in:

*** /dev/vda1 will be checked for errors at next reboot ***

This issue persisted already for quite a while. Rebooting seems not to help, and also the tips from the blog post “When “disk will be checked for errors at next reboot” won’t go away in Ubuntu” did not help in my case.

Further exploration showed that “Last checked” as shown when using sudo tune2fs -l /dev/vda1 indeed had a very old date. Is the file system check actually executed? Closely observing the reboot process showed that fsck.ext4 is executed:

Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... 
Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Will now check root file system ... fsck from util-linux 2.27.1
[/sbin/fsck.ext4 (1) -- /dev/vda1] fsck.ext4 -a -C0 /dev/vda1

However, the execution returns with a notice that the file system is “clean”. It seems that this does not lead to the date being updated. The man page documents the -f (force) flag to force a check. Digging in the scripts run at boot time (/usr/share/initramfs-tools/scripts/) unveiled that the scripts do add the -f flag in case the kernel parameter forcefsck is used.

So I added the flag in /etc/default/grub to the default Linux kernel arguments:


And updated grub

sudo update-grub

On the next reboot fsck.ext4 got called with the -f flag and did a proper file system check. And with that, the message on login disappeared.

Note: With forcefsck a full file system check is getting executed on every reboot. This might be not ideal for every environment. It seems that fsck.ext4 and the initramfs scripts do not agree on when a check is actually necessary.

Note 2: All machines also have been upgraded from previous Ubuntu LTS versions so it might be related to that.

Hibernate Debian running on Google Compute Engines preemptible VM

Googles Compute Engine VMs which are configured preemptible are massively cheaper than regular VMs, typically a fourth or even a fifth of the price of a regular machine. It seems quite lucrative for everything which is not mission critical.

However, it can be quite annoying when all state gets lost. Luckily Google does not just turn off the machine but sends an ACPI G2 Soft-Off signal. With Debian 9 (stretch) the ACPI daemon is processing the ACPI signals (acpid) and by default shuts down the machine. This post shows how to use hibernate instead.

Note: Since Google might start the machine on a different (virtual) hardware resuming the machine might not succeed, or even worse, lead to adverse effects. In practice, it seems to work quite well for me 🙂

Continue reading “Hibernate Debian running on Google Compute Engines preemptible VM”

i.MX 7 Cortex-M4 memory locations and performance

The NXP i.MX 7 SoC heterogeneous architecture provides a secondary CPU platform with a Cortex-M4 core. This core can be used to run a firmware for custom tasks. The SoC has several options where the firmware can be located: There is a small portion of Tightly Coupled Memory (TCM) close to the Cortex-M4 core. A slightly larger amount of On-Chip SRAM (OCRAM) is available inside the SoC too. The Cortex-M4 core is also able to run from external DDR memory (through the MMDC) and QSPI. Furthermore, the Cortex-M4 uses a Modified Harvard Architecture, which has two independent buses and caches for Code (Code Bus) and Data (System Bus). The memory addressing is still unified, but accesses are split between the buses using addresses as discriminator (addresses in the range 0x00000000-0x1fffffff are loaded through the code bus, 0x20000000-0xdfffffff are accessed through the data bus).

i.MX 7 Simplified Architecture Overview

Continue reading “i.MX 7 Cortex-M4 memory locations and performance”

OpenEmbedded recipes for WireGuard VPN

This weekend I finally came around to create OpenEmbedded recipes for WireGuard. The recipe currently awaits review and hopefully will get part of the meta-networking layer, part of the meta-openembedded repository of the upstream OpenEmbedded project. There are two recipes, one for the kernel module and one for the user space tools. The user space tools have the kernel module as a dependency, hence it is sufficient to install the wireguard-tools package, e.g. by using IMAGE_INSTALL_append in your local.conf:

IMAGE_INSTALL_append = " wireguard-tools"

The kernel module needs at least a kernel version 3.18 or later and has some requirements regarding kernel configuration. The WireGuard website maintains a list of kernel requirements. If you are using the Yocto kernel, the netfilter kernel feature (features/netfilter/netfilter.scc) is enabled by default and seems to be sufficient to run WireGuard. To get started with WireGuard, refer to the excellent Quick Start guide on

WireGuard on MIPS64
WireGuard on MIPS64

Continue reading “OpenEmbedded recipes for WireGuard VPN”