Posts

Posted on

Thread border router technical deep-dive (part 1)

This blog post series documents some deeper technical details on how Thread works, in particular border routers. Most of my insights are from real world findings while working on the OpenThread Border Router add-on for Home Assistant.

“Thread is an IPv6-based, low-power mesh networking technology for Internet of things (IoT) products.” according to the Wikipedia article of the Thread protocol, and that really sums it up nicely. Thread is based on the same physical layer as Zigbee: IEEE 802.15.4. Unlike Zigbee, Thread provides IP connectivity. It does not specify the application level protocol, e.g. how to actually control a light bulb. This is where other protocols come in, like Matter. Similar to Zigbee, some Thread devices act as routers within the mesh. However, there is no single predefined coordinator like in Zigbee networks. There is a single Thread leader in every Thread partition which manages the Thread routers. The Thread leader is selected automatically. And there can be multiple border routers which allow communication with Thread devices. This redundancy avoids a single point of failure and can also improve connectivity with the mesh. In my opinion, the use of IP, and the fact that multiple border routers can provide connectivity to the main network are the biggest advantages of Thread and sets it really apart from existing mesh networks.

This article is mostly focusing on this Thread border routers. To learn more about Thread in general, the excellent Thread Primer on openthread.io is a very good resource and is probably a good read in case you are not that familiar with Thread yet. This first part focuses on describing what type of IPv6 network prefixes are in use in a Thread network.

Continue reading “Thread border router technical deep-dive (part 1)”
Posted on

gedit open file pop-up

With the update to Gnome 3.36 gedit unfortunately removed the old open file pop-up (a screenshot of the old pop-up can be found in this how-to on howtogeek.com). In the commit message of the removal the developer mostly cites maintenance burden as the main reason. As a developer I can relate to that and it does make sense to avoid duplicating functionality (and code) which is already present elsewhere. However, I also must say that with the pop-up I somehow always managed to find the documents I was looking for, but the file open dialog’s recent history in its default configuration just does not show enough documents to find the ones I need. This post lists two improvements I found useful.

Continue reading “gedit open file pop-up”
Posted on

systemd boot counting and boot-complete.target

Recently I discovered the boot-complete.target and reading the systemd.special man page sounded like it was exactly what I was looking for:

This target is intended as generic synchronization point for services that shall determine or act on whether the boot process completed successfully.

Also reading the AUTOMATIC_BOOT_ASSESSMENT documentation sounded like I am on the right track, so I decided to use the target. Unfortunately the boot-complete.target seemed not to get activated, so I started to dig deeper.

My target was a (non-UEFI) ARM system. Since the Automatic Boot Assessment documentation indicated that the scheme should also work with non-UEFI systems, I was first assuming that systemd has the target already activated by default. But I was wrong… Continue reading “systemd boot counting and boot-complete.target”

Posted on

Getting coredumps of Qemu on Fedora

Recently it happened that a virtual machine crashed reproducible. journalctl contained messages from audit indicating the crash:

audit[88047]: ANOM_ABEND auid=4294967295 uid=107 gid=107 ses=4294967295 subj=system_u:system_r:svirt_t:s0:c422,c704 pid=88047 comm="qemu-system-x86" exe="/usr/bin/qemu-system-x86_64" sig=6 res=1

I was hoping to get a coredump from it, however, coredumpctl had no corefile (COREFILE column read “none”). There was another message in journalctl which also showed the reason:

systemd-coredump[90346]: Resource limits disable core dumping for process 88047 (qemu-system-x86).

However, ulimit -a (even as user qemu) showed that core file size is unlimited. It seems that something (probably virsh) adjusts limits for that particular process (Max core file size is set to 0 and 0 bytes). Continue reading “Getting coredumps of Qemu on Fedora”

Posted on

zsmalloc performance on ARM64 platform

To use zram the Linux kernel zsmalloc needs to be enabled. The zsmalloc functionality in turn allows to use two methods to access allocations of multiple pages: Copy-based or using VM mapping. Depending on platform one or the other is faster, and the configuration option already suggests that ARM the VM mapping method is typically faster. Hence I was wondering whether that is also true for ARM64 platforms (running in Aarch64 mode). Outcome: On a quad Cortex-A35 platform using Linux 4.14 VM mapping was ~20-50% faster.

Continue reading “zsmalloc performance on ARM64 platform”

Posted on

ARM Linux Kernel early startup code debugging

This post shows how to debug early (pre-decompression/pre-relocation) initialization code of an ARM (Aarch32) Linux kernel. Debugging kernel code is often not needed and anyway rather hard due to the interaction with real hardware and concurrency in play.  However, to watch, read and learn about early ARM initialization code, debugging can be really useful. Early Initialization is running without concurrency anyway, so this is not a problem in this case.

Before starting, I assume you have a working ARM cross compile environment, a compiled kernel and Qemu at hand. Make sure to compile the kernel with debug symbols (CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_INFO=y). I use the following arguments to start Qemu:

$ /usr/bin/qemu-system-arm -s -S -M virt -smp 1 \
  -nographic -monitor none -serial stdio \
  -kernel arch/arm/boot/zImage \
  -initrd core-image-minimal-qemuarm.cpio_.gz \
  -append "console=ttyAMA0 earlycon earlyprintk"

Especially the arguments -s -S are notable here, since the former makes sure Qemu’s built-in debugger is available at port 1234 and the latter stops the machine. This now allows to connect to Qemu using gdb. I use the gdb from my ARM cross compiler toolchain. Once I have a gdb prompt, lets immediately enable gdb’s automatic disassembler on next line before connecting:

$ arm-buildroot-linux-gnueabihf-gdb
...
(gdb) set disassemble-next-line on
(gdb) show disassemble-next-line
Debugger's willingness to use disassemble-next-line is on.
(gdb) target remote :1234
Remote debugging using :1234
0x40000000 in ?? ()
=> 0x40000000: 00 00 a0 e3 mov r0, #0

Continue reading “ARM Linux Kernel early startup code debugging”

Posted on

Control FTDI CBUS while TTY is open

For embedded projects I want to control the FTDI CBUS pins as GPIOs. The libFTDI1 library allows to program the CBUS pins as GPIOs by setting the CBUS function to CBUS_IOMODE. I created a Python script available at the python_ft232_cbus_config repository (must be run as root!). The script reprograms CBUS2/3 to IOMODE, while leaving CBUS0/1 at their default (TXLED/RXLED). It should be fairly easy to change the script for your needs.

Once the CBUS pins are programmed as IOMODE, they can be controlled through libFTDI1. There is a C example in the source repository at examples/bitbang_cbus.c. However, there is one downside when using the library: The FTDI kernel driver gets detached automatically. Since I am a screen user, this means that my open sessions get closed automatically too! Using DONT_DETACH_SIO_MODULE seems not to do the trick, the library returns with error code -5 (unable to claim device). It seems that controlling FTDI from user space inherently conflicts with the in-kernel FTDI driver.

Already a while ago I tried to fix this by adding GPIO support to the kernel driver. Unfortunately this did not work out/I lost interest. However, Peter had a valuable hint back then: Control Transfers should be possible without detaching the kernel driver. Continue reading “Control FTDI CBUS while TTY is open”

Posted on

IPv6 renew issue with Fiber7 and OpenWrt

Since using Fiber7, I was having issues with IPv6 connectivity using OpenWrt. Debugging revealed that Init7’s IPv6 server sets the DHCPv6 Unicast option along with a unicast address, but does not respond when using unicast addressing to renew the lease 20 minutes later. The lease then expires and breaks IPv6 connectivity.

A DHCPv6 server can choose to announce the unicast option, but if announced, the server is supposed to reply to unicast requests. So this seems to be a misbehaving DHCPv6 server. Luckily, the DHCPv6 client shipped with OpenWrt 18.06 now comes with a workaround (odhcp6c: add noserverunicast config option for broken DHCPv6 servers). The option noserverunicast ignores any advertised server unicast addresses and continues using multicast. Init7’s DHCPv6 server seems to answer renew requests using multicast just fine, providing stable IPv6 connectivity.

To enable this workaround, add the following line to the IPv6 WAN interface:

config interface 'wan6'
        ...
        option noserverunicast '1'

Then restart your router. Continue reading “IPv6 renew issue with Fiber7 and OpenWrt”

Posted on

iptable prevents nftables to be loaded

Since a while I am using nftables for my firewalling needs. My nftables.conf has some prerouting settings. After playing with docker, I had the issue that I was no longer able to reload my nftables:

/etc/nftables.conf:12:9-18: Error: Could not process rule: Device or resource busy
chain prerouting {
^^^^^^^^^^

Also disabling the Docker service did not help. It seems that the kernel module iptable_nat needs to be removed, but this is currently in use:

# rmmod iptable_nat
rmmod: ERROR: Module iptable_nat is in use

There are some iptable rules/chains active which prevent the module from unloading. By clearing the iptable configuration, especially the nat table, it is possible to remove iptable_nat and then using nftables again.

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

 

 

 

Posted on

Ubuntu 18.04 LTS (Bionic Beaver) Server Installer differences

Ubuntu 18.04 LTS Server comes now in two flavors with different installers:

  • Ubuntu Server (new Ubuntu-specific Subiquity installer, ubuntu-18.04-live-server-amd64.iso)
  • Alternative Ubuntu Server installer (Debian installer, ubuntu-18.04-server-amd64.iso)

Canonical itself refers to the traditional installer for advanced networking and storage features. However, there are also other differences, this blog post looks into them.

Ubuntu 18.04 LTS Server Live (Subiquity)

Ubuntu 18.04 LTS Server (Debian Installer)

Continue reading “Ubuntu 18.04 LTS (Bionic Beaver) Server Installer differences”