ARM Linux Kernel early startup code debugging

This post shows how to debug early (pre-decompression/pre-relocation) initialization code of an ARM (Aarch32) Linux kernel. Debugging kernel code is often not needed and anyway rather hard due to the interaction with real hardware and concurrency in play.  However, to watch, read and learn about early ARM initialization code, debugging can be really useful. Early Initialization is running without concurrency anyway, so this is not a problem in this case.

Before starting, I assume you have a working ARM cross compile environment, a compiled kernel and Qemu at hand. Make sure to compile the kernel with debug symbols (CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_INFO=y). I use the following arguments to start Qemu:

$ /usr/bin/qemu-system-arm -s -S -M virt -smp 1 \
  -nographic -monitor none -serial stdio \
  -kernel arch/arm/boot/zImage \
  -initrd core-image-minimal-qemuarm.cpio_.gz \
  -append "console=ttyAMA0 earlycon earlyprintk"

Especially the arguments -s -S are notable here, since the former makes sure Qemu’s built-in debugger is available at port 1234 and the latter stops the machine. This now allows to connect to Qemu using gdb. I use the gdb from my ARM cross compiler toolchain. Once I have a gdb prompt, lets immediately enable gdb’s automatic disassembler on next line before connecting:

$ arm-buildroot-linux-gnueabihf-gdb
...
(gdb) set disassemble-next-line on
(gdb) show disassemble-next-line
Debugger's willingness to use disassemble-next-line is on.
(gdb) target remote :1234
Remote debugging using :1234
0x40000000 in ?? ()
=> 0x40000000: 00 00 a0 e3 mov r0, #0

The debugger disassembled the first instruction which is going to be executed for us. The instruction will set r0 to immediate #0. Is this the first instruction of the kernel?

In order to be able to follow the code it is important to understand how the ARM kernel actually boots. The kernel uses a decompression header which decompresses the kernel first. Then the decompressor jumps to the kernels actual entry point. The decompressor is position independent code (PIC) and hence can be run from any point in RAM. At which address the decompressor is running depends on where the kernel has been loaded to, which is typically done by the bootloader. The decompressor’s entry point is at the start label in arch/arm/boot/compressed/head.S. Depending on whether the EFI stub is enabled, the first instructions might look different. The be sure what instruction we expect we can disassemble the first instructions of the kernel zImage:

$ arm-buildroot-linux-gnueabihf-objdump -D -marm \
  -b binary arch/arm/boot/zImage | head -n 20

arch/arm/boot/zImage:     file format binary


Disassembly of section .data:

00000000 <.data>:
       0:       13105a4d        tstne   r0, #315392     ; 0x4d000
       4:       13105a4d        tstne   r0, #315392     ; 0x4d000
       8:       13105a4d        tstne   r0, #315392     ; 0x4d000
       c:       13105a4d        tstne   r0, #315392     ; 0x4d000
      10:       13105a4d        tstne   r0, #315392     ; 0x4d000
      14:       13105a4d        tstne   r0, #315392     ; 0x4d000
      18:       13105a4d        tstne   r0, #315392     ; 0x4d000
      1c:       e1a00000        nop                     ; (mov r0, r0)
      20:       ea0003f6        b       0x1000
...

So the above instruction is actually not the first instruction of the kernels decompressor code! This is Qemu’s bootloader code. Qemu Aarch32 bootloader code can be found in Qemu’s source tree in hw/arm/boot.c

static const ARMInsnFixup bootloader[] = { 
	{ 0xe28fe004 }, /* add lr, pc, #4 */ 
	{ 0xe51ff004 }, /* ldr pc, [pc, #-4] */
	{ 0, FIXUP_BOARD_SETUP }, 
#define BOOTLOADER_NO_BOARD_SETUP_OFFSET 3 
	{ 0xe3a00000 }, /* mov r0, #0 */ 
	{ 0xe59f1004 }, /* ldr r1, [pc, #4] */ 
	{ 0xe59f2004 }, /* ldr r2, [pc, #4] */ 
	{ 0xe59ff004 }, /* ldr pc, [pc, #4] */ 
	{ 0, FIXUP_BOARDID }, 
	{ 0, FIXUP_ARGPTR_LO }, 
	{ 0, FIXUP_ENTRYPOINT_LO }, 
	{ 0, FIXUP_TERMINATOR } 
};

The code is initializing r0, r1 and r2 before jumping to the kernels load address, which is the board specific loader start address plus KERNEL_LOAD_ADDR (0x00010000). Use stepi to step through Qemu’s bootloader code. After the last instruction the machine jumps to the kernel’s initial instruction, in my case at 0x40010000:

...
(gdb) stepi
0x40010000 in ?? ()
=> 0x40010000:  4d 5a 10 13     tstne   r0, #315392     ; 0x4d000

After some stepi the expected instruction appears. In my case, the first instruction is actually a rather weird nop, which has being introduced to make the zImage also a valid PE/COFF binary for EFI (see arch/arm/boot/compressed/efi-header.S).

At this point we can laod the symbols for the decompressor. Since the decompressor is PIC, we need to tell the debugger to which address the decompressor has been laoded to:

(gdb) add-symbol-file arch/arm/boot/compressed/vmlinux 0x40010000
add symbol table from file "arch/arm/boot/compressed/vmlinux" at
        .text_addr = 0x40010000
(y or n) y
Reading symbols from arch/arm/boot/compressed/vmlinux...done.

Symbols are not that useful since we are dealing with assembler, but at least we can use labels to specify breakpoints. It is interesting stepping through the code line by line, but at times larger jumps are more useful.

What is important to know is where the kernel will get uncompressed to. This is determined pretty early in arch/arm/boot/compressed/head.S. For most ARM kernel configuration this is dynamically calculated (CONFIG_AUTO_ZRELADDR). It depends on the load address and the platform specific TEXT_OFFSET (see arch/arm/Makefile). In my case the text offset is `0x00208000` and the load address is 0x40010000, which then works out as 0x40010000 & 0xf8000000 + 0x00208000 = 40208000. Let’s jump to the label restart, where the relocation address should be in r4:

(gdb) info address restart
Symbol "restart" is at 0x40011088 in a file compiled without debugging.
(gdb) break restart
Breakpoint 1 at 0x40011088: file arch/arm/boot/compressed/head.S, line 257.
(gdb) continue
Continuing.

Breakpoint 1, restart () at arch/arm/boot/compressed/head.S:257
257     restart:        adr     r0, LC0
=> 0x40011088 <restart+0>:      73 0f 8f e2     add     r0, pc, #460    ; 0x1cc
(gdb) info reg r4
r4             0x40208001       1075871745

So the kernel will decompressed to 0x40208000. Debugging further actually reveled that the decompressor actually had to relocate the compressed kernel first since the compressed kernel overlaps that target address. Loading the kernel to a higher address avoids that, but as far as I know the kernels load address can not be influenced in Qemu.

To skip the whole decompression phase we can create a breakpoint at that address and continue debugging:

(gdb) break *0x40208000
Breakpoint 2 at 0x40208000
(gdb) continue
Continuing.

Breakpoint 2, 0x40208000 in ?? ()
=> 0x40208000:  3e 1e 04 eb     bl      0x4030f900

This is now the first instruction in arch/arm/kernel/head.S. Typically the kernel is linked to a different address than that. In my case the symbols in vmlinux (System.map) have an entry for stext (the entry point defined in head.S) at 0xc0208000. The uncompressed kernels initialization code is still position independent. The symbols will only match after the MMU has been setup. Unfortunately I did not found a way to load the symbols relocated to the current kernels relocation address. Using add-symbol-file with an text offset seems not to work for the kernel binary (hints welcome! [Update Februar 19, 2020, see Addendum below]). As a workaround addresses for breakpoints of interest can be calculated using System.map and the known relocation address.

Addendum (added February 19, 2020)

The ARM Linux has early print infrastructure which relies on a pre-initialized serial console. The Qemu virt machine provides an initialized PL011 UART (see create_uart function in Qemu’s hw/arm/virt.c). With the following Kernel configuration the debug UART can be enabled:

CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_LL_UART_PL01X=y
CONFIG_DEBUG_UART_PHYS=0x9000000
CONFIG_DEBUG_UART_VIRT=0xf8090000

With that earlyprintk is available even at the relocation and uncompression stage. The relocation stage has optional debugging available which can be enabled by defining DEBUG at the top of the arch/arm/boot/compressed/head.S assembly file). Starting a Linux kernel with that leads to the following output:

C:0x40011080-0x40776200->0x4135CE00-0x41AC1F80 
Uncompressing Linux... done, booting the kernel. 
[ 0.000000] Booting Linux on physical CPU 0x0 
...

The first line prints the relocation which is done before uncompression. This information can be used to calculate the offset in case code after relocation needs to be debugged: (0x4135CE00-0x40011080)+0x40010000.

To debug things step by step a breakpoint at cache_clean_flush (a label which is called right after relocation but before jumping to the relocated code). Then the symbol table can be reloaded to the new address and the restart label can be used as next breakpoint (the label which is called from the unrelocated code).

(gdb) set disassemble-next-line on
(gdb) show disassemble-next-line
Debugger's willingness to use disassemble-next-line is on.
(gdb) target remote :1234
Remote debugging using :1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x40000000 in ?? ()
=> 0x40000000: 00 00 a0 e3 mov r0, #0
(gdb) add-symbol-file arch/arm/boot/compressed/vmlinux 0x40010000
add symbol table from file "arch/arm/boot/compressed/vmlinux" at
.text_addr = 0x40010000
(y or n) y
Reading symbols from arch/arm/boot/compressed/vmlinux...(no debugging symbols found)...done.
(gdb) break cache_clean_flush
Breakpoint 1 at 0x40011764
(gdb) continue
Continuing.

Breakpoint 1, 0x40011764 in cache_clean_flush ()
=> 0x40011764 <cache_clean_flush+4>: df e6 b.n 0x40011526 <call_cache_fn>
(gdb) delete 1
(gdb) remove-symbol-file -a 0x40010000
Remove symbol table from file "/home/sag/projects/linux/arch/arm/boot/compressed/vmlinux"? (y or n) y

At this point Qemu should print the relocation information. Calculate the relocated offset and use it to load the symbols for this address offset (in this case 0x4135bd80):

(gdb) add-symbol-file arch/arm/boot/compressed/vmlinux 0x4135bd80
add symbol table from file "arch/arm/boot/compressed/vmlinux" at
        .text_addr = 0x4135bd80
(y or n) y
Reading symbols from arch/arm/boot/compressed/vmlinux...(no debugging symbols found)...done.
(gdb) break restart
Breakpoint 1 at 0x4135ce38
(gdb) break __enter_kernel
Breakpoint 2 at 0x4135d7a8
(gdb) continue
Continuing.

Breakpoint 1, 0x4135ce38 in restart ()
=> 0x4135ce38 <restart+56>:     d6 f8 00 e0     ldr.w   lr, [r6]
(gdb) continue
Continuing.

Breakpoint 2, 0x4135d7a8 in __enter_kernel ()
=> 0x4135d7a8 <__enter_kernel+8>:       20 47   bx      r4
(gdb) stepi
0x40208000 in ?? ()
=> 0x40208000:  01 90 8f e2     add     r9, pc, #1

The __enter_kernel call is where the decompression code from arch/arm/boot/compressed/ ends and the actual kernel code is called.

I was not able to load symbols for the early initialization code 0x40208000. The vmlinux symbol file comes with addresses when the MMU is enabled. However, the code which is run without MMU enabled is relatively short. The first label after the MMU has been enabled is called __mmap_switched and can already be used as a first breakpoint target:

(gdb) file vmlinux
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Load new symbol table from "vmlinux"? (y or n) y
Reading symbols from vmlinux...done.
(gdb) break __mmap_switched
Breakpoint 1 at 0xc11002e0: file arch/arm/kernel/head-common.S, line 79.
(gdb) break start_kernel
Breakpoint 2 at 0x410005a8: start_kernel. (2 locations)
(gdb) c
Continuing.

Breakpoint 1, __mmap_switched () at arch/arm/kernel/head-common.S:79
79 mov r7, r1
(gdb) c
Continuing.

Breakpoint 2, start_kernel () at init/main.c:581
581 set_task_stack_end_magic(&init_task);
(gdb)

One Reply to “ARM Linux Kernel early startup code debugging”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.