In this third part about KVM on ARMv7 I use kvmtool as the user-space part of the hypervisor. This lightweight hypervisor allowed me to run up to 72 virtual machines… This does not really serves any purpose other than demonstrating what is possible :-). See the video in the end how that looked like.
In 2011 Pekka Enberg announced kvmtool (native Linux KVM tool). Initially it was meant to live within the Linux kernel source tree, but it ended up in a separate git repository on kernel.og.. In between it has been ported to several architectures too, including ARM and ARM64. Its binary usually goes with the name lkvm. It strictly depends on KVM and is otherwise kept rather lean.
Similar to the previous blog post, I used a host root file system based on Ångströms development image. kvmtool was not yet available in any OpenEmbedded layer, hence I had to write a new recipe (it should turn up in the meta-virtualization layer mailing list soon).
Also like in the last post I built a guest root file system using Yocto/OpenEmbedded’s poky-tiny distribution and the qemuarm machine. During that process I found yet another issue with the generated Busybox’ inittab and initramfs. With this fix in place, and the changes to the qemuarm machine to use appropriate tuning and console (see below), the resulting cpio.gz ends up at around 700kB.
--- a/meta/conf/machine/qemuarm.conf +++ b/meta/conf/machine/qemuarm.conf @@ -3,10 +3,11 @@ #@DESCRIPTION: arm_versatile_926ejs require conf/machine/include/qemu.inc -require conf/machine/include/tune-arm926ejs.inc +#require conf/machine/include/tune-arm926ejs.inc #require conf/machine/include/tune-arm1136jf-s.inc +require conf/machine/include/tune-cortexa7.inc KERNEL_IMAGETYPE = "zImage" -SERIAL_CONSOLES = "115200;ttyAMA0 115200;ttyAMA1" +SERIAL_CONSOLES = "115200;ttyS0 115200;hvc0"
The Kernel has been built outside of OpenEmbedded using this defconfig. I tried to keep the configuration quite minimal. One difference to the Kernel I used for Qemu is that I also enabled 8250/16550 and compatible serial support (SERIAL_8250, SERIAL_8250_CONSOLE and SERIAL_OF_PLATFORM). kvmtool by default emulates a 16550A UART.
If you’re interested in the pre-built binaries, I uploaded the new artifacts: Linux 4.7 zImage as tar (defconfig), initramfs as cpio.gz.
With this setup, kvmtool can start a virtual machine and passing it a initramfs with as little as:
lkvm run --initrd core-image-minimal-qemuarm.cpio.gz zImage
If user-space starts a serial console (on ttyS0), after some information from lkvm a login shell should appear:
root@colibriimx7:~# lkvm run --initrd core-image-minimal-qemuarm.cpio.gz zImage # lkvm run -k zImage -m 320 -c 2 --name guest-1169 Info: Loaded kernel to 0x80008000 (2207912 bytes) Info: Placing fdt at 0x8fe00000 - 0x8fffffff Info: Loaded initrd to 0x8fd50818 (718821 bytes) Info: virtio-mmio.devices=0x200@0x10000:36 Poky (Yocto Project Reference Distro) 2.1+snapshot-20160904 qemuarm /dev/ttyS0 qemuarm login: root root@qemuarm:~# poweroff root@qemuarm:~# # KVM session ended normally.
The --params option allows to pass parameters to the kernel, e.g. “console=ttyS0 debug” can be helpful to get the kernel boot log. The option --console virtio can be used to use a para-virtualized serial console (in this case user-space needs to start a console on /dev/hvc0).
kvmtool is quite neat and has lots of interesting options especially for development and testing. There is not a lot of documentation, but the help commands lkvm help and lkvm help COMMAND are quite usable.
Run many machines using kvmtool and KVM
This was my main purpose for trying out kvmtool: My hope is to run more virtual machines than I could do with Qemu in my previous post… In this experiment, I use lkvm with a serial console (the processes stdio) and start each lkvm instance in its own screen session:
screen -m lkvm run -c1 -m14 --initrd core-image-minimal-qemuarm.cpio.gz zImage
Like I did when trying to start lots of machine using Qemu, I tried making use of KSM (Kernel Samepage Merging) again. Initially, enabling KSM did not take any effect at all. After digging a bit through kvmtool’s source code a bug turned up which did not annotate the pages correctly using madvise.
This video shows how I was able to start 72 VMs using kvmtool on a Toradex Colibri iMX7 module with 512MiB of RAM. Halfway in the video I enable KSM. After enabling KSM, the system needs a while to go through the pages and merge the same pages. So I ended up starting a bunch of machines and monitor the memory usage repeatably. One could probably automate the process using a script, but I left that as an exercise for somebody else :-). After starting the 73nd VM (at the very end of the video), the OOM (Out-Of-Memory) killer kicked in which somehow also locked up my ssh shell. The system was still accessible via serial console and the other 72 VMs continued to work fine:
root@colibriimx7:~# lkvm list | grep running | wc -l 72