Running KVM on hikey960 board

tutorial

#1

This is a guide for KVM Virtualization on the ARMv8 architecture and illustrates how to set up a KVM development environment on ARM64 processors. It showcases how to run a guest virtual machine on top of an arm hikey960 board host.

Requirements

The steps given in this guide are based on Ubuntu 16.04 system, however all the tools used herewith are also available for other Linux distributions

  • 64-bit ARM Cross Compiler

  • Buildroot

  • ARM64 Host and Guest Kernels 64-bit ARM kernels with KVM capability, which is mandatory for the Host system

  • Host and Guest 64-bit Userspace Images Userspace images for the systems we are going to run.

  • Kvmtool

  • usbstick

Prepare Guest kernel image, Guest disk image, kvmtool

Build guest kernel
The guest kernel has to be built with the following configuration:

  • For the default console output:
    CONFIG_SERIAL_8250=y
    CONFIG_SERIAL_8250_CONSOLE=y

  • For running 32bit images on 64bit hosts:
    CONFIG_IA32_EMULATION=y

  • Proper FS options according to image FS (e.g. CONFIG_EXT2_FS, CONFIG_EXT4_FS).

  • For all virtio devices listed below:
    CONFIG_VIRTIO=y
    CONFIG_VIRTIO_RING=y
    CONFIG_VIRTIO_PCI=y

  • For virtio-blk devices (–disk, -d):
    CONFIG_VIRTIO_BLK=y

  • For virtio-net devices ([–network, -n] virtio):
    CONFIG_VIRTIO_NET=y

  • For virtio-9p devices (–virtio-9p):
    CONFIG_NET_9P=y
    CONFIG_NET_9P_VIRTIO=y
    CONFIG_9P_FS=y

  • For virtio-balloon device (–balloon):
    CONFIG_VIRTIO_BALLOON=y

  • For virtio-console device (–console virtio):
    CONFIG_VIRTIO_CONSOLE=y

  • For virtio-rng device (–rng):
    CONFIG_HW_RANDOM_VIRTIO=y

  • For vesa device (–sdl or --vnc):
    CONFIG_FB_VESA=y

rootfs disk Image for the Guest
Create the disk image for the guest using the following steps:

$ dd if=/dev/zero of=./disk_oe64.img bs=1MiB count=512
$ mkfs.ext3 ./disk_oe64.img
$ mkdir mnt.disk
$ sudo mount -o loop disk_oe64.img mnt.disk
$ sudo tar -xf linaro-image-minimal-genericarmv8-20170127-888.rootfs.tar.gz
-C mnt.disk/
$ sudo umount mnt.disk && rmdir mnt.disk

Copy guest kernel image and guest disk image into a USB stick.

Building KVMtool for arm64
kvmtool depend on libfdt-dev, libbfd, zlib, aio, to simplify cross compile we will us buildroot to build kvmtool.

Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation.

  1. Download buildroot from release https://buildroot.org/download.html

  2. make menuconfig and select aarch64 toolchain

  3. make arm_juno_defconfig, select juno default configuration

  4. tweak kvmtool build cmd to build static version of kvmtool, in

buildroot/package/kvmtool/kvmtool.mk add lkvm-static
define KVMTOOL_BUILD_CMDS
$(TARGET_MAKE_ENV) ARCH=$(KERNEL_ARCH) $(MAKE) -C $(@D) $(KVMTOOL_MAKE_OPTS) lkvm-static
endef

  1. make kvmtool

Start kvm hypervisor to EL2

Due to hikey960 default bootloader can not support boot kernel from EL2 like u-boot, we will uses UEFI to boot kernel from EL2 and start kvm hypervisor.

Download prebuilt UEFI image
http://builds.96boards.org/snapshots/reference-platform/components/uefi-staging/55/hikey960/release/

Or you can build UEFI load from source
for details you can reference: https://github.com/96boards-hikey/tools-images-hikey960/blob/master/build-from-source/README-ATF-UEFI-build-from-source.md

Recover flash UEFI
Config board to recovery mode (1, 2 ON, 3 OFF)
This Hardware User Manual shows the meaning of different pins and components of this board.
Using tools in this dir to flash our new loader:
$ sudo ./hikey_idt -c config -p /dev/ttyUSB0, (ttyUSB0 is the one your type-c is connecting). After this step, the board will run a fastboot program which means we can flash other files.

partition table

$sudo fastboot flash ptable ${UEFI_BUILD_PATH}/prm_ptable.img

bootloader

$sudo fastboot flash xloader ${UEFI_BUILD_PATH}/sec_xloader.img
$sudo fastboot flash fastboot ${UEFI_BUILD_PATH}/l-loader.bin
$sudo fastboot flash fip ${UEFI_BUILD_PATH}/fip.bin

Update UEFI boot.img and rootfs.img
download UEFI boot.img and rootfs.img from rbp release:
boot-0.0+AUTOINC+ba45819943-ea12986b87-r0-hikey960-20180202071634-128.uefi.img -> boot.img (disk boot sector)
rpb-console-image-hikey960-20180202071634-128.rootfs.img.gz -> rootfs.img (rootfs disk image)
http://snapshots.linaro.org/reference-platform/embedded/morty/hikey960/latest/rpb/

$sudo fastboot flash boot uefi.img
$sudo fastboot flash system rootfs.img

Boot Debian from UEFI on hikey960

Config board to autoboot mode (1 ON, 2, 3 OFF)
UEFI will find disk boot sector and load Grub bootload
Grob will load kernel image from disk rootfs path /boot/Image (If you want to update kernel image, replace /boot/Image)
Make sure Hyp mode initialized successfully

/----------------------------------------------------------------------------
|*CE Reference Platform (HiKey960 rpb) |
| Fastboot |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
----------------------------------------------------------------------------/

  Use the ^ and v keys to select which entry is highlighted.
  Press enter to boot the selected OS, `e' to edit the commands
  before booting or `c' for a command-line. ESC to return previous
  menu.

The highlighted entry will be executed automatically in 0s.
Booting `CE Reference Platform (HiKey960 rpb)’

hikey960:/home/linaro# dmesg|grep -i kvm
[ 0.294921] kvm [1]: 8-bit VMID
[ 0.294924] kvm [1]: IDMAP page: a71000
[ 0.294926] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[ 0.295566] kvm [1]: Hyp mode initialized successfully
[ 0.295758] kvm [1]: vgic-v2@e82b4000
[ 0.295839] kvm [1]: vgic interrupt IRQ1
[ 0.295849] kvm [1]: virtual timer IRQ4

Run KVM on hikey960
mount usb stick and run KVM

hikey960:~# mount /dev/sde2 /mnt/
hikey960:/mnt/home/root/kvm# kvmtool-bed2bd9e1fbef5819090feeada7b86eed97ca5e2/lkvm-static run --kernel boot/Image -d disk_oe64.img -m 512 --console virtio –
params “earlyprintk=shm console=hvc0 root=/dev/vda”

lkvm run -k boot/Image -m 512 -c 8 --name guest-3395

Info: Loaded kernel to 0x80080000 (13773312 bytes)
Info: Placing fdt at 0x8fe00000 - 0x8fffffff
Info: virtio-mmio.devices=0x200@0x10000:36

Info: virtio-mmio.devices=0x200@0x10200:37

Info: virtio-mmio.devices=0x200@0x10400:38

[ 1.718158] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1.720656] brd: module loaded
[ 1.723143] loop: module loaded
[ 1.726397] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 1.726729] libphy: Fixed MDIO Bus: probed
[ 1.729531] sky2: driver version 1.30
[ 1.729779] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.729896] ehci-pci: EHCI PCI platform driver
[ 1.730034] ehci-platform: EHCI generic platform driver
[ 1.730143] ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
[ 1.730272] ohci-pci: OHCI PCI platform driver
[ 1.730427] usbcore: registered new interface driver usb-storage
[ 1.730868] IR NEC protocol handler initialized
[ 1.730985] IR RC5(x/sz) protocol handler initialized
[ 1.731064] IR RC6 protocol handler initialized
[ 1.731268] IR JVC protocol handler initialized
[ 1.731368] IR Sony protocol handler initialized
[ 1.731685] IR SANYO protocol handler initialized
[ 1.731775] IR Sharp protocol handler initialized
[ 1.732072] IR MCE Keyboard/mouse protocol handler initialized
[ 1.732186] IR XMP protocol handler initialized
[ 1.732790] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
[ 1.733391] usbcore: registered new interface driver usbhid
[ 1.733482] usbhid: USB HID core driver
[ 1.734476] Initializing XFRM netlink socket
[ 1.734604] NET: Registered protocol family 17
[ 1.734710] NET: Registered protocol family 15
[ 1.734848] 9pnet: Installing 9P2000 support
[ 1.734974] Key type dns_resolver registered
[ 1.736390] Btrfs loaded, crc32c=crc32c-generic
[ 1.736581] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.736969] hctosys: unable to open rtc device (rtc0)
[ 1.737115] ALSA device list:
[ 1.737385] No soundcards found.
[ 1.738247] EXT4-fs (vda): mounting ext3 file system using the ext4 subsystem
[ 1.738977] EXT4-fs (vda): INFO: recovery required on readonly filesystem
[ 1.739157] EXT4-fs (vda): write access will be enabled during recovery

[ 6.091847] EXT4-fs (vda): recovery complete
[ 6.562293] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[ 6.566264] VFS: Mounted root (ext3 filesystem) readonly on device 254:0.
[ 6.567145] devtmpfs: mounted
[ 6.570496] Freeing unused kernel memory: 832K
INIT: version 2.88 booting
Starting udev
/etc/rcS.d/S03systemd-udevd: line 76: /proc/sys/kernel/hotplug: No such file or directory
starting version 232
[ 7.677081] EXT4-fs (vda): re-mounted. Opts: data=ordered
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
Fri Jan 27 07:36:34 UTC 2017
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
INIT: Entering runlevel: 5
Configuring network interfaces… udhcpc (v1.24.1) started
Sending discover…
Sending select for 192.168.33.15…
Lease of 192.168.33.15 obtained, lease time 14400
done.
Starting OpenBSD Secure Shell server: sshd
done.
Starting rpcbind daemon…done.
starting statd: done
hwclock: can’t open ‘/dev/misc/rtc’: No such file or directory
exportfs: can’t open /etc/exports for reading
modprobe: can’t change directory to ‘/lib/modules’: No such file or directory
NFS daemon support not enabled in kernel
Starting syslogd/klogd: done
No makedumpfile found.
Starting auto-serial-console: done
INIT: no more processes left in this runlevel

Last login: Fri Jan 27 07:36:35 UTC 2017 on tty1
root@genericarmv8:~#


#2

Awesome! Thanks for the contribution!


#3

Agree. Very comprehensive.

BTW you might also be interesting in this blog post I put together over the weekend. It describes how you can boot ISO images (in this case the Debian installer) using QEMU and KVM:

Running an ISO installer image for arm64 (AArch64) using QEMU and KVM

I wasn’t working on Hikey960 but I think everything in the post should still apply (although you would have to reduce the VM memory down from -m 4096).