How to use Xen for linux(ubuntu) and Android

[scorpionzezz]

Hi @leo-yan

I tried your kernel version, I’m able to have shell access. But, somehow I still getting the XEN CPU error.
Here is the full log from boot till I got shell access: Flashing partition systemDone.hikey960 boarid:5301 xloader use UART6scsyss - Pastebin.com

root@linaro-developer:~# uname -a
Linux linaro-developer 5.0.0-rc3-315296-gff9fd24304a2 #2 SMP PREEMPT Thu Feb 14 09:05:08 CET 2019 aarch64x

Is the UEFI matters? I use this one: http://snapshots.linaro.org/96boards/reference-platform/components/uefi-staging/84/hikey960/debug/

And I use

1. boot-linaro-stretch-developer-hikey-20190120-27.img
2. rootfs-linaro-stretch-developer-hikey-20190120-27.img

I unsprase the rootfs, mount it, put the kernel, grub.cfg, xen.efi and so on to the boot. Then I sparse it again.
I also replace the grub to the efi folder (in the boot img)

Can you maybe kindly share your boot and rootfs img?

Cheers!

[leo-yan]

Hi @scorpionzezz,

@scorpionzezz

Is the UEFI matters? I use this one: http://snapshots.linaro.org/96boards/reference-platform/components/uefi-staging/84/hikey960/debug/

Can you maybe kindly share your boot and rootfs img?

Sure, I found I am using different versions for ARM-TF/UEFI and Xen:

ARM-TF/UEFI: http://snapshots.linaro.org/96boards/reference-platform/components/uefi-staging/latest/hikey960/debug/
Xen: I built Xen with its master branch, so I can see the version is: Xen 4.12-unstable (c/s Mon Dec 17 09:22:59 2018 +0000 git:a5b0eb3636) EFI loader

As @dradspd mentioned, we need to add below options ‘hmp-unsafe’ and ‘dom0_vcpus_pin’ into Xen command line thus can dismiss CPU4-CPU7 booting failure issue; In your log it show the booting failure not only for CPU4~7 (big CPUs) but also CPU1~3 failure (LITTLE CPUs), so firstly could try update Xen and UEFI/ARM-TF to check if can fix the booting issue for CPU1~CPU3.

xen_hypervisor /boot/xen.efi loglvl=all console=dtuart dtuart=/soc/ serial@fff32000 efi=no-rs hmp-unsafe dom0_vcpus_pin

Though Mani has released the latest boot and rootfs images, but I still use the older boot and rootfs images which downloaded from the folder: Linaro Snapshots ; seems to me the boot-xxx.img and rootfs-xxx.img will not introduce SMP CPU booting failure.

@scorpionzezz

I unsprase the rootfs, mount it, put the kernel, grub.cfg, xen.efi and so on to the boot. Then I sparse it again.
I also replace the grub to the efi folder (in the boot img)

I use a different way for this .

We can flash the prebuilt boot image: boot-linaro-stretch-developer-hikey-20180828-19.img.gz and with rootfs image for Debian system, thus we can firstly boot up board successfully with WIFI enabled. So after enabling WIFI I can copy Image/Image.dtb and modify grub.cfg; so in the next time we can select grub menu for Xen booting.

[scorpionzezz]

Hi @leo-yan

thanks! I updated the UEFI, added the hmp-unsafe and dom0_vcpus_pin, not all the CPUs are up
I managed to install the xen-tools as well
If you try to run “xl list”, can you see the Domain-0?

thanks you so much!

[scorpionzezz]

@Vikram

can you somehow explain how you able to run android as domU?

Cheers!

[ramblech]

if you want to see domain 0 in xl list,
you can run /etc/init.d/xencommons start after you install xen tools

[dradspd]

Hi Leo,

First, I apology for this long reply.

Just got back to the grind. First, a great idea to boot the pre-built and then push the binaries over to the board via ssh. I did exactly as you instructed. The prebuilt with kernel 4.15-hikey boots up with ease. The only issue I had was the missing USB Ethernet driver. I modprobe’d kernel module r8152.ko and even modified /etc/modules to load it at boot time, still, network interface enx70886b809fae fails to come up. To proceed, WiFi was used to pull xen.efi, Image, hi6220-hikey.dtb and modified grub from the host. As you have guessed it, my board with xen 4.12 efi still can’t boot. It seems none of cpus could come up this time. Below is the screenshot during Xen boot time:

Loading driver at 0x000B8C70000 EntryPoint=0x000B8D0C02C
Loading driver at 0x000B8C70000 EntryPoint=0x000B8D0C02C
Using modules provided by bootloader in FDT
Xen 4.12.0-rc (c/s Fri Feb 15 10:02:00 2019 +0000 git:365aabb) EFI loader
hikey960 boarid:5301 xloader use UART6
scsysstat_value[8].
clear reset source
last_keypoint0,reboot_type0
secdbg not DCU.
SecDbgVer exit

xloader chipid is: 0x36600110, start at 450ms.
Build Date: Dec 6 2017, 15:31:59
[clock_init] ++
hikey960 [hikey960_clk_init]
hi3660 [clk_setup]
[clock_init] –
storage type is UFS
ufs retry: 6 count v_tx:0 v_rx:0
ufs set v_tx:0 v_rx:0
Hikey960[5301] no need avs_init.
ddr ft:0xf20332a3,mode:1 target:4
UceLdOk
ch 0 gt_errfail, STATUS:0x00000060
ch 0 gdst_errfail, STATUS:0x00000040
ch 1 gt_errfail, STATUS:0x00000060
ch 1 gdst_errfail, STATUS:0x00000040
ch 2 gt_errfail, STATUS:0x00000060
ch 2 gdst_errfail, STATUS:0x00000040
ch 3 gt_errfail, STATUS:0x00000060
ch 3 gdst_errfail, STATUS:0x00000040
timeout
timeout
timeout
timeout
density: 0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000
ddr info 0x00000306
400M
685M
C2R,V0x00000030 e:66
1067M
C2R,V0x0000002b e:113
C2R,V0x0000002c e:66
C3R,V0x0000002d e:66
C0R,V0x0000002d e:66
C1R,V0x0000002d e:66
C2R,V0x0000002d e:66
C3R,V0x0000002e e:66
C0R,V0x0000002e e:66
C1R,V0x0000002e e:66
C2R,V0x0000002e e:66
C3R,V0x0000002f e:65
C0R,V0x0000002f e:65
C1R,V0x0000002f e:66
C2R,V0x0000002f e:65
C3R,V0x00000030 e:65
1244M
1866M
C2R,V0x00000015 e:66
C2R,V0x00000016 e:66
C0R,V0x00000017 e:66
C2R,V0x00000017 e:66
C3R,V0x00000017 e:66
iomcu_subsys_init
boot_c0 PROFILE 4
NOTICE: BL2: v2.0(release):v2.0-431-gc8765826
NOTICE: BL2: Built : 05:03:12, Dec 20 2018
NOTICE: BL2: Booting BL31
NOTICE: BL31: v2.0(release):v2.0-431-gc8765826
NOTICE: BL31: Built : 05:03:12, Dec 20 2018
I/TC:
I/TC: OP-TEE version: 3.0.0 #1 Thu Dec 20 05:03:10 UTC 2018 aarch64
I/TC: Initialized
UEFI firmware (version Alpha built at 05:02:51 on Dec 20 2018)
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll 0xBF18C000
Loading DxeCore at 0x00BF18B000 EntryPoint=0x00BF18C000
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll 0xBF18C000
HOBLIST address in DXE = 0xBEF4D018
Memory Allocation 0x00000004 0xBFFE8000 - 0xBFFE8FFF
Memory Allocation 0x00000004 0xBFFE7000 - 0xBFFE7FFF
Memory Allocation 0x00000004 0xBFFE6000 - 0xBFFE6FFF
Memory Allocation 0x00000004 0xBFFE5000 - 0xBFFE5FFF
Memory Allocation 0x00000004 0xBFFE9000 - 0xBFFFFFFF
Memory Allocation 0x00000004 0xBFFD5000 - 0xBFFE4FFF
Memory Allocation 0x00000004 0xBF8CF000 - 0xBFFD4FFF
Memory Allocation 0x00000004 0xBF1C9000 - 0xBF8CEFFF
Memory Allocation 0x00000004 0xBF18B000 - 0xBF1C8FFF
Memory Allocation 0x00000003 0xBF18B000 - 0xBF1C8FFF
FV Hob 0x1AC98000 - 0x1AD87FFF
FV Hob 0xBF1C9000 - 0xBF8CDF3F
FV2 Hob 0xBF1C9000 - 0xBF8CDF3F
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/PCD/Dxe/Pcd/DEBUG/PcdDxe.dll 0xBF0D1000
Loading driver at 0x000BF0D0000 EntryPoint=0x000BF0D1048 PcdDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll 0xBF0C1000
Loading driver at 0x000BF0C0000 EntryPoint=0x000BF0C1048 ArmCpuDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/RuntimeDxe/RuntimeDxe/DEBUG/RuntimeDxe.dll 0xBA0D0000
Loading driver at 0x000BA0C0000 EntryPoint=0x000BA0D0048 RuntimeDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/SecurityStubDxe/SecurityStubDxe/DEBUG/SecurityStubDxe.dll 0xBF0B1000
Loading driver at 0x000BF0B0000 EntryPoint=0x000BF0B1048 SecurityStubDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/EmbeddedMonotonicCounter/EmbeddedMonotonicCounter/DEBUG/EmbeddedMonotonicCounter.dll 0xBA030000
Loading driver at 0x000BA020000 EntryPoint=0x000BA030048 EmbeddedMonotonicCounter.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/ResetRuntimeDxe/ResetRuntimeDxe/DEBUG/Reset.dll 0xB9F90000
Loading driver at 0x000B9F80000 EntryPoint=0x000B9F90048 Reset.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/RealTimeClockRuntimeDxe/RealTimeClockRuntimeDxe/DEBUG/RealTimeClock.dll 0xB9EF0000
Loading driver at 0x000B9EE0000 EntryPoint=0x000B9EF0048 RealTimeClock.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/MetronomeDxe/MetronomeDxe/DEBUG/MetronomeDxe.dll 0xBF0A9000
Loading driver at 0x000BF0A8000 EntryPoint=0x000BF0A9048 MetronomeDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/SerialDxe/SerialDxe/DEBUG/SerialDxe.dll 0xBF0A1000
Loading driver at 0x000BF0A0000 EntryPoint=0x000BF0A1048 SerialDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Platforms/Hisilicon/HiKey960/HiKey960GpioDxe/HiKey960GpioDxe/DEBUG/HiKey960Gpio.dll 0xBF09A000
Loading driver at 0x000BF099000 EntryPoint=0x000BF09A048 HiKey960Gpio.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/ArmPlatformPkg/Drivers/PL061GpioDxe/PL061GpioDxe/DEBUG/PL061GpioDxe.dll 0xBF091000
Loading driver at 0x000BF090000 EntryPoint=0x000BF091048 PL061GpioDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Platforms/Hisilicon/HiKey960/HiKey960Dxe/HiKey960Dxe/DEBUG/HiKey960Dxe.dll 0xBF07C000
Loading driver at 0x000BF07B000 EntryPoint=0x000BF07C048 HiKey960Dxe.efi
[BDID]Adcin0:222
[BDID]Adcin0Remap:1
[BDID]Adcin1:0
[BDID]Adcin1Remap:0
[BDID]Adcin2:597
[BDID]Adcin2Remap:3
[BDID]boardid: 5301
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/DpcDxe/DpcDxe/DEBUG/DpcDxe.dll 0xBF074000
Loading driver at 0x000BF073000 EntryPoint=0x000BF074048 DpcDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Variable/EmuRuntimeDxe/EmuVariableRuntimeDxe/DEBUG/EmuVariableRuntimeDxe.dll 0xB9E50000
Loading driver at 0x000B9E40000 EntryPoint=0x000B9E50048 EmuVariableRuntimeDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/HiiDatabaseDxe/HiiDatabaseDxe/DEBUG/HiiDatabase.dll 0xBF043000
Loading driver at 0x000BF042000 EntryPoint=0x000BF043048 HiiDatabase.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DevicePathDxe/DevicePathDxe/DEBUG/DevicePathDxe.dll 0xBF02E000
Loading driver at 0x000BF02D000 EntryPoint=0x000BF02E048 DevicePathDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/CapsuleRuntimeDxe/CapsuleRuntimeDxe/DEBUG/CapsuleRuntimeDxe.dll 0xB9DB0000
Loading driver at 0x000B9DA0000 EntryPoint=0x000B9DB0048 CapsuleRuntimeDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/ArmGic/ArmGicDxe/DEBUG/ArmGicDxe.dll 0xBF024000
Loading driver at 0x000BF023000 EntryPoint=0x000BF024048 ArmGicDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultTolerantWriteDxe/DEBUG/FaultTolerantWriteDxe.dll 0xBF015000
Loading driver at 0x000BF014000 EntryPoint=0x000BF015048 FaultTolerantWriteDxe.efi
Ftw: Workspace or Spare block does not exist!
Error: Image at 000BF014000 start failed: Invalid Parameter
remove-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultTolerantWriteDxe/DEBUG/FaultTolerantWriteDxe.dll 0xBF015000
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/SetupBrowserDxe/SetupBrowserDxe/DEBUG/SetupBrowser.dll 0xBEFF8000
Loading driver at 0x000BEFF7000 EntryPoint=0x000BEFF8048 SetupBrowser.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll 0xBEFCD000
Loading driver at 0x000BEFCC000 EntryPoint=0x000BEFCD048 BdsDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/Drivers/DtPlatformDxe/DtPlatformDxe/DEBUG/DtPlatformDxe.dll 0xBEFC3000
Loading driver at 0x000BEFC2000 EntryPoint=0x000BEFC3048 DtPlatformDxe.efi
DtPlatformDxeEntryPoint: no DT/ACPI preference found, defaulting to DT
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/TimerDxe/TimerDxe/DEBUG/ArmTimerDxe.dll 0xBEFBA000
Loading driver at 0x000BEFB9000 EntryPoint=0x000BEFBA048 ArmTimerDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/DisplayEngine.dll 0xBEF97000
Loading driver at 0x000BEF96000 EntryPoint=0x000BEF97048 DisplayEngine.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/WatchdogTimerDxe/WatchdogTimer/DEBUG/WatchdogTimer.dll 0xBEF8F000
Loading driver at 0x000BEF8E000 EntryPoint=0x000BEF8F048 WatchdogTimer.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Console/ConPlatformDxe/ConPlatformDxe/DEBUG/ConPlatformDxe.dll 0xBEF84000
Loading driver at 0x000BEF83000 EntryPoint=0x000BEF84048 ConPlatformDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Console/TerminalDxe/TerminalDxe/DEBUG/TerminalDxe.dll 0xBEF71000
Loading driver at 0x000BEF70000 EntryPoint=0x000BEF71048 TerminalDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/Keyboard/VirtualKeyboardDxe/VirtualKeyboardDxe/DEBUG/VirtualKeyboardDxe.dll 0xBEF66000
Loading driver at 0x000BEF65000 EntryPoint=0x000BEF66048 VirtualKeyboardDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Platforms/Hisilicon/HiKey960/HiKey960MmcDxe/HiKey960MmcDxe/DEBUG/HiKey960MmcDxe.dll 0xBEF5D000
Loading driver at 0x000BEF5C000 EntryPoint=0x000BEF5D048 HiKey960MmcDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Pci/NonDiscoverablePciDeviceDxe/NonDiscoverablePciDeviceDxe/DEBUG/NonDiscoverablePciDeviceDxe.dll 0xB9D44000
Loading driver at 0x000B9D43000 EntryPoint=0x000B9D44048 NonDiscoverablePciDeviceDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/SdMmc/DwMmcHcDxe/DwMmcHcDxe/DEBUG/DwMmcHcDxe.dll 0xB9D30000
Loading driver at 0x000B9D2F000 EntryPoint=0x000B9D30048 DwMmcHcDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Sd/SdDxe/SdDxe/DEBUG/SdDxe.dll 0xB9D20000
Loading driver at 0x000B9D1F000 EntryPoint=0x000B9D20048 SdDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/MnpDxe/MnpDxe/DEBUG/MnpDxe.dll 0xB9D08000
Loading driver at 0x000B9D07000 EntryPoint=0x000B9D08048 MnpDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/ArpDxe/ArpDxe/DEBUG/ArpDxe.dll 0xB9CF8000
Loading driver at 0x000B9CF7000 EntryPoint=0x000B9CF8048 ArpDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/Ip4Dxe/Ip4Dxe/DEBUG/Ip4Dxe.dll 0xB9CD1000
Loading driver at 0x000B9CD0000 EntryPoint=0x000B9CD1048 Ip4Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/Udp4Dxe/Udp4Dxe/DEBUG/Udp4Dxe.dll 0xB9CBC000
Loading driver at 0x000B9CBB000 EntryPoint=0x000B9CBC048 Udp4Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/Tcp4Dxe/Tcp4Dxe/DEBUG/Tcp4Dxe.dll 0xB9C9A000
Loading driver at 0x000B9C99000 EntryPoint=0x000B9C9A048 Tcp4Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/Dhcp4Dxe/Dhcp4Dxe/DEBUG/Dhcp4Dxe.dll 0xB9C84000
Loading driver at 0x000B9C83000 EntryPoint=0x000B9C84048 Dhcp4Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/Mtftp4Dxe/Mtftp4Dxe/DEBUG/Mtftp4Dxe.dll 0xB9C6E000
Loading driver at 0x000B9C6D000 EntryPoint=0x000B9C6E048 Mtftp4Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Network/UefiPxeBcDxe/UefiPxeBcDxe/DEBUG/UefiPxe4BcDxe.dll 0xB9C57000
Loading driver at 0x000B9C56000 EntryPoint=0x000B9C57048 UefiPxe4BcDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OptionRomPkg/Bus/Usb/UsbNetworking/Ax88772b/Ax88772b/DEBUG/Ax88772b.dll 0xB9C10000
Loading driver at 0x000B9C00000 EntryPoint=0x000B9C10048 Ax88772b.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/Block/DwUfsHcDxe/DwUfsHcDxe/DEBUG/DwUfsHcDxe.dll 0xB9BA4000
Loading driver at 0x000B9BA3000 EntryPoint=0x000B9BA4048 DwUfsHcDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Ufs/UfsPassThruDxe/UfsPassThruDxe/DEBUG/UfsPassThruDxe.dll 0xB9B92000
Loading driver at 0x000B9B91000 EntryPoint=0x000B9B92048 UfsPassThruDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Scsi/ScsiBusDxe/ScsiBusDxe/DEBUG/ScsiBus.dll 0xB9B85000
Loading driver at 0x000B9B84000 EntryPoint=0x000B9B85048 ScsiBus.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Scsi/ScsiDiskDxe/ScsiDiskDxe/DEBUG/ScsiDisk.dll 0xB9B6F000
Loading driver at 0x000B9B6E000 EntryPoint=0x000B9B6F048 ScsiDisk.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/DiskIoDxe/DiskIoDxe/DEBUG/DiskIoDxe.dll 0xB9B62000
Loading driver at 0x000B9B61000 EntryPoint=0x000B9B62048 DiskIoDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.dll 0xB9B53000
Loading driver at 0x000B9B52000 EntryPoint=0x000B9B53048 PartitionDxe.efi
add-symbol-file /work/jenkins/workspace/ap-uefi-bin/EDK2_ARCH/ARM/EDK2_BINARY/FatPkg/label/sas-sw/Build/Fat/RELEASE_GCC49/AARCH64/FatPkg/EnhancedFatDxe/Fat/DEBUG/Fat.dll 0xBEF55260
Loading driver at 0x000BEF55000 EntryPoint=0x000BEF552A8 Fat.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UnicodeCollation/EnglishDxe/EnglishDxe/DEBUG/EnglishDxe.dll 0xBA115000
Loading driver at 0x000BA114000 EntryPoint=0x000BA115048 EnglishDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/DEBUG/UsbBusDxe.dll 0xB9B3E000
Loading driver at 0x000B9B3D000 EntryPoint=0x000B9B3E048 UsbBusDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/MdeModulePkg/Bus/Usb/UsbMassStorageDxe/UsbMassStorageDxe/DEBUG/UsbMassStorageDxe.dll 0xB9B2F000
Loading driver at 0x000B9B2E000 EntryPoint=0x000B9B2F048 UsbMassStorageDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Platforms/Hisilicon/HiKey960/HiKey960UsbDxe/HiKey960UsbDxe/DEBUG/HiKey960UsbDxe.dll 0xB9B1D000
Loading driver at 0x000B9B1C000 EntryPoint=0x000B9B1D048 HiKey960UsbDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Platforms/Hisilicon/HiKey960/HiKey960FastbootDxe/HiKey960FastbootDxe/DEBUG/HiKey960FastbootDxe.dll 0xB9B08000
Loading driver at 0x000B9B07000 EntryPoint=0x000B9B08048 HiKey960FastbootDxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/Usb/DwUsb3Dxe/DwUsb3Dxe/DEBUG/DwUsb3Dxe.dll 0xB9AF9000
Loading driver at 0x000B9AF8000 EntryPoint=0x000B9AF9048 DwUsb3Dxe.efi
add-symbol-file /home/buildslave/workspace/96boards-reference-uefi-staging/85/edk2/Build/HiKey960/DEBUG_GCC5/AARCH64/EmbeddedPkg/Drivers/AndroidFastbootTransportUsbDxe/FastbootTransportUsbDxe/DEBUG/FastbootTransportUsbDxe.dll 0xB9AF1000
Loading driver at 0x000B9AF0000 EntryPoint=0x000B9AF1048 FastbootTransportUsbDxe.efi

Press ESCAPE for boot options UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
.[Bds]Booting Boot from SD
[Bds]Booting Grub
Loading driver at 0x000B9A3B000 EntryPoint=0x000B9A3B400
Loading driver at 0x000B9A3B000 EntryPoint=0x000B9A3B400
Welcome to GRUB!

error: no suitable video mode found.

Any thoughts or suggestions would be appreciated. Thanks again for your help.

-M

[leo-yan]

@dradspd

Press ESCAPE for boot options UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
UfsExecScsiCmds() fails with Target Failure
.[Bds]Booting Boot from SD
[Bds]Booting Grub
Loading driver at 0x000B9A3B000 EntryPoint=0x000B9A3B400
Loading driver at 0x000B9A3B000 EntryPoint=0x000B9A3B400
Welcome to GRUB!

error: no suitable video mode found.

From this log, I suspect you are not using correct grubaa64.efi. If so, please refer the page [1] sections “Building GRUB” and “Update boot.img” for updating the new grub app for UEFI.

[1] HiKey960 - Xen

[leo-yan]

Hi @scorpionzezz,

@scorpionzezz

I managed to install the xen-tools as well
If you try to run “xl list”, can you see the Domain-0?

I built xen-tools by self but I cannot see the Domain-o with command “xl list”:

root@linaro-developer:~/xen# xl list
Name                                        ID   Mem VCPUs      State   Time(s)

Alternatively, if input “CTRL-a” for three times, the console will switch from Dom-0 to hypervisor console; so after input character ‘q’ it will dump info as below:

(XEN) 'q' pressed -> dumping domain info (now = 548585364583)
(XEN) General information for domain 0:
(XEN)     refcnt=3 dying=0 pause_count=0
(XEN)     nr_pages=131072 xenheap_pages=2 shared_pages=0 paged_pages=0 dirty_cpus={} max_pages=4294967295
(XEN)     handle=00000000-0000-0000-0000-000000000000 vm_assist=00000020
(XEN) p2m mappings for domain 0 (vmid 1):
(XEN)   1G mappings: 0 (shattered 0)
(XEN)   2M mappings: 272 (shattered 1)
(XEN)   4K mappings: 1096
(XEN) Rangesets belonging to domain 0:
(XEN)     Interrupts { 34-37, 56-59, 76-78, 80-81, 106-110, 113, 116-143, 146, 148, 150-151, 171-173, 175, 177, 191, 193, 224-225, 310, 314-317, 344, 346 }
(XEN)     I/O Memory { 20200-20400, 32000-32101, e896b-e896d, e8a06-e8a1d, e8a1f-e8a21, f3f20-f3f60, f4000-f4001, f5000-f5002, f6000-f8000, fdf00-fdf06, fdf0b-fdf0d, fdf30-fdf31, ff100-ff201, ff37e-ff380, ff3b0-ff3b7, ff3fd-ff400, ffd68-ffd69, ffd71-ffd75, ffd7e-ffd7f, fff04-fff05, fff0a-fff12, fff14-fff15, fff1d-fff1e, fff30-fff32, fff34-fff36 }
(XEN) NODE affinity for domain 0: [0]
(XEN) VCPU information and callbacks for domain 0:
(XEN)     VCPU0: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={0} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 0) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU1: CPU1 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={1} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 1) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU2: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={2} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 2) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU3: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={3} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 3) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU4: CPU4 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={4} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 4) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU5: CPU5 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={5} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 5) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU6: CPU6 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={6} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 6) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN)     VCPU7: CPU7 [has=F] poll=0 upcall_pend=00 upcall_mask=01 
(XEN)     cpu_hard_affinity={7} cpu_soft_affinity={0-7}
(XEN)     pause_count=0 pause_flags=1
(XEN) GICH_LRs (vcpu 7) mask=0
(XEN)    VCPU_LR[0]=0
(XEN)    VCPU_LR[1]=0
(XEN)    VCPU_LR[2]=0
(XEN)    VCPU_LR[3]=0
(XEN) No periodic timer
(XEN) Notifying guest 0:0 (virq 1, port 0)
(XEN) Notifying guest 0:1 (virq 1, port 0)
(XEN) Notifying guest 0:2 (virq 1, port 0)
(XEN) Notifying guest 0:3 (virq 1, port 0)
(XEN) Notifying guest 0:4 (virq 1, port 0)
(XEN) Notifying guest 0:5 (virq 1, port 0)
(XEN) Notifying guest 0:6 (virq 1, port 0)
(XEN) Notifying guest 0:7 (virq 1, port 0)

[dradspd]

Hi Leo and scorpionzezz,

I would suggest verifying whether xencommons and related services are running by default.

Leo, I still have not had any luck porting Xen with the images that you suggested. I recompiled and regenerated grubaa64.efi but the result is the same. I believed the grub messages that I included on the previous log were captured after the system failed to boot xen and right before booting Debian. As I indicated, Debian boots fine. Not sure how to proceed going forward. I will continue to try nevertheless. Thanks for all your help.

[leo-yan]

Hi @dradspd,

Thanks a lot for reminding for “xencommons”, after start xencommons service, now I can see below log:

root@linaro-developer:~# /etc/init.d/xencommons start
Starting /usr/local/sbin/oxenstored...Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenconsoled...
Starting QEMU as disk backend for dom0
root@linaro-developer:~# 
root@linaro-developer:~# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     8     r-----      14.0

[jackyard]

hi,
I just try to enable xen on hikey960, exactly followed xen wiki, but I got this error.

(XEN) Adding cpu 7 to runqueue 0
(XEN) CPU 7 booted.
(XEN) Brought up 8 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) I/O virtualisation disabled
(XEN) build-id: 81490b5a1308bf52325d48d2d05b8f7d5df4b8f0
(XEN) alternatives: Patching with alt table 00000000002abbd8 → 00000000002ac220
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Domd0 kernel from boot module @ 00000000b8eb2000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000030000000 (256MB)
(XEN) BANK[1] 0x00000090000000-0x000000a0000000 (256MB)
(XEN) Grant table range: 0x000000b8d86000-0x000000b8dc6000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 00000000b8eb2000 to 0000000020080000-00000000214bba00
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Unable to copy the kernel in the hwdom memory
(XEN) ****************************************

[jackyard]

hi, @dradspd
I saw your first failure log is the same as mine, would you mind share me how do you fix this?

[dradspd]

Hi,

Which version of ARM-TF/UEFI and Linaro Debian did you use? Have you tried the version provided by Leo on his earlier reply? Unfortunately, I was never able to solve the Xen booting issues even with the binaries suggested by Leo.

Please do let me know if you get it to work. Good luck.

[jackyard]

@dradspd

Which version of ARM-TF/UEFI and Linaro Debian did you use? Have you tried the version provided by Leo on his earlier reply?

hi, @dradspd
I tried my own Linux kernel and Leo’s 5.0 kernel, the results are the same, All show that
(XEN) Unable to copy the kernel in the hwdom memory
I completely got lost.

[jackyard]

@leo-yan

We can flash the prebuilt boot image: boot-linaro-stretch-developer-hikey-20180828-19.img.gz

hi, @leo-yan
if you flashed this boot image, how do you make its grub support Xen boot command, such as: xe_hypervisor?

[leo-yan]

I changed grub config file /boot/grub/grub.cfg in rootFS and added one menu entry as below:

menuentry 'Xen' {                                                                           
        xen_hypervisor /boot/xen.efi loglvl=all console=dtuart dtuart=/soc/serial@fff32000 efi=no-rs hmp-unsafe dom0_vcpus_pin                                                          
        xen_module /boot/Image console=tty0 console=hvc0 root=/dev/sdd13 rw efi=noruntime   
        devicetree /boot/Image.dtb                                                          
}

[jackyard]

@jackyard

boot-linaro-stretch-developer-hikey-20180828-19.img.gz

hi, @leo-yan
you suggested above boot image is grub 2.02. doesn’t support xen_hypervisor.
I think, you didn’t use boot-linaro-stretch-developer-hikey-20180828-19.img.gz.

[leo-yan]

Hi @jackyard,

I am using this boot image, but I manually built grubaa64.efi and updated it into the boot image. Uploaded the image on the server for your reference [1].

[1] http://people.linaro.org/~leo.yan/virtualization/boot-linaro-stretch-developer-hikey-20180828-19.img

[jackyard]

hi, @leo-yan
thanks, actally, I have updated the my grubaa64, and now using your 5.0 kernel. Also using master branch xen, but still cannot fix below error:

update: I just tried your boot image, its grub is 2.03, but unfortunately, problem is still there.
I think, problem is on the Xen side.
this is my xen version:

Xen 4.12.0-rc (c/s Thu Jan 24 14:03:48 2019 +0000 git:755eb64) EFI loader
Xen 4.12.0-rc
(XEN) Xen version 4.12.0-rc (root@) (aarch64-linux-gnu-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701]) debug=y Tue Mar 5 12:47:38 CET 2019
(XEN) Latest ChangeSet: Thu Jan 24 14:03:48 2019 +0000 git:755eb64

(XEN) I/O virtualisation disabled
(XEN) build-id: 81490b5a1308bf52325d48d2d05b8f7d5df4b8f0
(XEN) alternatives: Patching with alt table 00000000002abbd8 -> 00000000002ac220
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Domd0 kernel from boot module @ 00000000b8eb2000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000030000000 (256MB)
(XEN) BANK[1] 0x00000090000000-0x000000a0000000 (256MB)
(XEN) Grant table range: 0x000000b8d86000-0x000000b8dc6000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 00000000b8eb2000 to 0000000020080000-00000000214bba00
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Unable to copy the kernel in the hwdom memory
(XEN) ****************************************

[leo-yan]

Comparing the Xen log the kernel loading target address is quite different; you could see on my board (with 4GB memory) the kernel image is copied to 0xC000_0000 (above 3GB), as I know some Hikey960 boards only have 3GB DDR parts; could you confirm if this is your case?

And I am a bit concern about the booting images (ARM-TF and UEFI), which I built on my local PC with pretty new code bases (AFAIK ARM-TF and UEFI both have been updated) so I don’t use booting images from http://snapshots.linaro.org/96boards/reference-platform/components/uefi-staging/latest/hikey960/debug/ ; could you also confirm which booting images you are using?

(XEN) build-id: ea6618b0837011e4049a83a5185c9de93fb6e64b
(XEN) alternatives: Patching with alt table 00000000002abc00 -> 00000000002ac224
(XEN) CPU7 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) CPU4 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) CPU6 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) CPU5 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Domd0 kernel from boot module @ 00000000b8539000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x000000c0000000-0x000000e0000000 (512MB)
(XEN) Grant table range: 0x000000b840b000-0x000000b844b000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 00000000b8539000 to 00000000c0080000-00000000c14bba00
(XEN) Loading dom0 DTB to 0x00000000c8000000-0x00000000c8008852
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All