My mistake. I thought hikey960 still used eMMC and first UFS board was hikey970 so when I say sdd10 I was confused (however I was wrong hikey960 uses UFS too).
no problem 
I downloaded the Fastboot files from: Debian Downloads for HiKey960 - 96Boards
and then use boot-linaro-stretch-developer-hikey-20181217-25.img and rootfs-linaro-stretch-developer-hikey-20181217-25.img
it works fine, but when I tried to make own rootfs (like on xen wiki), the rootfs does not detected somehow.
You didn’t share full boot logs so I can’t check this myself but the kernel normally sends out debug messages as it parses the partition tables… did if find any partitions at all (e.g. has it failed to scan the UFS or is it simply that sdd10 does not exist).
PS I have no practical experience with Xen (some architecture knowledge but I’ve never actually tried to use it)… however “tried to make own rootfs” could easily create problems unrelated to Xen which is why I’m asking the above 
HI @danielt,
here is the full logs: Xen 4.11.1 (c/s Thu Nov 29 15:04:11 2018 +0100 git:96cbd0893f) EFI loader Xen - Pastebin.com
I tried also to did this: Board Recovery for HiKey960 - 96Boards
but still without any success to boot the Xen. Other things I tried to do is using the SDCard, I tried to flash the image to the SD card and replace the GRUB (from my own compiled GRUB) on the /boot/efi, but somehow when I rebooted it still uses the old GRUB instead of the one that I compiled.
Hmn… the logs show problems with UFS (messages like: ufshcd-hi3660 ff3b0000.ufs: ufshcd_query_flag: Sending flag query for idn 3 failed, err = -11) so the kernel is not enumerating any of the partitions.
If this doesn’t ring any bells with @Vikram then I think you might have to ask what could cause this type of problem within the Xen community.
In the mean time though it does look from the logs as though USB3 is working fine. You might be able to unblock yourself in the mean time but putting the rootfs on a USB stick.
Hello scorpionzezz,
We faced same issue with hynix UFS in linux kernel 4.14. UFS driver initialization has a issue. You need to add 2 lines of code which solves this issue. You can take 4.9 kernel as reference and fix this issue. Currently i dont have access to the code base
Hi @Vikram,
when you have access, can you zip all the images files and the UEFI that you used?
Cheers!
Hi @scorpionzezz,
Could you check this patch [1] has been included in your local code base? This patch has been merged into AOSP hikey kernel code base.
[1] https://android-review.googlesource.com/c/kernel/hikey-linaro/+/750912
Hi @leo-yan
thanks! I will take a look, I need to downgrade the UEFI first, because now I’m using the UEFI from 4.19.5 kernel based Debian snapshot image release for HiKey960 - #12 by Mani and seems like it even does want to load the Xen.
I will get back to you will the result.
Cheers!
Please refer [1] for enabling grub in UEFI. Just remind, you could see the doc says to put xen related binaries into the rootfs’s /boot folder, which is different from putting images into the boot_xxx.img.
[1] HiKey960 - Xen
Hi,
Have you had any success booting xen since your last post? I have gone to similar steps using the latest boot and rootfs (#27) images from Linaro Snapshots and creating and recompiling a new rootfs and grub2 as instructed on the Xen’s hikey960 wiki. Debian with vmlinuz-4.19.5-hikey boots fine but I can’t get xen to boot. I have attached the boot log for you to see. I appreciate if you or anyone could shed some lights on this. Thank you.
Hi,
I’m not yet try again since I have something else to do, but yes last time I faced similar issues, seems like it is more serious than before since at the first step, Xen does not able to “up” the CPUs.
Maybe @danielt or @leo-yan have any idea?
Cheers!
Thank you for you reply. I forgot to mention that although I followed Xen Hikey960 wiki page closely, I used the hikey kernel tree provided by Manivannan instead to produce the Image and Image.dtb to boot Xen. I was hoping to boot Xen with a working WiFi and Bluetooth. Previously, I have successfully booted up Xen using roofts and boot extracted from linaro-stretch-developer-20180416-89.tar.gz. But Wifi and Bluetooth did not work.
Be honest, I tried Xen on Hikey620 but don’t know well for Xen implementation; from the log, the big CPUs cannot boot up is one issue but it’s not the critical cause for booting failure.
Just want to confirm what’s your building steps for kernel so firstly I think we should ensure the XEN related configurations should be included into kernel building.
From the log another possibility is the Image size is out of range of the reserved memory region in Xen.
Hi Leo, thank you for your reply.
I did modify the .config file before building Mani’s kernel source tree to ensure that Xen related kernel flags are included and that CONFIG_USB_RTL8152 is set to Y instead. Since the config file is too large to load, I did a grep on the config file to list all the Xen related kernel flag:
CONFIG_XEN_DOM0=y
CONFIG_XEN=y
CONFIG_NET_9P_XEN is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_NETXEN_NIC is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_TCG_XEN is not set
CONFIG_XEN_WDT=y
CONFIG_DRM_XEN=y
CONFIG_DRM_XEN_FRONTEND=m
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_SND_XEN_FRONTEND=y
CONFIG_MMC_SDHCI_XENON=y
Xen driver support
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES_DEFAULT=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GNTDEV_DMABUF=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_GRANT_DMA_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PVCALLS_FRONTEND=y
CONFIG_XEN_PVCALLS_BACKEND=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
My kernel build steps are followed verbatim to Xen’s Hikey960 wiki:
export CROSS_COMPILE and ARCH to aarch64-linux-gnu, arm64
make menuconfig to verify all Xen flags are set
make -j8
copy ARM64 boot Image and Image.dtb to the rootfs image
I have tried many avenues to load Xen. For example, creating a new rootfs and copy all the files from the rootfs image provided by Mani with additional files including xen.efi to the boot directory (xen’s wiki way). I also tried just booting Mani’s rootfs image and modified the existing grub.cfg to include Xen’s menuentry extracted from Xen’s wiki. Grub2 is also recompiled to include xen_boot module and replaced with the corresponding boot.image
Finally, I tried adding hmp-unsafe and dom0_vcpus_pin to Xen commandline. All 8 cpus were brought up but still failed to boot.
Leo,
I have copied and pasted the latest boot log for you to see. Thank you again for looking into this.
Loading xen.efi…
Loading Image…
Loading Image.dtb…
Loading driver at 0x000B8362000 EntryPoint=0x000B83FE00C
Loading driver at 0x000B8362000 EntryPoint=0x000B83FE00C
Using modules provided by bootloader in FDT
Xen 4.12.0-rc (c/s Thu Jan 24 14:03:48 2019 +0000 git:755eb64) EFI loader
Xen 4.12.0-rc
(XEN) Xen version 4.12.0-rc (marge@) (aarch64-linux-gnu-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609) debug=y Tue Feb 5 14:17:35 EST 2019
(XEN) Latest ChangeSet: Thu Jan 24 14:03:48 2019 +0000 git:755eb64
(XEN) Processor: 410fd034: “ARM Limited”, variant: 0x0, part 0xd03, rev 0x4
(XEN) 64-bit Execution:
(XEN) Processor Features: 0000000000002222 0000000000000000
(XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD
(XEN) Debug Features: 0000000010305106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001122 0000000000000000
(XEN) ISA Features: 0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 00000131:00011011
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer Security
(XEN) Debug Features: 03010066
(XEN) Auxiliary Features: 00000000
(XEN) Memory Model Features: 10201105 40000000 01260000 02102211
(XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.1
(XEN) Using PSCI v1.1
(XEN) SMP: Allowing 8 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 1920 KHz
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=00000000e82b1000
(XEN) gic_cpu_addr=00000000e82b2000
(XEN) gic_hyp_addr=00000000e82b4000
(XEN) gic_vcpu_addr=00000000e82b6000
(XEN) gic_maintenance_irq=25
(XEN) GICv2: 384 lines, 8 cpus, secure (IID 0200143b).
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Allocated console ring of 64 KiB.
(XEN) Bringing up CPU1
(XEN) Adding cpu 1 to runqueue 0
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) Adding cpu 2 to runqueue 0
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) Adding cpu 3 to runqueue 0
(XEN) CPU 3 booted.
(XEN) Bringing up CPU4
(XEN) Adding cpu 4 to runqueue 0
(XEN) CPU 4 booted.
(XEN) Bringing up CPU5
(XEN) Adding cpu 5 to runqueue 0
(XEN) CPU 5 booted.
(XEN) Bringing up CPU6
(XEN) Adding cpu 6 to runqueue 0
(XEN) CPU 6 booted.
(XEN) Bringing up CPU7
(XEN) Adding cpu 7 to runqueue 0
(XEN) CPU 7 booted.
(XEN) Brought up 8 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) Data Abort Trap. Syndrome=0x5
(XEN) Walking Hypervisor VA 0x8000c0000000 on CPU0 via TTBR 0x00000000b8468000
(XEN) 0TH[0x100] = 0x00400000b8475f7f
(XEN) 1ST[0x3] = 0x0000000000000000
(XEN) CPU0: Unexpected Trap: Data Abort
(XEN) ----[ Xen-4.12.0-rc arm64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) PC: 0000000000232eb0 strcasecmp+0x10/0x50
(XEN) LR: 0000000000203af4
(XEN) SP: 00000000002cfd00
(XEN) CPSR: 00000249 MODE:64-bit EL2h (Hypervisor, handler)
(XEN) X0: 00008000c0000000 X1: 000000000027dcc8 X2: ffffffffffffffff
(XEN) X3: 0000000000000000 X4: 000000000026a0c8 X5: 0000000000000000
(XEN) X6: 00008000c0000000 X7: fefefefefefefefe X8: ffffffffffffffff
(XEN) X9: fffefefefefefefe X10: 7fffffffffffffff X11: 0101010101010101
(XEN) X12: 0000000000000038 X13: 000000000026aae8 X14: 0000000000000020
(XEN) X15: 0000000000000000 X16: fffffff800000000 X17: 3d3d3d3d3d3d3d3d
(XEN) X18: 0000000000286000 X19: 00008000c0000000 X20: 000000000027dcc8
(XEN) X21: 00008000bffd3250 X22: 0000000000000001 X23: 0000000000000000
(XEN) X24: 0000000000000008 X25: 00000000002b0198 X26: 0000000000000000
(XEN) X27: 00000000c0000000 X28: 0000000000000012 FP: 00000000002cfd00
(XEN)
(XEN) VTCR_EL2: 80023558
(XEN) VTTBR_EL2: 0000da97f07ebcd0
(XEN)
(XEN) SCTLR_EL2: 30cd183d
(XEN) HCR_EL2: 000000000000003a
(XEN) TTBR0_EL2: 00000000b8468000
(XEN)
(XEN) ESR_EL2: 96000005
(XEN) HPFAR_EL2: 00000000d69becf0
(XEN) FAR_EL2: 00008000c0000000
(XEN)
(XEN) Xen stack trace from sp=00000000002cfd00:
(XEN) 00000000002cfd30 0000000000204ea8 000000000026abf0 0000000000000001
(XEN) 00000000002cfd50 ffffbe7400204fc0 00000000002cfd60 00000000002930ac
(XEN) 00000000002867b0 0000000000000001 0000000000286810 00008000bffd3250
(XEN) 00000000002cfda0 0000000000291140 00008000bffd3250 0000000000000000
(XEN) 0000000000000001 0000000000000008 0000000000280020 0000000100000001
(XEN) 00000000002cfdd0 0000000000290f38 0000000000000000 0000000000284c80
(XEN) 0000000000314448 0000000200000000 00000000002cfdf0 0000000000299d18
(XEN) 0000000000000008 0000000000284c80 00000000bffff4c0 00000000b83621dc
(XEN) 00000000b8362000 00000000b8162000 00000000b8356000 0000000000000000
(XEN) 0000000000000870 0000000000000000 0000000000000001 0000000000000001
(XEN) 0000000000000001 0000000000000001 0000000000000000 000000000000c000
(XEN) 00000000b8356000 000000008d30e000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000300000000 ffffffff00000000 0000040000000040
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<0000000000232eb0>] strcasecmp+0x10/0x50 (PC)
(XEN) [<0000000000203af4>] dt_device_is_compatible+0x3c/0x84 (LR)
(XEN) [<0000000000204ea8>] dt_match_node+0x70/0x114
(XEN) [<00000000002930ac>] device_init+0x7c/0xd8
(XEN) [<0000000000291140>] iommu_hardware_setup+0x34/0x68
(XEN) [<0000000000290f38>] iommu_setup+0x48/0x1d0
(XEN) [<0000000000299d18>] start_xen+0xb60/0xcc4
(XEN) [<00000000b83621dc>] 00000000b83621dc
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Data Abort
(XEN) ****************************************
Hi Vikram,
Can you share your xen config for android as guest OS?
Now I’m working on hikey970 to boot android xen as guest OS, I can now boot linux as guest OS on it.
Thank you very much!
For DMA mapping, there have one broken issue for Xen with recent kernel; could you confirm if the patch [1] can resolve this issue?
I tried mainline kernel on Hikey960 and found there still have some minor issues for enabling Xen (one is CMA initialization failure and another is for DMA mapping as mentioned in previous replying).
So I created the branch [1] and tested it can work well at my side with Xen and also verified WIFI and USB can work well (though this branch doesn’t include any DRM and display drivers for Hikey960). Just for your reference.
You could use below command to build Image and dtbs:
make hikey960_defconfig
make -j8 Image dtbs
P.s. Actually I reused ARM power team branch for Hikey960 mainline support [2], so need to credit to ARM power team for their much efforts.
[1] https://github.com/Leo-Yan/linux/tree/hikey960_mainline_5.0_xen
[2] git://linux-arm.org/linux-power.git branch: refs/heads/eas/next/integration_20190208_0000_topic_hikey960_mainline
Leo! Thank you so much for providing me possible solutions to overcome Xen booting issues. I will try it out today and provide you feedback on my result.
