Hikey970 boot fails when Source built system.img is flashed

Hi,

I tried booting Hikey970 using the source built images.

The board boots successfully when flashed with cache.img, boot.img but fails when flashed with
system.img

following is the boot log:


[    5.888659] uart-pl011 fff32000.serial: no DMA platform data                 
[    5.896546] Freeing unused kernel memory: 4096K                              
[    5.908197] init: init first stage started!                                  
[    5.912580] init: Using Android DT directory /proc/device-tree/firmware/andr/
[    5.920093] init: First stage mount skipped (missing/incompatible fstab in d)
[    5.928143] init: Skipped setting INIT_AVB_VERSION (not in recovery mode)    
[    5.935056] init: Loading SELinux policy                                     
[    5.943601] SELinux: 4096 avtab hash slots, 15474 rules.                     
[    5.956781] SELinux: 4096 avtab hash slots, 15474 rules.                     
[    5.962120] (stk) : timed out waiting for ldisc to be un-installed           
SELinux:  1 users, 4 roles, 1098 types, 0 bools, 1 sens, 1024 cats              
[    5.974247] SELinux:  92 classes, 15474 rules                                
[    5.980276] SELinux:  Class bpf not defined in policy.                       
[    5.985428] SELinux: the above unknown classes and permissions will be denied
[    5.992585] SELinux:  Completing initialization.                             
[    5.997208] SELinux:  Setting up existing superblocks.                       
[    6.037557] audit: type=1403 audit(68.399:2): policy loaded auid=4294967295 5
[    6.046133] selinux: SELinux: Loaded policy from /sepolicy                   
[    6.046133]                                                                  
[    6.056520] selinux: SELinux: Loaded file_contexts                           
[    6.056520]                                                                  
[    6.065588] (stk) :ldisc_install = 1[    6.068297] init: init second stage s!
[    6.073028] init: Using Android DT directory /proc/device-tree/firmware/andr/
[    6.075596] selinux: SELinux: Loaded file_contexts                           
[    6.075596]                                                                  
[    6.075616] init: Running restorecon...                                      
[    6.079281] selinux: SELinux:  Could not stat /dev/block: No such file or di.
[    6.079281]                                                                  
[    6.079489] init: waitid failed: No child processes                          
[    6.079561] init: Couldn't load property file '/system/etc/prop.default': opy
[    6.079592] init: Couldn't load property file '/prop.default': open() failedy
[    6.079925] init: Couldn't load property file '/odm/default.prop': open() fay
[    6.079985] init: Couldn't load property file '/vendor/default.prop': open()y
[    6.172390] cgroup: init (1) created nested cgroup for controller "memory" w.
[    6.178933] ueventd: ueventd started!                                        
[    6.181178] selinux: SELinux: Loaded file_contexts                           
[    6.181178]                                                                  
[    6.181236] ueventd: Parsing file /ueventd.rc...                             
[    6.181965] ueventd: Parsing file /vendor/ueventd.rc...                      
[    6.182003] ueventd: Unable to read config file '/vendor/ueventd.rc': open()y
[    6.182025] ueventd: Parsing file /odm/ueventd.rc...                         
[    6.182050] ueventd: Unable to read config file '/odm/ueventd.rc': open() fay
[    6.182150] ueventd: Parsing file /ueventd.hikey970.rc...                    
[    6.237080] cgroup: "memory" requires setting use_hierarchy to 1 on the root 
[    6.244732] audit: type=1400 audit(68.607:3): avc:  denied  { create } for  1
[    6.364019] ueventd: Coldboot took 0.181 seconds                             
[    6.389650] Registered swp emulation handler                                 
[    6.395699] audit: type=1400 audit(68.759:4): avc:  denied  { add_name } for1
[    6.412344] audit: type=1400 audit(68.775:5): avc:  denied  { create } for  1
[    6.439039] EXT4-fs (sdd12): mounted filesystem without journal. Opts: (null)
[    6.439338] watchdogd: watchdogd started (interval 10, margin 20)!           
[    6.439363] watchdogd: Failed to open /dev/watchdog: No such file or directoy
[    6.461164] EXT4-fs (sdd5): Ignoring removed nomblk_io_submit option         
[    6.473781] EXT4-fs (sdd5): recovery complete                                
[    6.478377] EXT4-fs (sdd5): mounted filesystem with ordered data mode. Opts:t
[    6.509930] EXT4-fs (sdd5): mounted filesystem with ordered data mode. Opts:1
[    6.524194] EXT4-fs (sdd15): Ignoring removed nomblk_io_submit option        
[    6.534154] EXT4-fs (sdd15): recovery complete                               
[    6.538835] EXT4-fs (sdd15): mounted filesystem with ordered data mode. Optst
[    6.566725] EXT4-fs (sdd15): mounted filesystem with ordered data mode. Opts1
[    6.583472] audit: type=1400 audit(68.947:6): avc:  denied  { setattr } for 1
[    6.602867] audit: type=1400 audit(68.967:7): avc:  denied  { write } for  p1
[    6.620434] audit: type=1400 audit(68.983:8): avc:  denied  { add_name } for1
[    6.637660] audit: type=1400 audit(68.999:9): avc:  denied  { create } for  1
[    6.673112] audit: type=1400 audit(69.035:10): avc:  denied  { write } for  1
[    6.692831] file system registered                                           
[    7.174535] (stk) :ldisc installation timeout[    7.178743] (stk) :ldisc_ins0
(stk) : timed out waiting for ldisc to be un-installed[    8.310821] (stk) :ldi1
(stk) :ldisc installation timeout[    9.321438] (stk) :ldisc_install = 0        
(stk) : timed out waiting for ldisc to be un-installed[   10.454260] (stk) :ldi1
(stk) :ldisc installation timeout[   11.465411] (stk) :ldisc_install = 0        
(stk) : timed out waiting for ldisc to be un-installed[   12.598250] (stk) :ldi1
(stk) :ldisc installation timeout[   13.609410] (stk) :ldisc_install = 0        
(stk) : timed out waiting for ldisc to be un-installed[   14.742252] (stk) :ldi1
(stk) :ldisc installation timeout[   15.753203] (stk) :ldisc_install = 0        
(stk) : timed out waiting for ldisc to be un-installed[   16.781858] Bluetooth:2

Can anyone please suggest what might have gone wrong here?

Thanks and regards,
Nishad

I am having the same problem. Did you find a solution by any chance?

Sorry… not yet… I think there has been some change in the AOSP source code just recently. They seem to have integrated the kernel in the AOSP itself. I am downloading it again and trying to build it.

I am still facing the issue after downloading the latest source.

I happened to try to find out the difference between the pre-built system.img and source-built system.img
by mounting them on a normal directory on my host machine.

There a re a lot of differences. It seems there might be some system related configs missing while
building the source AOSP.

Looking into this.

Any update on your end?

Thanks and regards,
Nishad

I’ve pulled out kernel configuration from the stock boot.img and found out that kernel is in version 4.9.59 and following build instructions builds version 4.9.78. Building it with version 4.9.59 ended up in kernel panic for me. I was hoping to find missing/undocumented kernel configuration but the differences were insignificant as far as I can tell.

Ok, thanks for the info.

I was looking at the difference between the pre-built system.img and source AOSP built system.img.
For this I converted them to raw images first using simg2img and then mounted them on a directory.

The source built system.img has a directory called /system while the prebuilt one does not. When i checked the contents of the /system folder, they are almost the same as that of the prebuilt image.
Check this post for more details:

So I deleted all the other stuff in the source built system.img and brought the contents of the system directory to the top directory and deleted the system folder.

Then I converted back the raw system image to system.img using img2simg tool and flashed it to the hikey970 board.

The board successfully booted.

I am now checking if there is something missing in the AOSP configuration or build process which makes it build an incorrect system.img.

Thanks and regards,
Nishad

Thanks for sharing your progress. I’ve identified piece of the code that is responsible for the manipulation of the system partition. If you look at build/make/tools/releasetools/build_image.py you will find a function called SetUpInDirAndFsConfig . Its last part is responsible for merging root and system images which is where the difference comes from. I’ve modified it so it looks like:

def SetUpInDirAndFsConfig(origin_in, prop_dict):
  """Returns the in_dir and fs_config that should be used for image building.
  When building system.img for all targets, it creates and returns a staged dir
  that combines the contents of /system (i.e. in the given in_dir) and root.

  Args:
    origin_in: Path to the input directory.
    prop_dict: A property dict that contains info like partition size. Values
        may be updated.

  Returns:
    A tuple of in_dir and fs_config that should be used to build the image.
  """
  fs_config = prop_dict.get("fs_config")

  if prop_dict["mount_point"] == "system_other":
    prop_dict["mount_point"] = "system"
    return origin_in, fs_config

  if prop_dict["mount_point"] != "system":
    return origin_in, fs_config

  if "first_pass" in prop_dict:
    prop_dict["mount_point"] = "/"
    return prop_dict["first_pass"]

  return origin_in, fs_config

Using this i moved past the part with failing ldisk. My boot hangs on

[    7.104165] vdc: Waited 360ms for vold                                                                                                                                                                                                      
[    7.109328] vdc: Failed: Status(-8): '0: '                                                                                                                                                                                                  
[    7.138966] SELinux:  Context u:object_r:flags_health_check_exec:s0 is not valid (left unmapped).                                                                                                                                           
[    7.156623] SELinux:  Context u:object_r:hal_system_suspend_default_exec:s0 is not valid (left unmapped).                                                                                                                                   
[    7.175872] SELinux:  Context u:object_r:apexd_exec:s0 is not valid (left unmapped).                                                                                                                                                        
[    7.185433] SELinux:  Context u:object_r:gpuservice_exec:s0 is not valid (left unmapped).                                                                                                                                                   
[    7.205153] healthd: No battery devices found                                                                                                                                                                                               
[    7.210241] file system registered                                                                                                                                                                                                          
console:/ $ [    7.889495] audit: rate limit exceeded                                                                                                                                                                                          
[   12.149206] audit: rate limit exceeded                                                                                                                                                                                                      
[   17.154843] audit: rate limit exceeded                                                                                                                                                                                                      
[   22.166961] audit: rate limit exceeded                                                                                                                                                                                                      
[   27.182993] audit: rate limit exceeded                                                                                                                                                                                                      
[   82.260968] init: couldn't write 6648 to /dev/cpuset/camera-daemon/tasks: No such file or directory

I’ll look into that merging code and see if and how it might be fixed.

Hi,

Thanks for the information.

Can you check whether adb works where the boot hangs on in your case.

I haven’t tried your method yet, but some time back I was able to see the boot messages as the ones
you are getting towards the end, but my adb worked.

Thanks and regards,
Nishad

My adb is not working at the moment boot hangs.

Hi,

While investigating the source for the pre-built images,
I came across the following branch as the closest to the version of the pre-built images.

https://android.googlesource.com/platform/manifest/+/pie-dr1-dev

After building the same with source built ramdisk.img, boot.img, system.img, cache.img and userdata.img
the board boots successfully with HDMI display.

Note: Change the size of the userdata image to 24GB from 53GB in “device/linaro/hikey/hikey970/BoardConfig.mk”

file as follows:

include device/linaro/hikey/BoardConfigCommon.mk

TARGET_BOOTLOADER_BOARD_NAME := hikey970
TARGET_BOARD_PLATFORM := hikey970

TARGET_CPU_VARIANT := cortex-a73
TARGET_2ND_CPU_VARIANT := cortex-a73

TARGET_NO_DTIMAGE := false

BOARD_KERNEL_CMDLINE := androidboot.hardware=hikey970 console=ttyFIQ0 androidboot.console=ttyFIQ0
BOARD_KERNEL_CMDLINE += firmware_class.path=/system/etc/firmware loglevel=15
BOARD_KERNEL_CMDLINE += androidboot.selinux=permissive
ifneq ($(TARGET_SENSOR_MEZZANINE),)
BOARD_KERNEL_CMDLINE += overlay_mgr.overlay_dt_entry=hardware_cfg_$(TARGET_SENSOR_MEZZANINE)
endif
BOARD_MKBOOTIMG_ARGS := --base 0x0 --tags_offset 0x07a00000 --kernel_offset 0x00080000 --ramdisk_offset 0x07c00000

BOARD_SYSTEMIMAGE_PARTITION_SIZE := 4915724288  # 4688MB
BOARD_USERDATAIMAGE_PARTITION_SIZE := 25769803776 # 24GB
BOARD_CACHEIMAGE_PARTITION_SIZE := 268435456       # 256MB
BOARD_FLASH_BLOCK_SIZE := 512
~                                     

Thanks and regards,
Nishad

Hi Nishad, thank you so much for leaving the response. I tried building that branch myself, but I am stuck in the same spot. You’re using only the images you’ve built or some of them are stock images? When building the kernel you’re linking your own ramdisk or the stock image? Can you think of any other changes you might have done? Thanks in advance!

Hi,

I am using my own ramdisk image (Built as per the procedure given on the website from AOSP) to build the kernel.

Initially, I flashed all the stock images as per the procedure,

$ sudo fastboot flash ptable prm_ptable.img
$ sudo fastboot flash xloader sec_xloader.img
$ sudo fastboot flash fastboot l-loader.bin
$ sudo fastboot flash fip fip.bin
$ sudo fastboot flash boot boot.img
$ sudo fastboot flash cache cache.img
$ sudo fastboot flash system system.img
$ sudo fastboot flash userdata userdata.img

I then replaced the following images with the images that I built.

$ sudo fastboot flash boot boot.img (Built using the ramdisk.img that I had built)
$ sudo fastboot flash cache cache.img
$ sudo fastboot flash system system.img
$ sudo fastboot flash userdata userdata.img

Please also note that
You need to flash all four of the above images. For eg: if you flash only the boot.img, and other images are stock images then the board will not boot.
There seems to be some link between the ramdisk and the system.img.

Other than that, there were minor changes that we need to do:

1.changing the following build script:

make -j$[NCPU*2] Image.gz modules

to

make -j$[NCPU*2]

So that the kernel/hikey-linaro/arch/arm64/boot/Image.gz-dtb is created.

  1. Adding the following configs to the /arch/arm64/configs/hikey970_defconfig, as i had mentioned eariler:
CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE=y
CONFIG_BUILD_ARM64_APPENDED_KERNEL_IMAGE_NAME=Image.gz-dtb
CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE_NAMES=“hisilicon/kirin970-hikey970”
  1. Commenting out the hostapd.se.

Happy to help.

Thanks and regards,
Nishad

Hi Nishad, thank you a lot. With your help I managed to build it as well. In my case the problem was that build_kernel.sh generated boot.img (that is the one working) was overwritten by aosp build with a boot.img that was failing for me.