Mounting /usr file system fails after moving it to SD card

Products Support HiKey 960
#1

I had a working recent Debian image flashed to one of my HiKey960 boards, when I decided to “optimise” it and “move” /home and /usr from the internal flash memory to the SD card. Namely, I created /etc/fstab as follows:

# device                mountpoint      filesystem      options         backup          fsck
/dev/mmcblk0p1          none            swap            defaults        0               0
/dev/mmcblk0p2          /sd             ext4            defaults        0               0
/sd/usr                 /usr            none            defaults,bind   0               0
/sd/home                /home           none            defaults,bind   0               0
/sd/var                 /var            none            defaults,bind   0               0
/sd/tmp                 /tmp            none            defaults,bind   0               0

and copied /usr to /sd/usr, etc., using rsync. Furthermore, I removed the original /usr directory from the flash memory.

Unfortunately, on reboot, I got:

[    4.597675] Btrfs loaded, crc32c=crc32c-generic                            
Scanning for Btrfs filesystems                                                                     
done.                                                                             
Begin: Will now check root file system ... fsck from util-linux 2.29.2        
[/sbin/fsck.ext4 (1) -- /dev/sdd13] fsck.ext4 -a -C0 /dev/sdd13                                    
rootfs: clean, 144975/1531648 files, 1282125/6309888 blocks                       
done.                                                                                                            
[    4.722658] EXT4-fs (sdd13): mounted filesystem with ordered data mode. Opts: (null)             
done.                                                                  
Begin: Mounting /usr file system ... mount: No such file or directory                               
done.                                                                                                    
Begin: Running /scripts/local-bottom ... done.                                                                                       
Begin: Running /scripts/init-bottom ... done.                                                                                  
/sbin/init: error while loading shared libraries: liblz4.so.1: cannot open shared object file: No such file or directory     
[    4.815298] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00         
[    4.815298]                                                                                                               
[    4.824450] CPU: 6 PID: 1 Comm: init Tainted: G S               4.15-hikey #1               
[    4.831581] Hardware name: HiKey960 (DT)                                                                                  
[    4.835498] Call trace:                                                                     
[    4.837951]  dump_backtrace+0x0/0x198                                                   
[    4.841609]  show_stack+0x14/0x20                           
[    4.844923]  dump_stack+0x98/0xb8                                                       
[    4.848235]  panic+0x114/0x27c                           
[    4.851284]  do_exit+0x95c/0x960                                                        
[    4.854505]  do_group_exit+0x34/0x98                                           
[    4.858075]  __wake_up_parent+0x0/0x28                                                                                                                                                                  
[    4.861819]  el0_svc_naked+0x20/0x24
[    4.865393] SMP: stopping secondary CPUs
[    4.869548] Kernel Offset: disabled
[    4.873031] CPU features: 0x082004
[    4.876425] Memory Limit: none
[    4.879487] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[    4.879487]
[    4.889422] WARNING: CPU: 6 PID: 1 at kernel/sched/core.c:1188 set_task_cpu+0x128/0x130
[    4.897423] Modules linked in: btrfs xor zstd_decompress zstd_compress xxhash raid6_pq
[    4.905350] CPU: 6 PID: 1 Comm: init Tainted: G S               4.15-hikey #1
[    4.912480] Hardware name: HiKey960 (DT)
[    4.916397] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    4.921183] pc : set_task_cpu+0x128/0x130
[    4.925186] lr : try_to_wake_up+0x13c/0x3c0
[    4.929362] sp : ffff000008033d20
[    4.932669] x29: ffff000008033d20 x28: ffff0000090c9000
[    4.937977] x27: ffff0000090e6c80 x26: 0000000000000000
[    4.943284] x25: ffff0000090af000 x24: ffff0000090c9000
[    4.948591] x23: 0000000000000080 x22: ffff8000bbe5ed7c
[    4.953897] x21: 0000000000000004 x20: 0000000000000000
[    4.959204] x19: ffff8000bbe5e580 x18: 0000000000000010
[    4.964510] x17: 0000000000000000 x16: 0000000000000000
[    4.969817] x15: 0000000000000000 x14: 0000000000000400
[    4.975123] x13: 0000000000000000 x12: 0000000000000000
[    4.980430] x11: ffff000008ac17f0 x10: ffff8000bffb0200
[    4.985736] x9 : 0000000000000000 x8 : 0000000000000007
[    4.991043] x7 : 000000000000034c x6 : ffff8000b9f49300
[    4.996349] x5 : 00000000000000ff x4 : 0000000000000000
[    5.001656] x3 : 0000000000000000 x2 : ffffffffffffffff
[    5.006962] x1 : ffff0000090c9df0 x0 : 0000000000000040
[    5.012269] Call trace:
[    5.014707]  set_task_cpu+0x128/0x130
[    5.018363]  try_to_wake_up+0x13c/0x3c0
[    5.022191]  wake_up_process+0x14/0x20
[    5.025936]  swake_up_locked+0x28/0x50
[    5.029678]  swake_up+0x20/0x38
[    5.032815]  rcu_gp_kthread_wake+0x34/0x48
[    5.036907]  note_gp_changes+0x84/0xb8
[    5.040650]  rcu_process_callbacks+0xb0/0x3f8
[    5.045000]  __do_softirq+0x114/0x214
[    5.048656]  irq_exit+0xc8/0xf8
[    5.051792]  __handle_domain_irq+0x60/0xb8
[    5.055881]  gic_handle_irq+0x58/0xb0
[    5.059536]  el1_irq+0xb0/0x128
[    5.062670]  panic+0x230/0x27c
[    5.065717]  do_exit+0x95c/0x960
[    5.068938]  do_group_exit+0x34/0x98
[    5.072507]  __wake_up_parent+0x0/0x28
[    5.076248]  el0_svc_naked+0x20/0x24
[    5.079817] ---[ end trace 7d2b1daab1ca0cfd ]---

The way I interpret this the kernel tries to use liblz4.so.1 from /usr before the SD card gets mounted, but it fails as the original /usr no longer exists.

I thought that I only had to reflash rootfs to fix this, but unfortunately I have now tried many things but still can’t make it work. Please help!

#2

What have I tried tried:

  1. Reflashing boot and rootfs:
$ sudo fastboot flash boot boot-linaro-stretch-developer-hikey-20190620-32.img
$ sudo fastboot flash system rootfs-linaro-stretch-developer-hikey-20190620-32.img
  1. Erasing boot and rootfs:
$ sudo fastboot erase boot
$ sudo fastboot erase system

and then reflashing them as per 1.
3) Formatting boot and rootfs and then reflashing them as per 1.
4) Booting with and without the SD card inserted.
5) Flashing base firmware as per https://github.com/96boards-hikey/tools-images-hikey960 and then trying to flash boot and rootfs.

Does anyone have any further ideas?

#3 
/dev/mmcblk0p2 /sd ext4 defaults 0 0 
/sd/usr /usr none defaults,bind 0 0 
/sd/home /home none defaults,bind 0 0 
/sd/var /var none defaults,bind 0 0 
/sd/tmp /tmp none defaults,bind 0 0

I don’t think this is the right method, since ‘/sd/usr’, ‘/sd/home’, etc are not devices but only folders; this is the reason why kernel reports error: “Begin: Mounting /usr file system … mount: No such file or directory”.

Based on your provided information, I think it’s not related with the rootFS. It’s more likely related with booting images. After you have reflashed boot.img and rootfs image, you should can recovery the rootFS. Could you confirm after reflashing rootfs, does the output log have the same error with your first post?

For booting images recovery, please refer the page [1] for detailed steps.

[1] Board Recovery for HiKey960 - 96Boards

#4

Thank you @leo-yan!

I don’t think this is the right method, since /sd/usr, /sd/home, etc are not devices but only folders;

The same approach works on another HiKey960 of mine, although admittedly the system is installed differently there (everything’s on the SD card). I think (but not 100% sure) this also worked fine for this HiKey960, until I decided to “optimize” space and removed /usr from the internal flash as follows:

$ sudo rsync -aH /usr /sd
$ sudo touch /usr/OLD
$ sudo touch /sd/usr/NEW
$ sudo mount --bind / /mnt
$ ls /mnt/usr/OLD
$ sudo rm -rf /mnt/usr
$ sudo umount /mnt

I will try [1] as you suggested, and report here. So far I have only tried [2].

[1] Board Recovery for HiKey960 - 96Boards
[2] https://github.com/96boards-hikey/tools-images-hikey960

#5

I see what you’re doing with the bind mounts (@leo-yan : he’s bind mounting the directory /sd/usr → /usr after mounting the second partition of the sdcard to /sd/)

I think that the problem is that you deleted the mountpoint /usr rather than just its contents.

1 Like
#6

I must say that this part of the instructions is really puzzling:

The bootloader has now been installed. Now move to the Hikey960 console and press f during UEFI boot. This will allow the board to boot into fastboot mode.

What’s a “HiKey960 console”? How do I “move” there? How do I supposed to “press” f if I have no keyboard attached to the board?

(Luckily, I already know how to enter into the fastboot mode with dip switch settings.)

#7

Yes, so foolish of me. But I would have thought that reflashing the firmware, and then boot and rootfs should fix it. Alas, I’m still getting the same trace even after all these steps.

Is there perhaps a way to connect the board as a flash drive to my computer and create /usr without going through the booting process?

#8

By the way, Step 3 of [1] leads to [2]. Is [2] supposed to be equivalent to [3]? I have tried both to no avail.

[1] Linux Host Installation for HiKey960 - 96Boards
[2] https://github.com/96boards-hikey/tools-images-hikey960
[3] Board Recovery for HiKey960 - 96Boards

#9

Sorry, I think here introduced much confusion.

Firstly, [2] points to the legacy booting images which are delivered by Hisilicon propertied images. I checked again in the official AOSP releasing, so far we still use the legacy booting images.

[3] is equivalent to [2], but [3] uses the open sourced booting images (ARM-TF mainline code + UEFI).

Could you confirm one thing for ‘I have tried both to no avail’? Here your target is to recovery board or you have recoveried board successfully, but failed to move some folders into SD card?

If you want to enlarge the space for rootFS and get more sufficient disk size for your work, I suggest you to use the ‘data’ partition to flash the debian rootFS, and in the kernel command line you can specify to mount the ‘data’ partition as the root device. I am not sure this info is useful, maybe it’s irrelevant.

#10

If [3] is preferred over [2], I’d suggest to update Step 3 of [1] to link to [3] instead of [2]. (Only clarify “Now move to the Hikey960 console and press f during UEFI boot.”)

Could you confirm one thing for ‘I have tried both to no avail’? Here your target is to recovery board or you have recoveried board successfully, but failed to move some folders into SD card?

I’ve performed steps in [2] and [3], but this hasn’t helped me to solve the kernel panic issue. (Therefore, I think the board’s firmware was in a good state even before I performed [2] and [3].) To confirm, I see (via USB serial) the GRUB menu, but no matter whether I accept the default or select the recovery mode, the board ends up with a kernel panic after 5 seconds, and hangs. I thought that reflashing either boot or rootfs would help restore the deleted /usr mount point, but the issue is still present… That’s why I tried to erase and even format boot and rootfs, but there’s something I’m missing…

#11

I would expect that overwriting the rootfs should fix the damaged rootfs, so my guess is that rootfs may be in a different location than you are thinking.

Since you have access to grub, why don’t you set up an sdcard with all relevant partitions and boot that? Then you will be able to mount the damaged rootfs and repair it.

#12

I like this idea, but I’m not that experienced with grub, so not sure how to do it. Any links?

#13

I could give you any number of links to grub syntax. You really should hit google for this one, there is no shortage of resources for this.

For example; GRUB 2 bootloader - Full tutorial

1 Like
#14

From the log, you could see the root file system is using /dev/sdd13.

If we review the the ptable [1], the command ‘sudo fastboot flash system rootfs-linaro-stretch-developer-hikey-20190620-32.img’ will flash rootFS into /dev/sdd10, but actually you are using /dev/sdd13 to mount root fs.

[1] https://github.com/96boards-hikey/l-loader/blob/testing/hikey960_v1.2/generate_ptable.sh#L141

#15

@doitright Much appreciated. I probably misinterpreted your advice, as I somehow thought that I would be able to mount /usr from the SD card during the boot time (where I had a copy). Hence my difficulty in finding specific advice on how to edit the GRUB boot command with respect to mounting.

I believe my other board runs entirely from the SD card, so I can indeed try that.

#16

@leo-yan Good catch. I would be curious to know how I managed to arrive at this predicament. But right now I’m simply eager to make the board usable again.

I guess the easiest would be to flash rootFS onto /dev/sdd13. How can I achieve that? Would perhaps the following:

$ sudo fastboot flash userdata rootfs-linaro-stretch-developer-hikey-20190620-32.img

flash rootFS onto /dev/sdd13?

Ideally, I would like to erase everything on the board and run a sequence of recovery/flash commands to put everything in a consistent state again.

#17

The reason I mentioned userdata because I found it here [1]. However, it’s AOSP specific, while I’m trying to install Linux?

[1] https://github.com/96boards-hikey/l-loader/blob/testing/hikey960_v1.2/generate_ptable.sh#L148

#18

If you want to just start over, you could run the uefi-flash-all.sh from AOSP. That would at least reset everything to a clean slate where you can begin again.

#19

If you want to just start over, you could run the uefi-flash-all.sh from AOSP.

Thanks @doitright. I assume that installing Linux immediately after that should work?

I can see flash-all.sh mentioned in Step 5 of [1]. Is it the same? Unfortunately, I cannot find it in [2]?

[1] Linux Host Installation for HiKey960 - 96Boards
[2] https://snapshots.linaro.org/96boards/hikey960/linaro/aosp-master/latest/

#20

I don’t know anything about how the snapshots are organized. The uefi version installs open source bootloaders instead of the blobs.

https://android.googlesource.com/device/linaro/hikey/+/refs/heads/master/installer/hikey960/