Hikey960 soft brick?

manu · February 19, 2018, 12:40am

Hi all,

I already worked for quite a while with the Hikey960 board and mainly ran OP-TEE on it.
Today, I changed the OP-TEE core code in a way that led to a kernel panic during the OP-TEE boot process.

The problem was that I was not able to reflash a correct OP-TEE image on the board. Even after reflashing the hikey firmware (https://github.com/96boards-hikey/tools-images-hikey960 the board always booted the faulty OP-TEE image.

My guess was that the old OP-TEE image was not properly overwritten during the flash process so I tried to use ‘fastboot erase’ on the cache, userdata and system partitions of the board before flashing the hikey firmware.

This was probably not the best idea because now every time I try to run OP-TEE I am stuck at the second level bootloader.
I can still successfully flash the hikey firmware and also during the OP-TEE flash process everything seems fine.

Does anyone have an idea what I might have broken by using ‘fastboot erase’ and how I can fix it?

Best regards,
Manu

Flashing Hikey Fimware Log

./recovery-flash.sh
Config name: config
Port name: /dev/ttyUSB1
0: Image: ./sec_usb_xloader.img Downalod Address: 0x20000
1: Image: ./sec_uce_boot.img Downalod Address: 0x6a908000
2: Image: ./sec_fastboot.img Downalod Address: 0x1ac00000
Serial port open successfully!
Start downloading ./sec_usb_xloader.img@0x20000…
file total size 99584
downlaod address 0x20000
Finish downloading
Start downloading ./sec_uce_boot.img@0x6a908000…
file total size 23680
downlaod address 0x6a908000
Finish downloading
Start downloading ./sec_fastboot.img@0x1ac00000…
file total size 3430400
downlaod address 0x1ac00000
Finish downloading
< waiting for device >
target reported max download size of 471859200 bytes
sending ‘ptable’ (196 KB)…
OKAY [ 0.018s]
writing ‘ptable’…
OKAY [ 0.039s]
finished. total time: 0.057s
target reported max download size of 471859200 bytes
sending ‘xloader’ (151 KB)…
OKAY [ 0.015s]
writing ‘xloader’…
OKAY [ 0.214s]
finished. total time: 0.229s
target reported max download size of 471859200 bytes
sending ‘fastboot’ (3346 KB)…
OKAY [ 0.183s]
writing ‘fastboot’…
OKAY [ 0.036s]
finished. total time: 0.219s
target reported max download size of 471859200 bytes
sending ‘nvme’ (128 KB)…
OKAY [ 0.014s]
writing ‘nvme’…
OKAY [ 0.032s]
finished. total time: 0.046s
target reported max download size of 471859200 bytes
sending ‘fw_lpm3’ (212 KB)…
OKAY [ 0.018s]
writing ‘fw_lpm3’…
OKAY [ 0.014s]
finished. total time: 0.032s
target reported max download size of 471859200 bytes
sending ‘trustfirmware’ (145 KB)…
OKAY [ 0.015s]
writing ‘trustfirmware’…
OKAY [ 0.014s]
finished. total time: 0.029s

Flashing OP-TEE Log

fastboot flash ptable /home/optee_test/build/…/l-loader/prm_ptable.img
target reported max download size of 471859200 bytes
sending ‘ptable’ (24 KB)…
OKAY [ 0.008s]
writing ‘ptable’…
OKAY [ 0.029s]
finished. total time: 0.037s
fastboot flash xloader /home/optee_test/build/…/tools-images-hikey960/sec_xloader.img
target reported max download size of 471859200 bytes
sending ‘xloader’ (151 KB)…
OKAY [ 0.015s]
writing ‘xloader’…
OKAY [ 0.238s]
finished. total time: 0.253s
fastboot flash fastboot /home/optee_test/build/…/l-loader/l-loader.bin
target reported max download size of 471859200 bytes
sending ‘fastboot’ (1152 KB)…
OKAY [ 0.067s]
writing ‘fastboot’…
OKAY [ 0.025s]
finished. total time: 0.093s
fastboot flash fip /home/optee_test/build/…/arm-trusted-firmware/build/hikey960/release/fip.bin
target reported max download size of 471859200 bytes
sending ‘fip’ (1500 KB)…
OKAY [ 0.086s]
writing ‘fip’…
OKAY [ 0.032s]
finished. total time: 0.118s
fastboot flash nvme /home/optee_test/build/…/tools-images-hikey960/nvme.img
target reported max download size of 471859200 bytes
sending ‘nvme’ (128 KB)…
OKAY [ 0.014s]
writing ‘nvme’…
OKAY [ 0.052s]
finished. total time: 0.065s
fastboot flash boot /home/optee_test/build/…/out/boot-fat.uefi.img
target reported max download size of 471859200 bytes
sending ‘boot’ (65536 KB)…
OKAY [ 3.433s]
writing ‘boot’…
OKAY [ 0.480s]
finished. total time: 3.913s

Boot process OP-TEE Log

xloader chipid is: 0x36600110, start at 485ms.
Build Date: Sep 19 2017, 15:34:09
[clock_init] ++
hikey960 [hikey960_clk_init]
hi3660 [clk_setup]
[clock_init] –
storage type is UFS
ufs retry: 6 count v_tx:0 v_rx:0
ufs set v_tx:0 v_rx:0
Hikey960[5301] no need avs_init.
ddr ft:0xf20332a3,mode:1 target:4
UceLdOk
ch 0 gt_errfail, STATUS:0x00000060
ch 0 gdst_errfail, STATUS:0x00000040
ch 1 gt_errfail, STATUS:0x00000060
ch 1 gdst_errfail, STATUS:0x00000040
ch 2 gt_errfail, STATUS:0x00000060
ch 2 gdst_errfail, STATUS:0x00000040
ch 3 gt_errfail, STATUS:0x00000060
ch 3 gdst_errfail, STATUS:0x00000040
timeout
timeout
timeout
timeout
density: 0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000,0x0c0c0c0c,0x00000000
ddr info 0x00000306
400M
685M
1067M
C0R,V0x0000002c e:66
C1R,V0x0000002d e:66
C2R,V0x0000002c e:66
C3R,V0x0000002d e:66
C0R,V0x0000002d e:66
C1R,V0x0000002e e:66
C2R,V0x0000002d e:66
C3R,V0x0000002e e:66
C0R,V0x0000002e e:66
C1R,V0x0000002f e:66
C2R,V0x0000002e e:66
C3R,V0x0000002f e:65
C0R,V0x0000002f e:65
C1R,V0x00000030 e:65
C2R,V0x0000002f e:65
C3R,V0x00000030 e:65
1244M
1866M
C0R,V0x00000016 e:66
C0R,V0x00000017 e:66
C2R,V0x00000017 e:66
C3R,V0x00000017 e:66
iomcu_subsys_init
boot_c0 PROFILE 4
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v1.4(release):v1.4-666-g1504715
NOTICE: BL1: Built : 23:44:05, Feb 18 2018
ERROR: Failed to load BL2 firmware.

helg · February 19, 2018, 10:33am

bl2.bin is part of fip.bin (together with teeos.bin), you can analyse fip.bin using ‘fiptool’

usage: fiptool [--verbose] <command> [<args>]
Global options supported:
  --verbose     Enable verbose output for all commands.

Commands supported:
  info          List images contained in FIP.
  create        Create a new FIP with the given images.
  update        Update an existing FIP with the given images.
  unpack        Unpack images from FIP.
  remove        Remove images from FIP.
  version       Show fiptool version.
  help          Show help for given command.

The unpack operation should create these 5 images

soc-fw.bin <- bl31.bin
scp-fw.bin <- fw_lpm3.img=uce+ocbc+lpm3
nt-fw.bin  <- BL33_AP_UEFI.fd
tb-fw.bin  <- bl2.bin
tos-fw.bin <- tee_os.bin

I do not know why BL33_AP_UEFI.fd is included both in l-loader.bin and fip.bin, maybe someone more knowledgeable than me can provide a better explanation.

manu · February 19, 2018, 12:08pm

I seems bl2.bin is included in my fip.bin

Running fiptool update fip.bin creates:

soc-fw.bin
scp-fw.bin
nt-fw.bin
tb-fw.bin
tos-fw.bin
tos-fw-extra1.bin
tos-fw-extra2.bin

helg · February 19, 2018, 12:43pm

Does it have reasonable size ?
Here is the info output for the latest ‘official’ release (55):

$ fiptool info /scratch/HIKEY/EFI_55/uefi/fip.bin
Trusted Boot Firmware BL2: offset=0x200, size=0x4470, cmdline="--tb-fw"
SCP Firmware SCP_BL2: offset=0x4800, size=0x35100, cmdline="--scp-fw"
EL3 Runtime Firmware BL31: offset=0x39A00, size=0x8010, cmdline="--soc-fw"
Secure Payload BL32 (Trusted OS): offset=0x41C00, size=0x411F0, cmdline="--tos-fw"
Non-Trusted Firmware BL33: offset=0x82E00, size=0xF0000, cmdline="--nt-fw

As the last option you can disassemble the tb-fw.bin …

manu · February 19, 2018, 2:03pm

I get the following output fir fiptool info fip.bin

Trusted Boot Firmware BL2: offset=0x200, size=0x4470, cmdline="--tb-fw"
SCP Firmware SCP_BL2: offset=0x4800, size=0x35100, cmdline="--scp-fw"
EL3 Runtime Firmware BL31: offset=0x39A00, size=0x8010, cmdline="--soc-fw"
Secure Payload BL32 (Trusted OS): offset=0x41C00, size=0x1C, cmdline="--tos-fw"
Secure Payload BL32 Extra1 (Trusted OS Extra1): offset=0x41E00, size=0x45270, cmdline="--tos-fw-extra1"
Secure Payload BL32 Extra2 (Trusted OS Extra2): offset=0x87200, size=0x0, cmdline="--tos-fw-extra2"
Non-Trusted Firmware BL33: offset=0x87200, size=0xF0000, cmdline="--nt-fw"

However, I think that the problem is not caused by a wrong OP-TEE or ARM-TF image since I can’t boot anything on the board anymore.

Earlier, I was also able to boot the zircon kernel on the board. Now, also zircon gets stuck during the boot process.

load_kernel: boot_from_bl31: boot to trusted firmware. addr=0x00000000 

ZIRCON KERNEL PANIC

So it looks like a corrupted some essential parts with my ‘fastboot erase’ commands.

vchong · February 19, 2018, 4:15pm

@manu

recovery-flash.sh is an older script so suggest avoid using it. Instead, try running make recovery from /home/optee_test/build and follow the instructions there instead. Atm, you’re mixing both recovery-flash.sh and images from /home/optee_test/build, which I think won’t work.

ERROR: Failed to load BL2 firmware.

In any case, if make recovery doesn’t work, and you still see the error above, it might be related this [1] bug which has been reported, but hasn’t been updated since Jan 25.

[1] https://bugs.96boards.org/show_bug.cgi?id=685

@helg

scp-fw.bin ← fw_lpm3.img=uce+ocbc+lpm3

Where did you get and fw_lpm3.img and uce+ocbc+lpm3 from? Afaik, scp-fw.bin is just https://github.com/96boards-hikey/OpenPlatformPkg/tree/testing/hikey960_v1.3.4/Platforms/Hisilicon/HiKey960/Binary/lpm3.img, the image for the System Control Processor (SCP).

tos-fw.bin ← tee_os.bin

Note that the filename for this varies and is not always tee_os.bin.

BL33_AP_UEFI.fd is the UEFI firmware (Normal World bootloader) image.

@manu @helg

The latest ‘official’ release (55) and images in /home/optee_test/build are built differently, but both @manu’s fip.bin and release 55 look ok, i.e. in the sense that they both contain BL2.

As an aside, @helg, in the log you posted, did you mean to highlight BL32 in red, or BL2?

@manu In summary, please try again using make recovery and see if that helps.

vchong · February 20, 2018, 3:23am

I do not know why BL33_AP_UEFI.fd is included both in l-loader.bin and fip.bin

BL33_AP_UEFI.fd is the UEFI firmware (Normal World bootloader) image.

BL33_AP_UEFI.fd contains the fastboot facitility so it’s ‘re-used’ in l-loader.bin to provide fastboot services in recovery mode.

manu · February 19, 2018, 4:58pm

Nice!
Using make recovery did the trick.

I am also experiencing the bug you referenced where the BL2 is “sometimes” not found. I my case this happens mostly during the first boot after flashing the board. However, I had this “problem” already when I started working with the board. In my case it is not that big of a deal.

Thank you very much for your help!

helg · February 19, 2018, 5:53pm

It is the only non-opensource image in the bootchain, so i ran ‘strings lpm3.img | grep MAGIC’ for my notes and got

UCEMAGICNUMBER!!
OCBCMAGICNUMBER!
LPM3MAGICNUMBER!

I have just pasted the text output into the ‘code’ block and it got auto-highlighted by the board software.
Sorry about that. i will try to avoid random hightlighting in the future.

helg · February 20, 2018, 3:23am

IMHO, this has a potential to really confuse a lot of people, since there are also ‘fastboot’ and ‘fw_lpm3’ labeled partitions (also for Debian) which are unused in UEFI setup.

vchong · February 20, 2018, 3:40am

It is the only non-opensource image in the bootchain, so i ran ‘strings lpm3.img | grep MAGIC’ for my notes

Got it. Thanks.

I have just pasted the text output into the ‘code’ block and it got auto-highlighted by the board software.
Sorry about that. i will try to avoid random highlighting in the future.

No worries. It wasn’t something you did but rather that the system highlighted it automatically for I don’t know what reason.

IMHO, this has a potential to really confuse a lot of people, since there are also ‘fastboot’ and ‘fw_lpm3’ labeled partitions (also for Debian) which are unused in UEFI setup.

I agree things can be confusing especially for the new comers. The reuse is more of an implementation detail that shouldn’t affect most regular users.

The fw_lpm3 partition is legacy that shouldn’t be used anymore in current builds.

What do you mean exactly by UEFI setup? I don’t believe there’s a case where the fastboot partition is not used. As an aside, the label name does in a way tie back to the reason for the reuse.

helg · February 21, 2018, 9:28pm

It is still included in the LUN 3 partition table distributed with the UEFI images. I have just learned today that the “big” RPB image does not fit into available “system” partition anymore. It is probably a good idea to have separate partition table images for UEFI android and debian/rpb setups.

vchong · February 22, 2018, 4:18pm

I’m not familiar with LUN 3. Afaik the partition table images for android and debian/rpb setups are different [1]. Perhaps it’s worth a try to increase the size of “system” and rebuild the image, but I’ve a faint recollection of an unsuccessful attempt at it by someone else.

[1] https://github.com/96boards-hikey/l-loader/blob/testing/hikey960_v1.2/generate_ptable.sh#L119-L186

helg · February 27, 2018, 8:05pm

LUN3 is the fourth SCSI logical unit (LUN0=/dev/sda=xloader,…,LUN3=/dev/sdd) on UFS. The UEFI source code also refers to it in this way.