My HiKey960 board has recently stopped to attach to Wifi network and I cannot log in (with ssh) to a terminal.
Meantime the system seems alive.
Is there any other way to log in to console? Can I do this with USB-C?
If you were running Android, you would be able to use ADB, but since you’re talking about SSH, I imagine that you’re probably not using Android.
That means that your only options are going to be;
Network; you’ve apparently tried and failed,
Direct; i.e., keyboard and monitor,
UART; it connects to a few of the pins on the low speed header.
You’re right, board is running debian.
I forgot about keyboard and monitor, I’ll try this way, thanks!
Wondering whether usb-c is functional in debian and, if yes, is there any way to configure it somehow so that the board is providing tty over it (perhaps, I could configure it if I’ll get an access with keyboard)?
It is certainly possible to set up another form of console via the USB.
If you want to use the USB-C, then it would have to be by way of a gadget driver. It would probably be easier, however, to use a device connected to one of the USB-A ports.
Note that the Kirin 960 only has a single USB port, which is routed EITHER to the hub (and the A ports), or to the C port. They can’t all work at the same time. The Dragonboard 820C is much better in this area, since that SoC has multiple USB ports, allowing the device and host ports to all work at the same time.
I was not able to see a console with monitor (via HDMI) and used UART to figure out what happened.
It looks like the issue is caused by a stack of kernel while initializing SD card hardware on cold boot (if SD card is inserted into a slot).
My board is booting from emmc and some time ago I’ve inserted a 128GB SD card to use it as an additional storage. I’ve added it to /etc/fstab to be mounted on start.
I do not remember whether I’ve rebooted the board that time.
Few days after, after the power outage, the board did not boot. I’ve tried to remove SD card, the board seems alive (green led flashing) but I did not see it via WiFi (and its blue led blinks once on boot).
With UART I’ve found the if SD card is absent, the system enters emergency mode because the partition of SD card cannot be mounted. I’ve removed it from fstab and board booted successfully.
If I insert SD card into the slot, it can be mounted without any issue. Moreover, if I reboot the board from command line, it starts successfully too (partition is not added to fstab).
And the problem case is cold boot with SD card in the slot.
I’ve attached a minicom log of this cold start (at the end). The mmc0 is inited successfully, partition is detected but soon after that the boot stucks.
Here (for comparison) is the log of hot plugging of SD card:
[ 88.129260] mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 200000000Hz, actual 200000000HZ div = 0)
[ 88.405181] dwmmc_k3 ff37f000.dwmmc1: tuning ok best_clksmpl 15 tuning_sample_flag cf7fff7f
[ 88.413638] mmc0: new ultra high speed SDR104 SDXC card at address 59b4
[ 88.421206] mmcblk0: mmc0:59b4 U 119 GiB
[ 88.428089] mmcblk0: p1
[ 162.197498] EXT2-fs (mmcblk0p1): warning: mounting unchecked fs, running e2fsck is recommended
I’m currently using the following trick to bypass the emergency mode (SD card mount is added to fstab) for cold start: start with card is not inserted, wait for wifi led blink (the init process wait for some time for partition to be mounted), insert the card. Boot finishes successfully. Perhaps there could be a better way (maybe some option in fstab)?
The cold start log (the tail, do not know how to attach a whole log).
[ 1.735931] mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 200000000Hz, actual 200000000HZ div = 0)
[ 1.736552] console [ttyAMA6] enabled
[ 2.092861] dwmmc_k3 ff37f000.dwmmc1: tuning ok best_clksmpl 15 tuning_sample_flag cf7fff7f
[ 2.095750] ssp-pl022 ffd68000.spi: ARM PL022 driver, device ID: 0x00041022
[ 2.101191] mmc0: new ultra high speed SDR104 SDXC card at address 59b4
[ 2.105361] ssp-pl022 ffd68000.spi: mapped registers from 0x00000000ffd68000 to (ptrval)
[ 2.112301] mmcblk0: mmc0:59b4 USDU1 119 GiB
[ 2.118153] ssp-pl022 ffd68000.spi: setup for DMA on RX dma0chan0, TX dma0chan1
[ 2.841647] random: fast init done
[ 2.842613] mmcblk0: p1
[ 2.842798] ssp-pl022 ff3b3000.spi: ARM PL022 driver, device ID: 0x00041022
[ 2.842963] ssp-pl022 ff3b3000.spi: mapped registers from 0x00000000ff3b3000 to (ptrval)
[ 2.843018] ssp-pl022 ff3b3000.spi: setup for DMA on RX dma0chan2, TX dma0chan3
[ 2.852720] hisi_hikey_usb hisi_hikey_usb: extcon_hisi_pd_set_role:set usb role to 0
[ 3.591652] usb_switch_ctrl: switch to hub
[ 3.595792] usb_typec_power_ctrl: set typec vbus gpio to 0
[ 3.604513] dwmmc_k3 ff3ff000.dwmmc2: fifo-depth property not found, using value of FIFOTH register as default
[ 3.615132] hisi_hikey_usb hisi_hikey_usb: extcon_hisi_pd_set_role:set usb role to 0
[ 3.616739] dwmmc_k3 ff3ff000.dwmmc2: IDMAC supports 64-bit address mode.
[ 3.622921] usb_switch_ctrl: already switch to hub
[ 3.629760] dwmmc_k3 ff3ff000.dwmmc2: Using internal DMA controller.
[ 3.634493] usb_typec_power_ctrl: typec power no change
[ 3.646105] dwmmc_k3 ff3ff000.dwmmc2: Version ID is 270a
[ 3.651457] dwmmc_k3 ff3ff000.dwmmc2: DW MMC controller at irq 60,32 bit host data width,128 deep fifo
[ 3.660866] dwmmc_k3 ff3ff000.dwmmc2: Linked as a consumer to regulator.6
[ 3.667717] mmc_host mmc1: card is non-removable.
[ 3.746448] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[ 3.765537] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[ 3.789589] input: keys as /devices/platform/keys/input/input0
[ 3.797503] rtc-efi rtc-efi: setting system clock to 1970-01-01 00:00:19 UTC (19)
[ 3.802443] dwmmc_k3 ff3ff000.dwmmc2: card claims to support voltages below defined range
[ 3.805801] ALSA device list:
[ 3.816231] No soundcards found.
[ 3.819919] uart-pl011 fff32000.serial: no DMA platform data
[ 3.820433] hub 2-1:1.0: USB hub found
[ 3.824985] mmc_host mmc1: Bus speed (slot 0) = 25000000Hz (slot req 25000000Hz, actual 25000000HZ div = 0)
[ 3.826079] Freeing unused kernel memory: 1344K
[ 3.826492] mmc1: new SDIO card at address 0001
[ 3.829411] hub 2-1:1.0: 5 ports detected
[ 3.839209] Run /init as init process
Loading, please wait…
starting version 232
[ 3.880901] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[ 3.881011] random: udevadm: uninitialized urandom read (16 bytes read)
[ 3.888427] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[ 3.917709] usb 1-1: new high-speed USB device number 2 using xhci-hcd
It recommends running e2fsck on the sdcard partition. I’d start with that. Power failures can mess up filesystems.
No, that’s definitely have no relation to the issue. I’ve formatted another SD card (16GB) with the same ext2 partition, inserted it after the boot is complete, verified dmesg does not claim for errors on mount. Then I’ve powered down the board from command line, detached power, re-attached it and wait for boot. The board hangs on (couple seconds after the first green led turned on). I wait for couple minutes then detached power.
BTW, when I’ve restarted the board (with detached SD card), it was too warm. Dmesg shows warning something like “[ 41.548250] hisi_thermal fff30000.tsensor: THERMAL ALARM: 65370 > 65000”
So, finally, the issue looks like hangup on cold boot from emmc with SD card attached.
BTW2, option ‘nofail’ in /etc/fstab allows to bypass emergency mode if there is no SD card (more precisely, partition with specific guid in my case cannot be mounted).
ext2? Any particular reason you chose that outdated/ancient version?
Fwiw: these soc’s do run quite hot. Hitting the thermal alarm is nothing to be worried about.
It’s also worth noting that there is no issue like this in Android. Well at least not with the card formatted as ext4.
From what I gather, it sounds like the issue is exclusively tied to being in the fstab. If all else fails, a possible option for you may be to mount it from your rc.local.
That’s just one recommendation from internet for request “best filsystem for sd card”. Initially this SD card (128GB) was formatted with exfat, but it is supported via userspace mounts on linux. I was not sure it is a reliable solution for mounts on boot. So, I’ve asked that question… There are some filesystems dedicated for flash memory, but these might be not needed since modern SD cards has their own controller that optimizes writes. At the same time, journalling filesystems could add some overhead for just simple operations (this seems could be partially fixed with ‘noatime’ mount option) - and this causes significant DWPD (Drive Writes Per Day) - well, frankly, that is discussed in relation to SSD. That’s why someone just use ext2. I gave this option a chance…
I’ve noted this to pay some attention that something goes not good when board hangs on.
The difference could be a) kernel version or set of features/drivers inited on boot; b) some timings.
Actually, I’m suspecting some kind of momentary power loss since there are also logs for wifi init.
I would try to investigate it further, but maybe need suggestions. For example, is there any watchdog? Can it be activated so that I could take a last_kmsg on next boot? Anything else I can do to dig into this?
Well. I’m treating it opposite. Another SD card has a different UUID so it should not be attempted to mount (fstab record uses UUID instead of /dev/mmcblk0p1)
I’ll do couple more tests to clarify this: 1) with ext4 partition, 2) with no filesystem in partition, 3) with no patritions/partition table at all.
Here are the results of additional tests: in all the cases the board stuck at boot.
I’ve used GParted to format the partition to ext4. GParted was also used to clear partition (my desktop shows this partition as not formatted). And finally I’ve used dd to write all zeroes to the whole drive.
I’ve commented out the line for my 128GB sd card in fstab (16GB sd card was used for these tests).
BTW, I’ve noticed that warm boot (sudo reboot) begins to stuck too. So, perhaps successful boot is just a matter of chance.
Ok, given the result of the testing, you should probably try different kernel versions, or different ages of kernel of the same major version. If you can find a kernel that works, then you can bisect.
If you want to run without a journal, you can use “mkfs.ext4 -O ^has_journal”, however, from a resiliency perspective (data protection against unexpected events), the journal is definitely a good idea.