Stuck in Initramfs

Products Support DragonBoard410c
#21

I’m ignoring your earlier question about how to build buildroot since it looks like you’ve got it running. Very glad to hear this. buildroot can bring in lots of useful diagnostics (making it easier to run the stress testing @anon91830841 recommended).

The file that is not found is probably fsck.ext4 rather than /dev/mmcblk0p10 . You need to enable it in the buildroot config (make menuconfig → Target packages → Filesystem and flash utilities → e2fsprogs → fsck).

Basically the next step is to tour make menuconfig in buildroot turning on some of the more interesting diagnostic tools. Personally I tend to use stress-ng (in Debugging, profiling and benchmark) but there’s many different tools you can choose from.

Daniel.

#22

I could launch a few tests; I use stress and ramspeed for the memory tests.
Ramspeed and stress -c applied on the RAM gives good results.

For the emmc, it does not seems bad either. It fails with “stress -v -m 4 -t 2” but I double checked on my dev board (which works fine) and it fails also. Stress results are fine with stress -m parameters smaller than 4.
I monitored the voltages on the emmc while stressing, did not notice anything wrong.
Fyi, here are logs of buildroot console with tests such as manually mounting rootfs.

So this doesn’t tell me why my rootfs is not mounted at “normal” boot. I will give more trials on modifying the GPT; and see about this 5/4.5 emmc

#23

It’s seems pretty clear my troubles are related to my hardware/custom memories. Any chance I have to modify the cdt (“sbc_1.0_8016.bin”) with my memory informations to get a correct boot? Or should I only have a look at the GPT?

Thank you,

Victor

#24

I tried to flash Android for comparizon, using official release files from here
But I end up with this error at fastboot flash:

target reported max download size of 268435456 bytes
erasing ‘userdata’…
OKAY [ 0.350s]
sending ‘userdata’ (125938 KB)…
OKAY [ 18.367s]
writing ‘userdata’…
FAILED (remote: size too large)
finished. total time: 18.722s

Not sure I want to go through the pain of compiling new android kernel, as I do not have this error when flashing debian.

#25

userdata is supposed to be an ‘empty’ partition on first boot. you should be able to resize it. I just tried, it’s not empty, but has small content, even though the image is 5GB.

here are commands to resize the image (to 1GB):

tar xf userdata.img.tar.xz 
simg2img userdata.img userdata.img.raw
e2fsck -f userdata.img.raw 
resize2fs userdata.img.raw 1G
img2simg userdata.img.raw userdata-resized.img

You should be able to flash it after that.

#26

Thank you @anon91830841, that worked.

Android boot fails with following logs. There are some stranges numbers sequences in the logs. After that, It repeats constantly with:

[ 18.641896] msm_dba_get_probed_device: Device not found (adv7533, 0)
[ 18.647202] msm_dba_register_client: Device not found (adv7533, 0)
[ 18.653378] mdss_dba_utils_init: ds not configured

then reboots and start again…
This makes sense as I have no HDMI on my custom board.
I guess I have to modify the config and rebuild the android kernel… but I would rather use linux directly

In my custom Linux, I had modified the dtbs file by removing everything related to adv in the dtsi file. My kernel boots on a custom board with a SOM that has no HDMI, but does not boot on my custom board. Maybe there is something more I should do about that?

#27

In buildroot, I have this result with fdisk -l:

# fdisk -l
Found valid GPT with protective MBR; using GPT

Disk /dev/mmcblk0: 7634944 sectors, 3728M
Logical sector size: 512
Disk identifier (GUID): 98101b32-bbe2-4bf2-a06e-2bb33d000c20
Partition table holds up to 12 entries
First usable sector is 34, last usable sector is 7634910

Number  Start (sector)    End (sector)  Size Name
     1          131072          131075  2048 cdt
     2          262144          263167  512K sbl1
     3          263168          264191  512K rpm
     4          264192          266239 1024K tz
     5          266240          267263  512K hyp
     6          267264          267295 16384 sec
     7          267296          269343 1024K aboot
     8          269344          400415 64.0M boot
     9          400416          402463 1024K devinfo
    10          402464         7634910 3531M rootfs
Disk /dev/mmcblk0boot1: 2 MB, 2097152 bytes, 4096 sectors
64 cylinders, 4 heads, 16 sectors/track
Units: cylinders of 64 * 512 = 32768 bytes

Disk /dev/mmcblk0boot1 doesn't contain a valid partition table
Disk /dev/mmcblk0boot0: 2 MB, 2097152 bytes, 4096 sectors
64 cylinders, 4 heads, 16 sectors/track
Units: cylinders of 64 * 512 = 32768 bytes

Disk /dev/mmcblk0boot0 doesn't contain a valid partition table

not sure if this Disk /dev/mmcblk0boot0 doesn’t contain a valid partition table error could be a hint for me?
Otherwise my last hope is to find an hardware mistake now…

#28

From Buildroot busybox, I can manually mount and access the rootfs with # mount -t ext4 /dev/mmcblk0p10 /root
Any idea how I can manually switch the root to this rootfs instead of the busybox?

#29

Make the ramdisk invalid (the empty file should work) and the kernel will start honouring the root= argument again. Without a ramdisk you must use specify the rootfs explicitly (/dev/mmcblk0p10) rather than by name (/dev/disk/by-partlabel/boot).

#30

Thank you Daniel;
Without ramdisk the error I have is more repetable (I still have this weird fault: /lib/systemd/system-generators/systemd-gpt-auto-generator terminated by signal SEGV. )
Logs look like this and I still cannot have a full correct boot.
Will have a deeper look at my hardware today.

#31

I tried a new emmc with 8GB (MTFC8GAKAJCN-1M-WT) but the problem is still there.
One difference between my board and dragonboard is the following logs:
On my board: mmcblk0boot0: mmc0:0001 Q2J55L partition 1 2.00 MiB
On dragonboard, mmcblk0boot0 and boot1 are 4 MiB.

Does that matter? I am not sure about the purpose of those mmcblk0boot0/1 partitions…
Thank you,

#32

The boot partitions are separate hardware partititons within eMMC. They need a special protocol to access and are intended to make it easier to separate the bootloader from the OS.

I don’t think it matters much in since I don’t think the DB410C boot ROM uses them. However it does means that you will still need a custom partition table because your main eMMC partition will be 4MB “too big”.

Not sure what the best way to resize them is… Looks like there might be support in u-boot to allow it but I’ve never tried doing any changes to the boot partitions myself.

#33

Thank you for your anwer.
This is interesting, as I see it it could be the source of all my troubles
Will have a deeper look in the next few days, thank you again !

#34

For completeness, here are the exact differences between my board and dragonboard:
dragonboard:
mmcblk0boot0: partition 1 4.00 MiB
mmcblk0boot1: partition 2 4.00 MiB
mmcblk0rpmb: partition 3 4.00 MiB
my custom board:
mmcblk0boot0: partition 1 2.00 MiB
mmcblk0boot1: partition 2 2.00 MiB
mmcblk0rpmb: partition 3 512 KiB

So my guess is that this is the reason there are troubles during the boot.
Here are the different options I consider to solve the problem:

  • Resize the boot partitions in the emmc
    I am not even sure my specific memory allows that
    or
  • Replace the emmc with another chip that has same footprint and boot partitions sizes
    or
  • Customize the partition tables (gpt files) so that the shift in size is taken into account.

I intend to try the latest option that seems the easiest way to do it. But I am not sure if those boot partitions sizes are taken into account in the GPT or in the rawprogram.xml file … Otherwise I could have a deeper look deeper to u-boot to resize the partitions but I have no experience with it (yet).
Thank you for your help,

#35

Hello @danielt
Could you confirm that I need to change my partition table because of this difference in the emmc hardware boot partitions sizes?
I do not see how I could solve the “4MB too big” problem. As far as I know partitions are flashed at sector 0 of main emmc partition anyway and the last partition is resized to fit the remaining space.

#36

Being too big is not a huge problem in terms of accessing partitions… but the GPT will still be technically invalid because the duplicate partition table at then end of the disk won’t actually be at the end of the disk.

We’ve long assumed that the reason for the problems you observe is because your partition table is corrupt (taking as evidence that the initramfs can’t access them by name). However we’ve never really know how it is corrupt since the study you did with buildroot revealed no problems. However we can also guess from your July logs that when you workaround the first problem a second problem emerges, namely a systemd component that parses the GPT starts crashing. Thus the theory continues that somehow your partition table isn’t right.

So I can’t, in all honesty, be sure if you need to change your partition table because we never really understood why it didn’t work for you in the first place.

I wonder if going back to work in buildroot might be useful. In particular experimenting with sfdisk to create a new partition table.

Basically if you run the following then what you get is an script that sfdisk can read to restore the partition table:

sfdisk --dump /dev/mmcblk0 > parts

The partition table is restored using the following:

sfdisk /dev/mmcblk0 < parts

I would recommend removing the first-lba: and last-lba: stanzas from parts before restoring although I confess this is absolute paranoia on my side. When it is restored then the partitions themselves will be the same as before but the partition table will be rebuilt from scratch. Nothing “weird” should survive this.

#37

Hello Daniel,

Thank you for your message.
Any idea how to add sfdisk in buildroot?
I cannot find it in the config… All I have that seem close is fdisk.
I am building buildroot from https://github.com/buildroot/buildroot

Thank you,

#38

Thank you for your message.
Any idea how to add sfdisk in buildroot?
I cannot find it in the config… All I have that seem close is fdisk.
I am building buildroot from https://github.com/buildroot/buildroot

I tried git grep sfdisk (on the buildroot tree) and it looks like you
need to go into the Target packages → System tools → util-linux and
enable both util-linux and the basic set. You can check you have it
right by looking for BR2_PACKAGE_UTIL_LINUX_BINARIES in the .config
file.

PS I didn’t test…

#39

Edit: I could successfully use sfdisk. After that I reflash the rootfs but it does not seems to have solve the problem unfortunately.

#40

I arrived to a point where I can most certainly exclude a software issue. Modifying the boot partitions on the emmc didn’t help.
So I plan to deep check the harware. As far as I know, here are my options:

  • manual check (measurements on the board)
  • checks using buildroot/initramfs
  • JTAG
  • third party verification

I did not thind anything wrong using buildroot, but I can not guarantee I checked everything deeply. Does anyone have any hints on how to make deep clean tests to find any hardware errors with the apq8016?
From others thread, JTAG debug seems quite a pain and not fully suported (see JTAG support: how? - #7 by ljking). Do you think it is worth exploring this regarding my problems?

Also, do you know any third party that can help regarding design/layout verification?
@ljking suggested valydate.com in other threads but their website seems down.
Anyway, any help/suggestion regarding hardware bug finding is very welcome.
Thank you,