Stuck in Initramfs

I blamed that on a logging error ;-). It looked like there is an interruption to the logs at this point and in the other logs (which have similar symptoms) the partitions detect OK.

@Vix: Any clue what’s happening with the missing bits of the first log?

Hello, thank you for your answers;
@danielt @Loic I tried to change the root to point to root=/dev/mmcblk0p10, see the command line below:
(not sure how I am supposed to know that mmcblk0p10 is the rootfs, though)

mkbootimg --kernel Image.gz+dtb
–ramdisk initrd.img
–output boot-db410c.img
–pagesize 2048
–base 0x80000000
–cmdline “root=/dev/mmcblk0p10 rw rootwait console=ttyMSM0,115200n8”

But I still couldn’t get a correct boot (full logs here):

run-init: opening console: No such file or directory
No init found. Try passing init= bootarg.
(initramfs)

You can also notice in the logs that I have no “disk” repository in /dev nor “udev” repository in initramfs…

For information, here are a few details about what I do:

sudo …/qdl prog_emmc_firehose_8916.mbn rawprogram.xml patch.xml

  • then I can access fastboot mode and I flash successfully my board with 18.01 release, then getting problems such as described earlier

@anon91830841 I couldn’t get a boot from SD yet but I am working on it.

About the missings bits in the logs: I notice sometimes the boot is completely different, maybe rebooting from the above state is causing more boot errors… The boots logs given above are those I have directly after the fastboot flash

Last information that could be interesting: in my board power come either from the USB port, or from a battery. Above logs are given when powering from USB port (that could be an issue as USB_HS_ID is then high). When I boot from battery (USB_HS_ID is low), I get the following logs. A bit different but the error is the same.

Sorry if something is not clear, I am not a linux expert and might miss something basic; hence do not hesitate, any help is welcome. Thank you very much !

Looks to me like you haven’t correctly populated your eMMC. Partition 10 is clearly there but is had the wrong partition label and it looks like its content is corrupt or missing. I also note your eMMC is half the size of the DB410C one. Are you sure you have shrunk everything down correctly?

Hello @danielt

Thank you, that is probably the problem indeed.
I am not sure I shrank everything as needed. Could you please highlight me the differents points that needs to be modified or give me some hints about how to do that exactly?

That would be very helpful to me
Thank you for your help,

I think its just a matter of correcting the partition tables (gpt*
files) to that the rootfs partition doesn’t extend past the end of the
disk.

Note that the GPT gets written to the first and last sectors of the
disk. Depending on how you are populating your eMMC you might have
to update whatever it is that writes the GPT to the last sectors.

The rootfs image automatically resizes to fill the partition but you
should check that your partition is large enough to hold the raw image
(gunzip the .img.gz, then use simg2img to convert the sparse image to a
raw image, finally check the raw image fits into whatever you have
allocated for the rootfs).

I think my disk is large enough; for example when flashing 17.06.1 developer rootfs and boot.img, the raw image for rootfs is 2G, even with 4GB emmc I should have plenty of space…

I tried several methods for flashing: QFIL, QDL, fastboot. They all work but I always fail at boot with typical boot logs as here

Note that I sometime have random log errors such as segment violation …

I am not sure about how to correct the partition tables files (gpt_backup0.bin and gpt_main0.bin, or gpt_both0.bin when flashing with fastboot). The only way I see is using db-boot-tools which can generate gpt_both0.bin from a partitions.txt file. But even when trying this way, I could’t get a proper boot… yet

Note: weirdly, I couldn’t get a boot from SD card; it gets stuck at boot:

Error code 3038 at boot_elf_loader.c Line 334

as described here

I am starting to think that it is a problem related to my hardware config.

I tried to boot from SD using U-Boot, following the instructions given here.

Even that way I couldn’t get a proper boot, it ends in kernel panic, see full logs (not always exactly the same, but always the same kind of errors)

So my bet is that something in the config makes the boot fail.
I guess I have to disable features in the boot config and test until I get a clean boot? Or is there something else I am missing?
Thank you,

Firstly I’m a little surprised you are still working with 17.06… I doubt that’s causing your immediate problem but you should plan to upgrade.

On the hardware side it does looks like the board cannot reliably access the SD card (perhaps try an older SD card without UHS-1 support) however there is no equivalent evidence that eMMC is failing for any reason other than that the contents of the eMMC are badly authored.

It is clear you’ve not got the eMMC partition table right:

[    2.242598] Alternate GPT is invalid, using primary GPT.

The message above Indicates that gpt_backup0.bin is either incorrect or has not been written at the right part of the disk.

As I mentioned before, you should use the prompt on the initramfs to explore the system to determine what is happening. The contents of directories such as /dev/disk/by-partlabel is useful to see if the kernel detected the partition labels correctly. Similarly the contents of /sys/devices/platform/soc/7824900.sdhci/mmc_host/mmc0/mmc0:0001/block/mmcblk0 is also likely to be interesting. From here (for example) you can examine mmcblk0p10/size to see if you did manage to update the size correctly. You can also experimentally mount filesystems, etc.

BTW the other useful trick to see how badly broken things are is to create an initramfs using buildroot and replace the one in the boot image. This gives you a tiny busybox rootfs that allows you to explore more easily and use more advanced tools (such as e2fsck or memory tests) to explore what does and doesn’t work on your board.

I am now working with 18.01, but in my quest for debug I also tried with previous versions.

My SD has been soldered by hand, and as it is a bit dirty, this could explain why the board cannot access it reliably. I’'ll try to improve that.

In the previous post, I used a custom GPT that was given to me (this explain the GPT error message).
A bit of summary could be usefull:

  1. Flashing with original GPT

Here, I remove the SD, and flash the emmc with fastboot, using original GPT from dragonboard410c_bootloader_emmc_linux-88, either:

  • with original kernel command line root=/dev/disk/by-partlabel/rootfs (boot logs)
  • or with modified kernel command line root=/dev/mmcblk0p10 (boot logs)
    At boot:
    → I have no /disk in /dev
    ->the size of mmcblk0p10 in /sys/devices/platform/soc/7824900.sdhci/mmc_host/mmc0/mmc0:0001/block/mmcblk0
    is 7237567, which gives 3.7GB if I’m right.
  1. Flashing with custom GPT

As I have no idea (yet) how to modify/customize the gpt_**.bin files, I gave a try to a custom GPT that I was given; it is named emmc3o5_gpt.bin and flashed with QFIL.
Boot still fails in initramfs (logs for reference)

  1. Flashing with my custom GPT

This is what I want to do but I still have to figure out how to modify the gpt_main/gpt_backup.bin … I never touched hex binary files and I have a hard time figuring out how to do that correctly !

I will keep investigate and with the hints you have given as well, hoping to figure this out ! Thank you

I had a deeper look to the GPT.

First, when flashing with QDL, the secondary GPT is written “automatically” to the last sectors, as seen in the rawprogram.xml by the line:

start_sector="NUM_DISK_SECTORS-33

So this should not be a problem with my 4GB emmc.

Then, I tried to update the addresses of the last sector address for rootfs in gpt_backup and gpt_main0.bin but this does not result in correct boot yet.

When I type blkid I got the following results:

(initramfs) blkid
/dev/mmcblk0p10: LABEL=“rootfs” UUID=“b1026c73-62e0-42fc-a94f-0a02dc1ccef4” TYPE=“ext4” PARTLABEL=“rootfs” PARTUUID=“30d9c2ab-3507-a876-eaa8-ec59e5048b20”
/dev/mmcblk0: PTUUID=“98101b32-bbe2-4bf2-a06e-2bb33d000c20” PTTYPE=“gpt”
/dev/mmcblk0p1: PARTLABEL=“cdt” PARTUUID=“4cdc17cb-23fd-55ef-af7e-44760994d255”
/dev/mmcblk0p2: PARTLABEL=“sbl1” PARTUUID=“39840824-d4da-2c14-5776-7f3d7088ba88”
/dev/mmcblk0p3: PARTLABEL=“rpm” PARTUUID=“6a563696-ad51-02ea-ce78-bbb555315750”
/dev/mmcblk0p4: PARTLABEL=“tz” PARTUUID=“80cca449-efd0-9190-e451-2a815991aa35”
/dev/mmcblk0p5: PARTLABEL=“hyp” PARTUUID=“c048e013-801a-a8cf-f6fe-dc679e035355”
/dev/mmcblk0p6: PARTLABEL=“sec” PARTUUID=“b339503b-ebf9-7913-5291-0daf5ab3f87b”
/dev/mmcblk0p7: PARTLABEL=“aboot” PARTUUID=“5db473d0-3cce-598b-d028-c947638fb5f7”
/dev/mmcblk0p8: PARTLABEL=“boot” PARTUUID=“10019f1b-9b0f-5bb2-1a5a-e47af678998a”
/dev/mmcblk0p9: PARTLABEL=“devinfo” PARTUUID=“e97897fc-ba7d-2902-b148-7c90c646bdf3”
(initramfs)

The partuuid for rootfs is correct (same as in gpt file).

Not sure if this can help, but I also have this error at boot which seems strange, as I have not SD device plugged in:

Assertion ‘path_startswith(device->syspath, “/sys/”)’ failed at …/src/libsystemd/sd-device/sd-device.c:681, function sd_device_get_sys path(). Aborting.

I still do not understand why the symlink are not generated (no /disk in /dev); from @Loic previous post it can be either bad GPT or udev rules. How can I check those udev rules are correct or wrong?

If you are using the initramfs from DB410C then you can be pretty confident the udev rules are correct.

Also note that the “sd” in “sd-device” stands for systemd and is not necessarily anything to do with an SD ard.

Thanks for the help.

When comparing logs between my custom board and a dev board, my logs start to fail after the error mentionned above:

Assertion ‘path_startswith(device->syspath, “/sys/”)’ failed at …/src/libsystemd/sd-device/sd-device.c:681, function sd_device_get_syspath( ). Aborting.
Aborted

Any idea what this error message is about? I have only found this bug report…
Is seems linked to “systemd-gpt-auto-generator”

I am giving it a try; From what I understood I need to generate rootfs.cpio with buildroot, use it as ramdisk when generating new boot.img that I can flash with fastboot and enjoy the new tools in initramfs to check my emmc, etc…

Is that correct?
What do you suggest to test the emmc?

edit: OK I could install buildroot, I can now make more tests

Using buildroot console, I tried fsck on /dev/mmcblk0p10 but got an error

fsck -t ext4 /dev/mmcblk0p10
fsck from util-linux 2.32
fsck: error 2 (No such file or directory) while executing fsck.ext4 for /dev/mmcblk0p10

I can use mmc tool to inspect my emmc but this doesn’t help my boot.

May I ask how do you see that the partition label is wrong ?

I could also “manually” mount the rootfs and then access it. But no correct full boot yet.

mount -t ext4 /dev/mmcblk0p10 /root
[ 238.639330] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts: (null)

Also, for the record, I tried to boot without initrd (empty file), and with the command line modified to root=/dev/mmcblk0p10. Then I have more repeatable boot error: systemd-gpt-auto-generator error, see logs.

Lastly, could my troubles be due to the fact I use emmc5.0 (and not 4.5?). But I do not think so…

hi,

the gpt_both.bin file is generated from

https://git.linaro.org/landing-teams/working/qualcomm/db-boot-tools.git/

See:

https://github.com/96boards/documentation/blob/master/consumer/dragonboard410c/guides/customize-emmc-partition.md

gpt_both.bin is used when flashing the GPT with ‘fastboot flash partition’. When you are using QDL, you are using gpt_main.bin and gpt_backup.bin, and these files are released as binaries, and you can’t rebuild them yourself. So if you want to generate your own GPT, you have to use gpt_both.bin and fastboot to flash it.

The missing links in /dev could be related to this error in the boot:

[ 2.277884] systemd-udevd[1511]: unhandled level 2 translation fault (11) at 0xaaaaec000000, esr 0x92000006, in libc-2.26.so[ffff85483000+141000]

Things look really odd in that thread… I am tempting to say that the root cause is something different… like under powered CPU/board… Have you ever found a build that worked well (even a QCOM Android build)?

In any case, the best thing to do right now, is to boot into a ramdisk and stress test the system and the eMMC from the initrd (create/modify GPT on eMMC, read/write stress tests, … ).

oh… and just now, i notice your last sentence about emmc5.0… i don’t think it has been tested yet… see eMMC 4.5 vs eMMC 5.0

I’m ignoring your earlier question about how to build buildroot since it looks like you’ve got it running. Very glad to hear this. buildroot can bring in lots of useful diagnostics (making it easier to run the stress testing @anon91830841 recommended).

The file that is not found is probably fsck.ext4 rather than /dev/mmcblk0p10 . You need to enable it in the buildroot config (make menuconfigTarget packagesFilesystem and flash utilitiese2fsprogsfsck).

Basically the next step is to tour make menuconfig in buildroot turning on some of the more interesting diagnostic tools. Personally I tend to use stress-ng (in Debugging, profiling and benchmark) but there’s many different tools you can choose from.

Daniel.

I could launch a few tests; I use stress and ramspeed for the memory tests.
Ramspeed and stress -c applied on the RAM gives good results.

For the emmc, it does not seems bad either. It fails with “stress -v -m 4 -t 2” but I double checked on my dev board (which works fine) and it fails also. Stress results are fine with stress -m parameters smaller than 4.
I monitored the voltages on the emmc while stressing, did not notice anything wrong.
Fyi, here are logs of buildroot console with tests such as manually mounting rootfs.

So this doesn’t tell me why my rootfs is not mounted at “normal” boot. I will give more trials on modifying the GPT; and see about this 5/4.5 emmc

It’s seems pretty clear my troubles are related to my hardware/custom memories. Any chance I have to modify the cdt (“sbc_1.0_8016.bin”) with my memory informations to get a correct boot? Or should I only have a look at the GPT?

Thank you,

Victor

I tried to flash Android for comparizon, using official release files from here
But I end up with this error at fastboot flash:

target reported max download size of 268435456 bytes
erasing ‘userdata’…
OKAY [ 0.350s]
sending ‘userdata’ (125938 KB)…
OKAY [ 18.367s]
writing ‘userdata’…
FAILED (remote: size too large)
finished. total time: 18.722s

Not sure I want to go through the pain of compiling new android kernel, as I do not have this error when flashing debian.