Enable wcn36xx/wifi in FAI built image

Products Support DragonBoard410c
#1

I’ve begun experimenting with using FAI to build a Debian distro for the DB410c.

The build has been successful, however, when comparing the official Linaro Buster release with the one output by FAI, there is a key difference which I cannot figure out.

The official release has wifi enabled by default. The kernel modules wcn36xx, qcom_wcnss_pil, and qcom_common seem to be the culprits.

On my image, using:

modprobe wcn36xx

works to get the wcn36xx module up but the other two modules don’t seem to exist and the wifi card does not show up when I run:

ip a

The FAI version I generated is basically stock (added a few packages to be installed).

What am I missing between the official release and the one i’ve generated?

#2

hi,

can you please share the commands you’ve used?

For reference, the “official” Linaro images are built from these Jenkins jobs/scripts:

https://git.linaro.org/ci/job/configs.git/tree/lt-qcom-debian-images-dragonboard410c.yaml
https://git.linaro.org/ci/job/configs.git/tree/lt-qcom-debian-images/builders-fai.sh

Do you have a full log of your FAI run?

When you mention the official image, which image is that (the last release or the latest build)?

From our most recent build, I can see that this is how we called FAI:

**00:13:47.931** + sudo fai-diskimage -v --cspace /home/buildslave/workspace/lt-qcom-debian-images-dragonboard410c --hostname linaro-alip -S 3G --class SAVECACHE,BUSTER,DEBIAN,LINARO,QCOM,ALIP,DB410C,RAW builddir/linaro-buster-alip-dragonboard-410c-483.img.raw

How to Generate the linaro-***.img.gz?
#3

hi @anon91830841,

Thanks for getting back to me so quickly.
Attempting to generate the latest release, Debian Buster.

Our linaro FAI git repo is on commit: 7f00b15048b5752d6afcd6f8b0beacb744de22e7

This is the command we’re running:

fai-diskimage -v --cspace /home/ubuntu/fai-combined --hostname aurora-debian -S 1800M --class SAVECACHE,BUSTER,DEBIAN,LINARO,QCOM,DB410C,RAW,AURORA aurora-debian-deploy.raw

We’re running headless thus no ALIP

The logs were too long to post in here so i’ve put them here.

Please let me know if there is any other info which may help debug this.

I’m going to try to run the stock build command and see if I get the same problem.

#4

I can see a few errors in the fai.log, but nothing that would explain what you are seeing.

Maybe it would be good to build the same image that we build and check if they work for you, just to check your setup only.

One thing which is important as well is the boot image (and DTS) that you are using, this is not done by FAI, so that could be a source of the problems.

If you have a full boot log (UART) that would be interesting as well.

#5

I ran the FAI command you posted above to build a ‘stock’ image but I still have the same problem.

We’re using a custom kernel boot image (modified apq dtsi) but that likely isn’t the problem because it is the same boot image used with the latest release buster image released on the Linaro site (linaro-buster-developer-dragonboard-410c-359.img.gz) and the combination works fine.

Here is the UART output at boot for both the custom and stock image:
https://s3-us-west-2.amazonaws.com/temp-log-files/logs/kernel_uart.tar.gz

#6

thanks for the boot log, i think it helped :wink:

if you inspect the log, you will find lots of errors! I suppose the “linaro_default” log is not with the official Linaro image, but with your own FAI image built “like Linaro”, and using your custom kernel.

the following error:

[ 9.605646] drm: Unknown symbol __arch_clear_user (err 0)
[ 9.632951] videodev: Unknown symbol __arch_copy_in_user (err 0)
[ 9.633082] videodev: Unknown symbol __arch_clear_user (err 0)

Suggests that you are rebuilding the kernel Image file with a custom/modified kernel, like you said, but then you are booting this Image file (in your boot image) without the corresponding kernel modules files which are installed during the FAI process.

It is a bit unfortunate for now, but the kernel Image and the modules used throughout the build are put in 2 different places (the boot image and the rootfs) and they need to match. These errors mean that the kernel cannot properly load the modules because the modules in the rootfs correspond to a different kernel Image.

You need to deploy your own kernel modules to the rootfs before you can boot.

In the FAI process the kernel modules are installed by our custom linux-image-xxx .deb package.

#7

Hi @anon91830841,

The linaro_default log was for the FAI image built with the FAI repo, without modifications. But yes, with the custom boot image.

What i’m confused about is this:

I used the same custom kernel boot image with all three instances–linaro latest release rootfs image, our custom rootfs image, and the ‘stock’ rootfs image as mentioned above. Again, with the latest release image, everything works fine (despite the custom boot image, which differs only by the apq8016.dtsi/dtb).

I guess I just don’t understand how the latest release image (18.01) can function but building with FAI, without modifications, will not. Am I missing something?

Is it possible there is a discrepancy between the current version of the linux-image-xxx.deb package used in the FAI process and the one which is installed in the 18.01 image? Would this explain why the fresh stock build could fail while the 18.01 offical image would not? Is there a place I can check which git commit the linux-image-xxx.deb was built from?

(I suspect I should try to run the FAI ‘stock’ image with the latest release boot image and see if the same wifi error occurs?)

Second, and perhaps i’m missing something, at one point I built all the kernel modules from scratch, using the latest release page as a guide and installed them on the custom rootfs. Is this what you mean when you said:

You need to deploy your own kernel modules to the rootfs before you can boot.

If so, are there any resources you can point me towards to get a better understanding of what i’m doing wrong and/or how to fix it?

#8

thanks again for the good details you provide, it helps!

here are more thoughts…

When the 18.01 image was created a certain version of the kernel was used, and the modules in the image correspond to that version (e.g. a commit in our kernel tree).

When you build an image with FAI today you are pulling the linux-image package from the Linaro debian overlay, which corresponds to our kernel branch today, so basically when you build with FAI today, you are not really rebuilding the same image as in 18.01 release.

so basically, i think your analysis is correct.

The commit ID for the kernel is recorded in the linux-image-xxx package version. Looking at the build log for 18.01:

Get:11 http://obs.linaro.org/linaro-overlay-buster/buster ./ linux-image-4.14.0-qcomlt-arm64 4.14.15-00139-g5511441d1b48-20 [13.3 MB]

and looking at your build log:
Get:110 http://obs.linaro.org/linaro-overlay-buster/buster ./ linux-image-4.14.0-qcomlt-arm64 4.14.69-00699-g64c00cd3213f-40 [14.6 MB]

they are clearly different.

if you make customization in the kernel, you probably need to either create a linux-image debian package for your kernel, or at least install/overlays the kernel modules that correspond to your custom kernel, like you said you did once.

Making a .deb package would be more work. The kernel makefile has a deb-pkg target that can possibly be used to create a .deb file for your own kernel, that would be something to explore, but it’s been some time since i haven’t tried that. if you manage to get a linux-image .deb file for your kernel you need to install it in your custom image, with FAI either using your own package archive or using some custom scripts.

It might be a good thing for us to document that… i think it is a very valid use case, to be honest… but as far as i know this is not documented.

#9

Excellent! Thanks for this! I think we’re on the right path and i’ll get back to you as I make progress! I’ll try to document what I do and make it available to you if it works!

This would make sense why the kernel modules I built wouldn’t work, they were built from the same source as my boot image, and therefore not the same as in the rootfs.

I really appreciate the time you’ve given me! Happy holidays/new year!

#10

Hi @anon91830841,
Is there any way older versions of the kernel .deb packages can be kept in the Linaro package repository? Or, some way of indicating which commit in the kernel git the repository is currently holding?

The ideal scenario would be the ability to generate the “exact” same build as the official release. Otherwise, I foresee there being ongoing problems trying to keep the kernel versions in sync…

#11

@anon91830841
The kernel mismatch does appear to be the problem. I was able to find the 4.14.69 kernel commit and build the boot image.

The hexagon chip is now crashing repeatedly but this is, currently, not a huge deal as we’re not using it.