No network devices available after Updating Debian

Today I installed the latest copy of UNIX Demain (4-14). I was able to add my network WIFI.
Then I did the
apt-get update command

and I added all the updates and rebooted but now I cannot connect to the network.
The icon says “No network devices available”
When I run nmtui, I can see my network connection under “Edit” but I cannot activate it. The list of networks to activate is empty.

This really looks like this:

https://bugs.96boards.org/show_bug.cgi?id=744

If so, yes, this is a known problem (and I am not sure how to fix it :wink: , but there is a workaround in the bug report as well.

I thought that on kernel/modules update, old kernel/modules were NOT automatically removed (e.g. on my debian laptop I can boot on old kernel version if not manually removed). So in this condition kernel should be able find the corresponding (old) modules. Is it different with Dragonboard ?

@Loic, the kernel package is upgraded, it’s not a new kernel package. The binary kernel package installed in the image is “linux-image-4.14.0-qcomlt-arm64”, and that’s the one that gets bump’d when we build a new kernel.

the problem is that we pushed a new kernel , so the binary package is upgraded in the rootfs, so the modules are upgraded too. but we will be booting with the original Image file… not the new one. that’s because the Image file is embedded in the boot image along with the bootloader, even though the Image file is also there in the rootfs…

there is no ‘trivial’ solution…

  • we could create a new kernel binary package for each build, and the kernel would never be upgraded at all
  • we could use uboot instead of LK, and have uboot fetch the image file from rootfs
  • we could maybe get LK to fetch from rootfs (not sure…)
  • we could recreate the boot image when installing a kernel and reflashing it into ‘boot’ partition (but what if the user didn’t use boot partition, but ‘fastboot boot’ command?)

Is there any ETA on when this will be fixed? I might be able to do the fix suggested in the bug report but I need more detailed instructions.
Is loading an older version of Debian an option,

Is there any ETA on when this will be fixed? I might be able to do the fix suggested in the bug report but I need more detailed instructions.
Is loading an older version of Debian an option,

Fixing up the damage is a little tricky (because there is no network)
however if you can reinstall it should be OK to reinstall 18.01. You can
avoid the problem happening by marking the kernel package so apt doesn’t
try to upgrade it.

If you run: sudo apt-mark hold linux-image-4.14.0-qcomlt-arm64
before the upgrade then it should prevent any problems.

To be doubly sure you might also like to sudo apt install abootimg
before you upgrade too (you can use abootimg to repair networking on
a broken system by copying the new kernel to the boot partition).

Daniel.

1 Like

@danielt, the workaround does not require network access to the board, since you can download the ‘latest’ boot image from snapshots.linaro.org and reboot with this image.

but you are right, that another workaround would be to regenerate the boot after from the board after the upgrade.

in fact something like

abootimg -u /dev/disk/by-partlabel/boot -k <Image.gz+dtb>

would work

@dstrower, I don’t have an ETA for a proper fix, at this point.

This really looks like this:

https://bugs.96boards.org/show_bug.cgi?id=744

If so, yes, this is a known problem (and I am not sure how to fix it
:wink: , but there is a workaround in the bug report as well.

I think it is to do with how we have versioned the kernel package.

The normal debian kernel has a version number that gets bumped when
modules become incompatible.

linux-image-4.16.0-2-amd64 (4.16.12-1)
linux-image-4.16.0-1-amd64 (4.16.5-1)

The distro tools automatically prevent linux-image-4.16.0-1 from being
removed during an upgrade (assuming it met the properties requires to
be protected, such as being the running kernel). In our case all the
unique numbers are exclusively in the version number so the distro
scripts don’t protect our kernels in the same way.

linux-image-4.14.0-qcomlt-arm64 (4.14.15-00139-g5511441d1b48-20)

@danielt, the workaround does not require network access to the board, since you can download the ‘latest’ boot image from snapshots.linaro.org and reboot with this image.

Agree. Using fastboot to workaround does not require to fix networking
access… but it does require people to know how long ago they did the
upgrade that broke the system (and some folks don’t reset their boards
very often).

@danielt
I did your suggestion of reloading the UNIX and then doing the command:
sudo apt-mark hold linux-image-4.14.0-qcomlt-arm64
before the upgrade.
This worked. My Dragonboard is running now.

Thank you.

Thanks @dstrower for posting this question, I had the exact same issue. Thanks @danielt for the suggested fix.

@anon91830841, I think It’s important to fix/workaround this issue quickly, apt-upgrade is commonly used which will lead to many new issues opened on the forum for various reasons (WiFi, BT, camera, video encoder) because of missing modules.

after a bit of discussion, we decided to add @danielt workaround in our image by default. so the kernel package will be held by default and the kernel will never be upgraded… which hopefully will not lead to too many issues…

‘power users’ who wants to use the latest kernel, will have to un-hold the package, but they will know that they have to worry about the boot image. for users that don’t want to worry about that, then apt-get upgrade should be less error prone…

it’s far from being perfect, but hopefully it’s a bit better…

the change has just been submitted here https://review.linaro.org/#/c/25914/, once it’s merged the new images will have it.

1 Like

Did a new install to eMMC of 528 image from Linaro, then sudo apt-get update, upgrade, dist-upgrade. And no Network devices.
Is the above issue still the reason? After more then 18 months?

what’s the output of

lsmod

and

dmesg | grep wcn

Thanks for the quick reaction Loic. Here they are

before upgrade
after upgrade but before reboot
after reboot

But txt files are impossible to upload?

[ 10.560884] qcom-wcnss-pil a204000.wcnss: a204000.wcnss supply vddcx not found, using dummy regulator
[ 10.589602] remoteproc remoteproc0: a204000.wcnss is available
[ 10.607325] remoteproc remoteproc0: Direct firmware load for wcnss.mdt failed with error -2
[ 10.620321] remoteproc remoteproc0: powering up a204000.wcnss
[ 10.657819] remoteproc remoteproc0: Direct firmware load for wcnss.mdt failed with error -2

A lot less then the before update lsmod

Module Size Used by
snd_soc_hdmi_codec 16384 1
venus_enc 28672 0
venus_dec 28672 0
crc32_ce 16384 0
qcom_wcnss_pil 16384 0
qcom_common 16384 1 qcom_wcnss_pil
qcom_glink_smem 16384 1 qcom_common
qcom_sysmon 16384 1 qcom_wcnss_pil
remoteproc 49152 3 qcom_wcnss_pil,qcom_sysmon,qcom_common
qmi_helpers 24576 1 qcom_sysmon
adv7511 32768 0
snd_soc_lpass_apq8016 16384 2
qrtr 20480 4
snd_soc_msm8916_digital 36864 1
snd_soc_lpass_cpu 16384 1 snd_soc_lpass_apq8016
snd_soc_lpass_platform 16384 1 snd_soc_lpass_cpu
qcom_camss 131072 0
msm 503808 15
venus_core 77824 2 venus_dec,venus_enc
videobuf2_dma_sg 20480 3 qcom_camss,venus_dec,venus_enc
v4l2_mem2mem 24576 3 venus_dec,venus_core,venus_enc
v4l2_fwnode 16384 1 qcom_camss
mdt_loader 16384 3 qcom_wcnss_pil,msm,venus_core
drm_kms_helper 204800 2 msm,adv7511
videobuf2_memops 16384 1 videobuf2_dma_sg
videobuf2_v4l2 24576 4 qcom_camss,venus_dec,venus_enc,v4l2_mem2mem
snd_soc_apq8016_sbc 16384 0
snd_soc_msm8916_analog 40960 1
videobuf2_core 49152 6 qcom_camss,venus_dec,videobuf2_v4l2,venus_core,venus_enc,v4l2_mem2mem
msm_rng 16384 0
videodev 233472 7 videobuf2_core,qcom_camss,venus_dec,videobuf2_v4l2,venus_core,venus_enc,v4l2_mem2mem
leds_qcom_lpg 20480 0
drm 462848 12 drm_kms_helper,msm,adv7511
rng_core 20480 1 msm_rng
media 45056 2 videodev,qcom_camss
rpmsg_char 16384 0
rmtfs_mem 16384 0
ip_tables 24576 0
x_tables 45056 1 ip_tables
rtc_pm8xxx 16384 1
i2c_qcom_cci 16384 0

Only the after upgrade and reboot details added …

Look like a firmware issue, could you please look into /lib/firmware dir, if wcnss.* files are presents:

find /lib/firmware -name wcn*

It’s possible driver expects them a a new location… if you find them copy them directly into /lib/firmware and reboot.

I will try that Loic.

In the meantime I did a
$ sudo fastboot flash boot boot.img as suggested in the bugreport.
See picture below. This did not resolve the issue …

The files are deeper in that dir:
./qcom/msm8916/wcnss.* has 9 files including *.mdt
In total in that dir 22 files with modem.* and mba.*
Also has another dir at same level ./wlan
and at level of msm8916 a dir ./venus-1.8

copied only the wcnss.* files into /lib/firmware
reboot
and wireless is working again

Thank you Sir!

I had the same issue as well, no wifi after upgrade and this solution fixed it.