Dragonboard 410c wifi problems

Hi all,
For the last two years I am facing wifi problems on my DB410c that affects our developments seriously.
we are trying to work around it, but resetting the wifi from time to time and reconnect to the wifi, but still there are serious problems.

I am using Linaro, debian 9.4 stretch, and the hostapd -v is 2.4

the current problem is that some processes are completely stuck, after running hotspotd (to make the wifi to be in access point mode).

I have the dmsg log and the /var/log/messages log if it helps - how can I attach them as a file to here?

Debian stretch implies you are using the 2017.09 release (or earlier). How many of your issues reproduce on 2019.01?

To be clear, there remain some bugs on 2019.01, especially https://bugs.96boards.org/show_bug.cgi?id=538 but running an old release is going to give you additional troubles with WiFi.

Hi Daniel, and thanks for the great help!

  1. “How many of your issues reproduce on 2019.01?” - I don’t know. To know, I need to build the operating system 2019.01, install everything and put the code, build the openCV, our custom drivers, etc. it takes around a full week of work (quite an effort). Is there a way to easily upgrade the system from the command line, or should I invest this full week of work?

  2. Are you interested in the dmsg and var log?

  3. Isn’t there a way to upgrade packaged, like hostapd, etc, instead of everytime build the kernel and invest a week of work?

One discovery: I have two dragonboards running in the same room with the same MAC. Can it be the reason?

Multiple boards with the same MAC address - Very Bad…

Network protocols are based on the concept that every device has a globally unique MAC address.

I am not sure what the current process is to change the MAC address, but having the MAC address permanently stored on the eMMC is something I had asked for years ago. I don’t know if it has been implements. I had also asked Arrow to place a sticker on the board with the MAC address (even if the MAC address is not programmed onto the board), does your board have a sticker with a MAC address on it?

What is the current state of MAC address storage and stickers on the board? I do know that at one time when Debian first booted it was supposed to generate a random MAC address to prevent the collisions you seem to be seeing.

-L-

So this is the thing: We always create one dragonboard (build all the packages there, install the code, etc, etc), and then simply duplicate it by copying the eMMC (bitwise).
So, I guess, the MAC address was copied as well…

This resulted serious problems, and also some processes (especially sudo-run processes) on the board to get hanged, with no indication of the duplicate MAC problem. (below I put the stack trace of the dmsg error, but nothing said MAC there…)

Do you know how can I permanently change the MAC address from within the code itself, on the board (without flashing, or using external cables)?

=====

[23840.426254] Process hostapd (pid: 25898, stack limit = 0xffff800035208020)
[23840.427823] Stack: (0xffff80003520bad0 to 0xffff80003520c000)
[23840.434508] bac0: ffff80003520bb00 ffff000000ce0358
[23840.440328] bae0: ffff800031fab520 ffff800035284400 0000000000000001 ffff800031fab5f0
[23840.448145] bb00: ffff80003520bb40 ffff000000cddee8 0000000000000080 ffff000000ce5410
[23840.455952] bb20: ffff800031fab54c ffff800031fab520 ffff80003520bb40 0000000000cddecc
[23840.463766] bb40: ffff80003520bb80 ffff000000c4cd98 ffff800031faa700 0000000000000000
[23840.471578] bb60: ffff800031faa700 ffff80003513c910 0000000000000001 0000000000008914
[23840.479390] bb80: ffff80003520bba0 ffff000000c5df5c ffff80003513c900 ffff80003513c000
[23840.487202] bba0: ffff80003520bc00 ffff000000c5e60c ffff80003513c000 ffff000000c8a8c8
[23840.495021] bbc0: 0000000000000001 ffff80003513c048 0000000000001002 0000000000008914
[23840.502831] bbe0: ffff80003594c200 0000ffffdbac3a00 0000000000000914 ffff0000088b9eec
[23840.510639] bc00: ffff80003520bc20 ffff0000088b9f68 ffff80003513c000 ffff0000088b9ea0
[23840.518452] bc20: ffff80003520bc60 ffff0000088ba22c ffff80003513c000 0000000000001003
[23840.526263] bc40: 0000000000000001 0000000000000000 ffff80003513c000 ffff80003513c000
[23840.534075] bc60: ffff80003520bca0 ffff0000088ba310 ffff80003513c000 00000000ffffff9d
[23840.541888] bc80: 0000000000001002 0000000000000000 0000000000000001 0000000000000000
[23840.549710] bca0: ffff80003520bcd0 ffff000008927978 ffff80003520bd48 00000000ffffff9d
[23840.557514] bcc0: ffff80003594c210 0000000000000000 ffff80003520bd70 ffff000008929a3c
[23840.565325] bce0: 0000000000008914 ffff000009039080 ffff000009039080 0000000000000007
[23840.573139] bd00: 0000ffffdbac3a00 0000000000008914 0000000000000123 000000000000001d
[23840.580951] bd20: ffff0000089f2000 ffff800035208000 ffff80003594c210 ffff80003513c000
[23840.588765] bd40: ffff0000089f2000 000000306e616c77 0000000000000000 0000000000001003
[23840.596575] bd60: 0000000000000000 0000000000000000 ffff80003520bd80 ffff0000088977b0
[23840.604398] bd80: ffff80003520bdb0 ffff000008898530 0000000000008914 0000ffffdbac3a00
[23840.612213] bda0: 0000000000000010 0000ffffdbac3a00 ffff80003520bdf0 ffff0000081ee7b0
[23840.620014] bdc0: ffff800036107300 0000ffffdbac3a00 ffff800032e7c030 0000000000000007
[23840.627826] bde0: 0000ffffdbac3a00 ffff0000081f9eec ffff80003520be80 ffff0000081eef04
[23840.635638] be00: ffff800036107300 0000000000000000 ffff800036107300 0000000000000007
[23840.643455] be20: 0000ffffdbac3a00 ffff00000889b93c 0000000000000000 0000000000000000
[23840.651264] be40: ffff80003520be80 ffff0000081eeec0 ffff800036107300 0000aaaad1129000
[23840.659079] be60: ffff800036107300 0000000000000007 0000ffffdbac3a00 ffff0000081eeea4
[23840.666889] be80: 0000000000000000 ffff000008082ef0 0000000000000000 0000aaaad1129000
[23840.674701] bea0: ffffffffffffffff 0000ffff8e557eac 0000000000000000 0000000000000015
[23840.682512] bec0: 0000000000000007 0000000000008914 0000ffffdbac3a00 0000aaaafab444f6
[23840.690326] bee0: 0000ffffdbac3a0f 0000ffff8e5db9b8 0000000000000000 0000000000000000
[23840.698150] bf00: 000000000000001d 0000aaaafab44910 5d9bd09800000002 0000000029006528
[23840.705950] bf20: 000500160000001c 290065285d9bd098 0000000000000023 0000000000000000
[23840.713763] bf40: 0000aaaad1129ad8 0000ffff8e557ea0 0000000000000000 0000000000000000
[23840.721575] bf60: 0000aaaad1129000 0000000000000007 0000ffffdbac3a00 0000aaaafab444f0
[23840.729389] bf80: 0000000000000001 0000ffffdbac3b38 0000aaaad10ca0f0 0000aaaafab45140
[23840.737201] bfa0: 0000aaaad112b2b0 0000ffffdbac39c0 0000aaaad107ea54 0000ffffdbac39c0
[23840.745013] bfc0: 0000ffff8e557eac 0000000000000000 0000000000000007 000000000000001d
[23840.752833] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[23840.760644] Call trace:
[23840.768444] Exception stack(0xffff80003520b900 to 0xffff80003520ba30)
[23840.770706] b900: ffff8000211cec00 0001000000000000 ffff80003520bad0 ffff000000cdf1ac
[23840.777305] b920: 0000000000000004 0000000000008000 ffff0000090959f0 0000000000000003
[23840.785117] b940: ffff000008f896d0 0000000000000004 000000040000070f 00000000000bfc00
[23840.792930] b960: 0000000000000000 0000000000000000 ffff80003520b980 ffff000008394d78
[23840.800743] b980: ffff80003520b9d0 ffff0000081d6d90 ffff0000090959d0 0000000000007e00
[23840.808554] b9a0: 00000000a77fa000 ffff8000277faf00 0000000000000040 000000000000003f
[23840.816372] b9c0: 0000000000000002 ffff7dfffe000000 ffff80003610f208 0000000000000000
[23840.824179] b9e0: 0000000000000001 0000000000000040 ffff0000090477c0 0000000000000001
[23840.831992] ba00: 0000000000000068 ffff800037f34450 0000000000000000 0000000000000000
[23840.839804] ba20: ffff800031de2b40 0000000000000000

Well, a fix has been merged in the second stage bootloader (LK), to derive a generate the mac address from the eMMC controller hardware ID, which should give a unique mac address (and Bluetooth BD address). You need (at least) to flash the last version of LK.

The newest firmware package is available at:
http://snapshots.linaro.org/96boards/dragonboard410c/linaro/rescue/latest/dragonboard-410c-bootloader-emmc-linux-*.zip

@Loic, so that means that with this fix, if we copy the eMMC of one dragonboard to many others, each will have difference MAC address? If so, that’s good.
So I basically need to upgrade our DB from 2017 kernel to 2019 kernel plus the boot loader update in order to solve the wifi issues? Another question: Is it possible to upgrade the debian 2017 to 2019 from the command line without flashing it? If we must flash, can the ‘user space’ stay the same, or we have to start ‘fresh’ and make/build everything from scratch?

yes, it would fix this problem. However doing a bitwise copy of a Linux system can have other problems, there are other files which are not supposed to be duplicated across linux systems. For example:

  • /etc/machine-id
  • the SSH keys in /etc/ssh

so generally speaking you could have other problems once you fix the MAC address issue…

@ndec what’s the importance of /etc/machine-id ?

@danielt,

Changing linaro from 17.09 to 19.01 is hard, since all our drivers patches to the kernel (for camera) need to be rewritten.
However, when I do sudo apt-get upgrade I can get to debian 9.11 (instead of current 9.4).

Would the debian change 9.4->9.11 be enough to solve the wifi instability problems?

No. AFAIR most of the important changes are in the kernel.