Hikey970 running Debian image: Two PCIe devices cause kernel crash (possible race condition?)


#1

Hi all,

First of all, I apologize for the long post but I couldn’t find a simple and easy way to explain the problem without providing at least this information.

I am facing often kernel crashes when using/connecting two PCIe devices on the Kirin970 platform.

Hardware setup and Image:
My development scenario requires ethernet connection (using onboard PCIe Realtek 8168 SoC) and an additional 802.11ac radio module (attached to the mini-PCIe slot), which should work simultaneously.
The radio modules that I’ve tried were QCA6174 (M.2 radio module which requires an additional M.2 to mini-PCIe adapter; I’ve tried to fit it to the M.2 slot but it does not fit properly and even if forced, it is not detected as the connection contacts seem to be misaligned) and, more recently, a QCA9994 (mini-PCIe radio module, directly connected to the slot). Both Qualcomm radios rely on the Ath10k driver but to enable VHT160 or VHT80+80 modes, QCA9994 requires some patching either on the ATH10k and on the respective FW.
The image running on the Hikey970 is based on Debian, which was custom compiled based on the instructions provided in this post. To better debug this problem, my current kernel config is very similar to the default provided hikey970_defconfig config file (I am just enabling some debug options and some ath10k related options and disabling android_paranoid option).

Making the Ath10k work (MSI vs INT interrupt mode):
The first issue that I saw was that the Ath10k was constantly failing to detect the HTT core and/or load the FW, which makes it fail the initial device probe process. I discovered that loading the Ath10k kernel module forcing the legacy interrupt mode, allows the radios to enumerate successfully and they work. However, I know from other previous designs that, at least the QCA6174 can work fine in MSI interrupt mode…which started to suggest me that there was some problem with the Kirin970/designware PCIe driver layer. I did try to debug why it was failing but I couldn’t get a solution (it seemed to hang up forever waiting for an interrupt/value that never happened). So, what I did was to provide boot argument pci=nomsi to the kernel and force all PCIe devices to work in INT (aka legacy) mode.

Behavior when using just one of the specified devices:
If just “enabling one PCIe device at a time”, they seem to work perfectly fine (tested for at least 48 consecutive hours each one).
The problem is that “enabling one PCIe device at a time” means to either:

  • just have the Realtek 8168 connected (onboard PCIe connection); no radio module fitted in either the M.2 or mini-PCIe slots;
  • fit one radio module to the mini-PCIe slot but remove one of the devices (either the Realtek or the QCA radio module) from the PCI enumeration list during boot;

With the two devices enumerated at the same time on the system, the chances for a crash increase drastically, especially if under heavy network traffic load.

Crash dump analysis:
Although the crashes can have different output information, they seem to happen always due to the same reason: an attempt to read/access information on a PCIe IO memory area that is not accessible at that precise moment.
The following crash dump example illustrates the typical crash info output I see:

Exceptions seen:

[ 36.601101] Unhandled fault: synchronous external abort (0x96000210) at 0xffff00000e000018
[ 36.601104] Unhandled fault: synchronous external abort (0x96000210) at 0xffff00001139703e

Looking at the PC register per CPU when the exception occurred:

CPU0:
[ 36.601188] PC is at rtl8168_interrupt+0x40/0x408 [r8168]

which corresponds to drivers/net/ethernet/realtek/r8168_n.c:28165, function: rtl8168_interrupt, instruction:
status = RTL_R16(IntrStatus);

and more specifically, to following assembly code:

0000000000066348 <rtl8168_interrupt>:
(…)
__raw_readw():
/home/vagrant/projects/hikey970_linux_k4_9/linux/./arch/arm64/include/asm/io.h:80
asm volatile(ALTERNATIVE(“ldrh %w0, [%1]”,
66388: 79400022 ldrh w2, [x1]

where X1 register contains:

x1 : ffff00001139703e

CPU1:
[ 38.696705] PC is at kirin_pcie_rd_own_conf+0x6c/0x140

which corresponds to drivers/pci/host/pcie-kirin.c:62, function: kirin_pcie_rd_own_conf, instruction:
*val = readl(pp->dbi_base + (where & ~0x3));

which corresponds to following assembly code:

00000000000001b0 <kirin_pcie_readl_rc>:
(…)
__raw_readl():
/home/vagrant/projects/hikey970_linux_k4_9/linux/./arch/arm64/include/asm/io.h:91
2e4: b9400000 ldr w0, [x0]

where X0 register contains:

x0 : ffff00000e000018

Both X1 and X0 registers contain valid addresses that correspond to the allocated PCIe IO memory areas (“R8168->ioaddr” and “kirin->dbi_base” respectively).
I have ensured that on crash, these registers contain valid addresses because either they were accessible previously (typically a thousand multiple times) before a crash or, when I am lucky enough (which happens few times), the system can run for hours without crashes performing those accesses.
PCIe power management is disabled (through kernel boot argument pcie_aspm=off) and I have ensured this is not related with an hardware fault, either because

  • the radio modules work fine on other platforms
  • because different Hikey970 exhibit the same crash behavior.

It seems that there is a race condition that allows two or more entities to access the PCIe bus at the same time (through the IO memory resource).
Looking at the typical functions call back trace, I am pretty unsure where to put the mutual exclusion logic to prevent this situation. (Please note that the first functions listed were the last ones running before the crash.)
Example 1:

CPU_A:
[ 81.120100] [<ffff000000d34388>] rtl8168_interrupt+0x40/0x408 [r8168]
[ 81.120169] [<ffff00000811b7ac>] __handle_irq_event_percpu+0x5c/0x148
[ 81.120169] [<ffff00000811b8cc>] handle_irq_event_percpu+0x34/0x88
[ 81.120175] [<ffff00000811b968>] handle_irq_event+0x48/0x78
[ 81.120197] [<ffff00000811f868>] handle_fasteoi_irq+0xd8/0x1
[ 81.120199] [<ffff00000811a88c>] generic_handle_irq+0x24/0x38
[ 81.120200] [<ffff00000811af14>] __handle_domain_irq+0x84/0xf0
[ 81.120221] [<ffff00000808173c>] gic_handle_irq+0x54/0xa8
[ 81.120329] [<ffff0000080831e4>] el1_irq+0xe4/0x188
[ 81.120348] [<ffff0000080c9850>] irq_exit+0x90/0xd0
[ 81.120348] [<ffff00000811af18>] __handle_domain_irq
[ 81.120359] [<ffff00000808173c>] gic_handle_irq+0x54/0xa8
CPU_B:
[ 84.351246] [<ffff000008436820>] kirin_pcie_readl_rc+0x20/0x40
[ 84.357087] [<ffff000008435508>] dw_pcie_readl_rc+0x18/0x38
[ 84.362667] [<ffff000008435734>] dw_pcie_prog_outbound_atu+0x20c/0x390
[ 84.369202] [<ffff000008435c78>] dw_pcie_rd_conf+0x150/0x1a0
[ 84.374873] [<ffff000008411278>] pci_bus_read_config_word+0x80/0xe0
[ 84.381200] [<ffff000000d3df5c>] rtl8168_esd_timer+0xf4/0x560 [r8168]
[ 84.387654] [<ffff00000812ed2c>] call_timer_fn.isra.5+0x24/0x80
[ 84.393583] [<ffff00000812ee34>] expire_timers+0xac/0xd0
[ 84.398904] [<ffff00000812ef54>] run_timer_softirq+0xfc/0x1a0
[ 84.404658] [<ffff000008081a18>] __do_softirq+0x128/0x230
[ 84.410065] [<ffff0000080c9850>] irq_exit+0x90/0xd0
[ 84.414954] [<ffff00000811af18>] __handle_domain_irq+0x88/0xf0
[ 84.420793] [<ffff00000808173c>] gic_handle_irq+0x54/0xa8

Example 2:

CPU_A:
[ 34.079436] [<ffff0000084369c8>] kirin_pcie_readl_rc+0x20/0x40
[ 34.079446] [<ffff000008435508>] dw_pcie_readl_rc+0x18/0x38
[ 34.079456] [<ffff000008435734>] dw_pcie_prog_outbound_atu+0
[ 34.079466] [<ffff000008435c78>] dw_pcie_rd_conf+0x150/0x1a0
[ 34.079475] [<ffff0000084111b0>] pci_bus_read_config_byte+0x7
[ 34.079491] [<ffff000000d3e19c>] rtl8168_esd_timer+0x334/0x560
[ 34.079562] [<ffff00000812ed2c>] call_timer_fn.isra.5+0x24/0x80
[ 34.079582] [<ffff00000812ee34>] expire_timers+0xac/0xd0
[ 34.079593] [<ffff00000812ef54>] run_timer_softirq+0xfc/0
[ 34.079607] [<ffff000008081a18>] __do_softirq+0x128/0x230
[ 34.079619] [<ffff0000080c9850>] irq_exit+0x90/0xd0
[ 34.079621] [<ffff0000080906c8>] handle_IPI+0x138/0x
[ 34.079641] [<ffff000008081788>] gic_handle_irq+0xa0/0xa
[ 34.079748] [<ffff0000080831e4>] el1_irq+0xe4/0x188
CPU_B:
[ 37.076376] [<ffff000000d34388>] rtl8168_interrupt+0x40/0x408 [r8168]
[ 37.082827] [<ffff00000811b7ac>] __handle_irq_event_percpu+0x5c/0x148
[ 37.089276] [<ffff00000811b8cc>] handle_irq_event_percpu+0x34/0x88
[ 37.095463] [<ffff00000811b968>] handle_irq_event+0x48/0x78
[ 37.101048] [<ffff00000811f868>] handle_fasteoi_irq+0xd8/0x1c8
[ 37.106894] [<ffff00000811a88c>] generic_handle_irq+0x24/0x38
[ 37.112650] [<ffff00000811af14>] __handle_domain_irq+0x84/0xf0
[ 37.118494] [<ffff00000808173c>] gic_handle_irq+0x54/0xa8

Looking at these function call back trace, it is possible to infer that:

  • the R8168 interrupt handler has no call dependency on the kirin or designware-host pcie driver layers;
  • the only common path that I can see is the gic_handle_irq but I am not sure if this would be the best place to put the mutual exclusion logic;

At first glance, it seems I should prevent the two interrupt handlers from running at same time but this is not the only condition that makes it crash.
If R8168 is loaded and device enumerated, just running a simple lspci command while TX/RX data through the interface makes it crash (even if in MSI interrupt mode and the only PCIe device present). In this case, one of the calls is not in interrupt context but it also tries to access the same resource.

lspci inconsistent output:
More recently, while debugging just the crashes when running the lspci command, I noticed that the hex dump output (of the config space) for the Root PCIe device is not always the same (even if running in the same Linux session and consecutively lspci calls…when the system does not crash).

lspci run 1:

0000:00:00.0 PCI bridge: Huawei Technologies Co., Ltd. Device 3670 (rev 01)
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12
00: e5 19 70 36 07 01 10 00 01 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 ff 00 f0 00 00 20
20: 00 01 30 01 40 01 40 01 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
(…)
0000:05:00.0 Network controller: Qualcomm Atheros Device 0046 (rev 01)
Subsystem: Qualcomm Atheros Device cafe
Kernel driver in use: ath10k_pci
00: 8c 16 46 00 06 00 10 00 01 00 80 02 00 00 00 00
10: 04 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 fe ca
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00

lspci run 2:

0000:00:00.0 PCI bridge: Huawei Technologies Co., Ltd. Device 3670 (rev 01)
Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12
00: e5 19 70 36 ff ff ff ff 01 00 04 06 00 00 01 00
10: 00 00 00 00 ff ff ff ff 00 01 ff 00 ff ff ff ff
20: 00 01 30 01 40 01 40 01 00 00 00 00 ff ff ff ff
30: 00 00 00 00 ff ff ff ff 00 00 00 00 ff 01 00 00
(…)
0000:05:00.0 Network controller: Qualcomm Atheros Device 0046
Subsystem: Qualcomm Atheros Device cafe
Kernel driver in use: ath10k_pci
00: 8c 16 46 00 06 00 10 00 00 00 80 02 00 00 00 00
10: 04 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 fe ca
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00

Comparing these output, it is possible to see that, for some reason, on lspci run 2, some config space reads failed (ff). In this precise example, command register should be the same and even status register read should not return ff; I have seen it fail for Device ID and Vendor ID as well. This seems to be frequent on the Root Bridge but less on the other devices. On the other devices, other than the fields that should naturally change (e.g. status register), the previous read fail ff was not seen…but as can be seen on run 2, sometimes the revision is wrongly read on the QCA9994 PCIe device…
I do not see any errors on the bootlog related with the PCIe training so, although unsure, I believe this is only related with wrong mediation of the PCIe bus (concurrent multiple access allowed).

Some of the attempts already tried without success:

  • compile the new driver (v8.046) provided directly by Realtek but it didn’t make any difference.
  • at some point and after consulting the ARM A53/A73 erratas, I started suspecting if this could be also related with one of the bugs that the ALTERNATIVES framework tries to solve (e.g. in certain situations, between the instruction that sets the address on a X register and a ldr instruction that uses that X register as source location, there should be at least two other instructions that don’t modify the X register to avoid invalid data on the X register)…
    Looking at the assembly code, it is not clear to me that these conditions are being satisfied, which was one of the reasons why I wanted to debug using the JTAG (see post).
  • compile kernel 4.19.1 (which includes updated pcie patches and different structure, as well as other patches in the basic IO kernel subsystem - including ALTERNATIVES framework)…but I couldn’t get LDO33 and five other regulators to initialize properly (system hangs waiting for a specific status); without these regulators, I can only see the root bridge and none of the PCIe ports or devices;
  • force IRQ affinity to just one core; this allows it to run much more reliably (as expected) but it can still fail because “data reading/writing processes” (e.g. Iperf or lspci) can still force a direct (and concurrent) access using any other core (and of course, limiting everything to run in one core is not an optimal solution);
    It is possible that this race condition is “visible” because I am forcing the legacy interrupt mode instead of the MSI, which could potentially avoid the race condition…however, as referred previously, I couldn’t get it to work in MSI interrupt mode…and in the case of the R8168, just running a lspci while TX/RX data, also makes it crash.

Questions:

  • has anyone used the Hikey970 with two PCIe devices without facing/spotting this issue?
  • has anyone seen this issue and solved it?
  • did anyone got a QCA radio (based on ath10k driver) working with MSI IRQ support?

Currently, I am trying to implement some mutual exclusion logic shared between the pcie-kirin and r8168 drivers based on spinlocks (not sure if it will be needed on pcie-designware as well). So far, the main problem is to avoid deadlock situations.
Not sure if this is the best approach to solve the race condition but it seems the most quickest and the one that “apparently” needs less modification on r8168 and pcie-kirin drivers (or others).

Any help, guidance or suggestions would be very much appreciated.

Kind regards


#2

Just an update:

Implementing a mutual exclusion mechanism based on spinlocks and sharing the mechanism between the r8168 and the kirin-pcie drivers, I managed to solve the crashes that were related with using the two network interfaces at the same time.

However, it can stilI crash when running lspci (this makes it crash almost instantly when reading the R8168 config space…I read that some drivers don’t behave well when doing this but I am not sure if this explains it). I have seen it crashing when running Iperf…

[ 373.423886] Unhandled fault: synchronous external abort (0x96000210) at 0xffff00001564803e
[ 373.424195] Bad mode in Error handler detected on CPU2, code 0xbf000002 – SError
[ 373.424202] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[ 373.424226] Modules linked in: r8168 ath10k_pci ath10k_core ath mac80211 cfg80211
[ 373.424234] CPU: 2 PID: 3311 Comm: iperf3 Tainted: G S 4.9.78-147536-g7304d83b6c94-dirty #56
[ 373.424236] Hardware name: HiKey970 (DT)
[ 373.424241] task: ffff8001b2a8ae00 task.stack: ffff8001b2978000
[ 373.424248] PC is at 0xffff95e9eaa4
[ 373.424251] LR is at 0xffff95f48e44
[ 373.424254] pc : [<0000ffff95e9eaa4>] lr : [<0000ffff95f48e44>] pstate: 00000000
[ 373.424256] sp : 0000ffffc7335550
[ 373.424262] x29: 0000ffffc7335550 x28: 0000000000000005
[ 373.424267] x27: 0000aaaaca4c9260 x26: 0000000000000001
[ 373.424273] x25: 0000aaaaca4c9340 x24: 0000aaaaca4c93c0
[ 373.424278] x23: 0000ffffc73356a8 x22: 0000000000020000
[ 373.424283] x21: 0000000000000005 x20: 0000ffff95d02000
[ 373.424289] x19: 0000000000000005 x18: 0000ffffc7335294
[ 373.424294] x17: 0000ffff95e9ea78 x16: 0000ffff95f5ed38
[ 373.424300] x15: 00000dd2835ff6f1 x14: 00057bcef2063000
[ 373.424305] x13: 00000000fa000000 x12: 0000000000000016
[ 373.424311] x11: 000000000005aba7 x10: 000000005a6df487
[ 373.424316] x9 : 000ee6b280000000 x8 : 000000000000003f
[ 373.424321] x7 : 00000000000237b0 x6 : 00000000000e9d32
[ 373.424326] x5 : 0000000000000000 x4 : 0000ffffc73355e8
[ 373.424331] x3 : 0000ffff95f341a8 x2 : 0000000000020000
[ 373.424337] x1 : 0000ffff95d02000 x0 : 000000000000111c
[ 373.424339]
[ 373.424342] Process iperf3 (pid: 3311, stack limit = 0xffff8001b2978028)
[ 373.424350] —[ end trace f9b013c43f2deaec ]—
[ 373.590839] Internal error: : 96000210 [#2] PREEMPT SMP
[ 373.596066] Modules linked in: r8168 ath10k_pci ath10k_core ath mac80211 cfg80211
[ 373.603609] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G S D 4.9.78-147536-g7304d83b6c94-dirty #56
[ 373.613346] Hardware name: HiKey970 (DT)
[ 373.617266] task: ffff8001b8cbae00 task.stack: ffff8001b8cdc000
[ 373.623215] PC is at rtl8168_interrupt+0x34/0x360 [r8168]
[ 373.628628] LR is at rtl8168_interrupt+0x30/0x360 [r8168]
[ 373.634024] pc : [] lr : [] pstate: a04001c5
[ 373.641416] sp : ffff8001bfececf0
[ 373.644725] x29: ffff8001bfececf0 x28: ffff8001b8cdc000
[ 373.650045] x27: 0000000000000001 x26: ffff000008f1a820
[ 373.655365] x25: ffff0000095fb1fb x24: ffff000015648000
[ 373.660685] x23: ffff00001564803e x22: ffff00000c2bdcf8
[ 373.666005] x21: 0000000000000000 x20: ffff8001ae0c0000
[ 373.671325] x19: ffff8001ae0c0b00 x18: 0000000000000000
[ 373.676645] x17: 0000000000000000 x16: 0000000000000623
[ 373.681965] x15: ffff8001b828e804 x14: 00000000fef34399
[ 373.687285] x13: ffff0000095e4f00 x12: ffff8001b8cbb730
[ 373.692604] x11: ffff00000a272000 x10: ffff8001b8cdc000
[ 373.697925] x9 : ffff00000952d000 x8 : ffff0000094e8490
[ 373.703245] x7 : 0000000000000000 x6 : 0000000000000002
[ 373.708564] x5 : ffff000009d11000 x4 : ffff8001b8cbb758
[ 373.713884] x3 : fffffffffffffe50 x2 : 0000000000000000
[ 373.719204] x1 : 0000000000000000 x0 : 00000000000001c0
[ 373.724526]
[ 373.724526] SP: 0xffff8001bfecec70:
[ 373.729487] ec70 [ 373.731234] 0c2bdcf8
ffff0000[ 373.734294] 1564803e
ffff0000[ 373.737353] 15648000
ffff0000[ 373.740411] 095fb1fb
ffff0000[ 373.743469]
[ 373.744956] ec90 [ 373.746702] 08f1a820
ffff0000[ 373.749761] 00000001
00000000[ 373.752819] b8cdc000
ffff8001[ 373.755877] bfececf0
ffff8001[ 373.758935]
[ 373.760422] ecb0 [ 373.762167] 00e4dc28
ffff0000[ 373.765225] bfececf0
ffff8001[ 373.768283] 00e4dc2c
ffff0000[ 373.771342] a04001c5
00000000[ 373.774400]
[ 373.775887] ecd0 [ 373.777632] ae0c0b00
ffff8001[ 373.780690] ae0c0000
ffff8001[ 373.783748] 00000000
00010000[ 373.786806] 08f1a820
ffff0000[ 373.789863]
[ 373.791350] ecf0 [ 373.793095] bfeced50
ffff8001[ 373.796153] 0813690c
ffff0000[ 373.799211] ae16d300
ffff8001[ 373.802269] b7bb5a20
ffff8001[ 373.805326]
[ 373.806814] ed10 [ 373.808558] 00000000
00000000[ 373.811617] bfecede4
ffff8001[ 373.814676] 00000147
00000000[ 373.817733] b7bb5a00
ffff8001[ 373.820791]
[ 373.822278] ed30 [ 373.824024] 095fb1fb
ffff0000[ 373.827082] 08f1a820
ffff0000[ 373.830140] 00000001
00000000[ 373.833198] b8cdc000
ffff8001[ 373.836254]
[ 373.837739] ed50 [ 373.839483] bfecedb0
ffff8001[ 373.842539] 08136a2c
ffff0000[ 373.845594] 094e8000
ffff0000[ 373.848650] b7bb5a20
ffff8001[ 373.851705]
[ 373.853192]
[ 373.853192] X3: 0xfffffffffffffdd0:
[ 373.858150] fdd0 [ 373.859901] ********
********[ 373.862957] ********
********[ 373.866013] ********
********[ 373.869067] ********
********[ 373.872123]
[ 373.873608] fdf0 [ 373.875353] ********
********[ 373.878408] ********
********[ 373.881464] ********
********[ 373.884520] ********
********[ 373.887575]
[ 373.889060] fe10 [ 373.890805] ********
********[ 373.893861] ********
********[ 373.896916] ********
********[ 373.899973] ********
********[ 373.903028]
[ 373.904513] fe30 [ 373.906257] ********
********[ 373.909313] ********
********[ 373.912369] ********
********[ 373.915424] ********
********[ 373.918479]
[ 373.919964] fe50 [ 373.921708] ********
********[ 373.924764] ********
********[ 373.927819] ********
********[ 373.930875] ********
********[ 373.933930]
[ 373.935415] fe70 [ 373.937159] ********
********[ 373.940214] ********
********[ 373.943269] ********
********[ 373.946325] ********
********[ 373.949380]
[ 373.950866] fe90 [ 373.952610] ********
********[ 373.955665] ********
********[ 373.958720] ********
********[ 373.961776] ********
********[ 373.964832]
[ 373.966317] feb0 [ 373.968062] ********
********[ 373.971117] ********
********[ 373.974173] ********
********[ 373.977229] ********
********[ 373.980285]
[ 373.981770]
[ 373.981770] X4: 0xffff8001b8cbb6d8:
[ 373.986729] b6d8 [ 373.988473] 0107810f
00000000[ 373.991528] 080cee24
ffff0000[ 373.994584] 080831a4
ffff0000[ 373.997639] 0107810e
0107810f[ 374.000694]
[ 374.002179] b6f8 [ 374.003924] 00000000
00000001[ 374.006978] 080ced1c
ffff0000[ 374.010033] 08081c0c
ffff0000[ 374.013089] 01077e91
01077e8e[ 374.016144]
[ 374.017629] b718 [ 374.019374] 00000000
00000001[ 374.022429] 226255f2
b91f712a[ 374.025485] 00000002
00000000[ 374.028539] 00000000
00000000[ 374.031594]
[ 374.033079] b738 [ 374.034823] 089da888
ffff0000[ 374.037879] 0950c0b8
ffff0000[ 374.040935] 00000000
00000000[ 374.043989] 00022018
00000000[ 374.047044]
[ 374.048530] b758 [ 374.050274] 563553e6
ed37afc6[ 374.053330] 00e4dc28
ffff0000[ 374.056386] 0c2bdd10
ffff0000[ 374.059441] 00000000
00000000[ 374.062496]
[ 374.063981] b778 [ 374.065725] 000c637d
00000000[ 374.068781] a5d38122
623d38d5[ 374.071837] 08a8b0d8
ffff0000[ 374.074892] aceb2898
ffff8001[ 374.077947]
[ 374.079433] b798 [ 374.081176] 00000000
00000000[ 374.084232] 0004278a
00000000[ 374.087288] b6bd730f
b2cca7ab[ 374.090343] 08a289f0
ffff0000[ 374.093399]
[ 374.094885] b7b8 [ 374.096628] 094efd10
ffff0000[ 374.099683] 00000000
00000000[ 374.102739] 000e24d2
00000000[ 374.105794] b877ce3e
edd083aa[ 374.108849]
[ 374.110337]
[ 374.110337] X10: 0xffff8001b8cdbf80:
[ 374.115382] bf80 [ 374.117126] 00000000
00000000[ 374.120181] 00000000
00000000[ 374.123237] 00000000
00000000[ 374.126292] 00000000
00000000[ 374.129347]
[ 374.130832] bfa0 [ 374.132576] 00000000
00000000[ 374.135631] 00000000
00000000[ 374.138687] 00000000
00000000[ 374.141742] 00000000
00000000[ 374.144797]
[ 374.146282] bfc0 [ 374.148027] 00000000
00000000[ 374.151082] 00000005
00000000[ 374.154138] 00000000
00000000[ 374.157193] 00000000
00000000[ 374.160248]
[ 374.161733] bfe0 [ 374.163478] 00000000
00000000[ 374.166532] 00000000
00000000[ 374.169588] ffffffef
ffffffff[ 374.172643] ffffff7f
fffff7ff[ 374.175699]
[ 374.177185] c000 [ 374.178928] 00000002
00000000[ 374.181984] ffffffff
ffffffff[ 374.185039] b8cbae00
ffff8001[ 374.188094] 01618000
00000000[ 374.191149]
[ 374.192634] c020 [ 374.194378] 00010102
00000000[ 374.197434] 57ac6e9d
00000000[ 374.200490] ffdfffff
fffeffff[ 374.203544] ffffffff
ffffffff[ 374.206600]
[ 374.208085] c040 [ 374.209829] ffffffff
7bfffffe[ 374.212884] feffffff
ffffffff[ 374.215940] ffffff7f
ffffffbf[ 374.218995] fff7ffff
fcffffff[ 374.222050]
[ 374.223535] c060 [ 374.225279] ffffffff
ffffffff[ 374.228335] fffbffff
bfffffff[ 374.231390] ffbfff7d
fffd7ffd[ 374.234445] ffffffff
7ff7ffff[ 374.237501]
[ 374.238988]
[ 374.238988] X12: 0xffff8001b8cbb6b0:
[ 374.244033] b6b0 [ 374.245778] 00000000
00000000[ 374.248833] 00000000
00000000[ 374.251889] 00000000
00000000[ 374.254944] 00000000
00000000[ 374.257999]
[ 374.259484] b6d0 [ 374.261228] 00000000
00000000[ 374.264283] 0107810f
00000000[ 374.267339] 080cee24
ffff0000[ 374.270394] 080831a4
ffff0000[ 374.273449]
[ 374.274934] b6f0 [ 374.276678] 0107810e
0107810f[ 374.279733] 00000000
00000001[ 374.282789] 080ced1c
ffff0000[ 374.285844] 08081c0c
ffff0000[ 374.288899]
[ 374.290385] b710 [ 374.292129] 01077e91
01077e8e[ 374.295185] 00000000
00000001[ 374.298240] 226255f2
b91f712a[ 374.301295] 00000002
00000000[ 374.304351]
[ 374.305836] b730 [ 374.307580] 00000000
00000000[ 374.310636] 089da888
ffff0000[ 374.313691] 0950c0b8
ffff0000[ 374.316746] 00000000
00000000[ 374.319802]
[ 374.321287] b750 [ 374.323031] 00022018
00000000[ 374.326086] 563553e6
ed37afc6[ 374.329142] 00e4dc28
ffff0000[ 374.332198] 0c2bdd10
ffff0000[ 374.335253]
[ 374.336739] b770 [ 374.338482] 00000000
00000000[ 374.341538] 000c637d
00000000[ 374.344593] a5d38122
623d38d5[ 374.347648] 08a8b0d8
ffff0000[ 374.350703]
[ 374.352188] b790 [ 374.353932] aceb2898
ffff8001[ 374.356988] 00000000
00000000[ 374.360042] 0004278a
00000000[ 374.363098] b6bd730f
b2cca7ab[ 374.366153]
[ 374.367640]
[ 374.367640] X15: 0xffff8001b828e784:
[ 374.372685] e784 [ 374.374429] ffff0000
00000000[ 374.377485] 00000000
00001000[ 374.380541] 00000000
00000000[ 374.383595] 00000000
09518a98[ 374.386651]
[ 374.388136] e7a4 [ 374.389880] ffff0000
81000152[ 374.392936] 000031d8
00000000[ 374.395991] 00000000
00000001[ 374.399046] 00000000
09518b40[ 374.402101]
[ 374.403586] e7c4 [ 374.405330] ffff0000
00000000[ 374.408385] 00000000
00000000[ 374.411440] 00000000
08f2c958[ 374.414495] ffff0000
b828e130[ 374.417550]
[ 374.419036] e7e4 [ 374.420780] ffff8001
08ff05d8[ 374.423835] ffff0000
b828e590[ 374.426891] ffff8001
00000000[ 374.429946] 00000000
00000278[ 374.433002]
[ 374.434487] e804 [ 374.436231] 00000004
00000004[ 374.439287] ffffffff
00000000[ 374.442341] 00000098
00000130[ 374.445397] ffffffff
ffffffff[ 374.448452]
[ 374.449938] e824 [ 374.451681] 00000000
00000098[ 374.454737] 00000130
ffffffff[ 374.457792] 00000001
b5f66500[ 374.460847] ffff8001
00000000[ 374.463902]
[ 374.465387] e844 [ 374.467131] 00000000
00000000[ 374.470186] 00000000
00000000[ 374.473242] 00000000
00000000[ 374.476297] 00000000
00000000[ 374.479352]
[ 374.480837] e864 [ 374.482581] 00000000
00000000[ 374.485637] 00000000
00000000[ 374.488692] 00000000
00000000[ 374.491748] 00000000
00000000[ 374.494802]
[ 374.496289]
[ 374.496289] X19: 0xffff8001ae0c0a80:
[ 374.501334] 0a80 [ 374.503078] 00000000
00000000[ 374.506133] 00000000
00000000[ 374.509189] 00000000
00000000[ 374.512244] 00000000
00000000[ 374.515300]
[ 374.516785] 0aa0 [ 374.518529] 00000000
00000000[ 374.521585] 00000000
00000000[ 374.524640] 00000000
00000000[ 374.527695] 00000000
00000000[ 374.530750]
[ 374.532235] 0ac0 [ 374.533979] 00000000
00000000[ 374.537034] 00000000
00000000[ 374.540090] 00000000
00000000[ 374.543145] 00000000
00000000[ 374.546200]
[ 374.547685] 0ae0 [ 374.549429] 00000000
00000000[ 374.552484] 00000000
00000000[ 374.555539] 00000000
00000000[ 374.558594] 00000000
00000000[ 374.561649]
[ 374.563134] 0b00 [ 374.564878] 15648000
ffff0000[ 374.567934] b79eb000
ffff8001[ 374.570989] ae0c0000
ffff8001[ 374.574045] 00000000
00000000[ 374.577100]
[ 374.578585] 0b20 [ 374.580329] 00000000
00000000[ 374.583384] 00000000
00000000[ 374.586440] 00000000
00000000[ 374.589494] 00000000
00000000[ 374.592549]
[ 374.594035] 0b40 [ 374.595779] 00000000
00000000[ 374.598834] 00000000
00000000[ 374.601889] 00000000
00000000[ 374.604945] 00000000
00000000[ 374.608000]
[ 374.609486] 0b60 [ 374.611230] 00000000
00000000[ 374.614285] 00000000
00000000[ 374.617340] 00000000
00000000[ 374.620396] 00000000
00000000[ 374.623451]
[ 374.624937]
[ 374.624937] X20: 0xffff8001ae0bff80:
[ 374.629982] ff80 [ 374.631727] 000319e4
00000000[ 374.634782] 00112090
00000000[ 374.637838] 00000102
0000001a[ 374.640893] 0002ce49
00000000[ 374.643948]
[ 374.645433] ffa0 [ 374.647178] 0011209d
00000000[ 374.650232] 00000102
0000001a[ 374.653288] 00027734
00000000[ 374.656344] 001120aa
00000000[ 374.659398]
[ 374.660884] ffc0 [ 374.662628] 00000102
0000001a[ 374.665684] 000299ed
00000000[ 374.668739] 001120b7
00000000[ 374.671795] 00000102
0000001a[ 374.674850]
[ 374.676336] ffe0 [ 374.678079] 0002983c
00000000[ 374.681134] 001120d8
00000000[ 374.684190] 00000102
0000001a[ 374.687245] 00031f77
00000000[ 374.690300]
[ 374.691786] 0000 [ 374.693529] 36706e65
00003073[ 374.696585] 00000000
00000000[ 374.699640] 00000000
00000000[ 374.702695] b8705af0
ffff8001[ 374.705751]
[ 374.707237] 0020 [ 374.708981] 00000000
00000000[ 374.712036] 00000000
00000000[ 374.715091] 00000000
00000000[ 374.718146] 15648000
ffff0000[ 374.721201]
[ 374.722687] 0040 [ 374.724431] 00000147
00000002[ 374.727487] 00000003
00000000[ 374.730542] b8efa050
ffff8001[ 374.733596] b3000050
ffff8001[ 374.736651]
[ 374.738137] 0060 [ 374.739880] ae0c0060
ffff8001[ 374.742936] ae0c0060
ffff8001[ 374.745991] ae0c0070
ffff8001[ 374.749046] ae0c0070
ffff8001[ 374.752101]
[ 374.753589]
[ 374.753589] X28: 0xffff8001b8cdbf80:
[ 374.758635] bf80 [ 374.760378] 00000000
00000000[ 374.763433] 00000000
00000000[ 374.766489] 00000000
00000000[ 374.769544] 00000000
00000000[ 374.772599]
[ 374.774085] bfa0 [ 374.775829] 00000000
00000000[ 374.778885] 00000000
00000000[ 374.781940] 00000000
00000000[ 374.784995] 00000000
00000000[ 374.788050]
[ 374.789536] bfc0 [ 374.791279] 00000000
00000000[ 374.794335] 00000005
00000000[ 374.797390] 00000000
00000000[ 374.800445] 00000000
00000000[ 374.803500]
[ 374.804985] bfe0 [ 374.806729] 00000000
00000000[ 374.809785] 00000000
00000000[ 374.812840] ffffffef
ffffffff[ 374.815895] ffffff7f
fffff7ff[ 374.818950]
[ 374.820435] c000 [ 374.822179] 00000002
00000000[ 374.825235] ffffffff
ffffffff[ 374.828290] b8cbae00
ffff8001[ 374.831345] 01618000
00000000[ 374.834400]
[ 374.835886] c020 [ 374.837629] 00010102
00000000[ 374.840685] 57ac6e9d
00000000[ 374.843741] ffdfffff
fffeffff[ 374.846796] ffffffff
ffffffff[ 374.849851]
[ 374.851337] c040 [ 374.853080] ffffffff
7bfffffe[ 374.856136] feffffff
ffffffff[ 374.859192] ffffff7f
ffffffbf[ 374.862247] fff7ffff
fcffffff[ 374.865302]
[ 374.866788] c060 [ 374.868531] ffffffff
ffffffff[ 374.871587] fffbffff
bfffffff[ 374.874642] ffbfff7d
fffd7ffd[ 374.877698] ffffffff
7ff7ffff[ 374.880752]
[ 374.882239]
[ 374.882239] X29: 0xffff8001bfecec70:
[ 374.887285] ec70 [ 374.889028] 0c2bdcf8
ffff0000[ 374.892084] 1564803e
ffff0000[ 374.895139] 15648000
ffff0000[ 374.898194] 095fb1fb
ffff0000[ 374.901250]
[ 374.902735] ec90 [ 374.904479] 08f1a820
ffff0000[ 374.907535] 00000001
00000000[ 374.910590] b8cdc000
ffff8001[ 374.913645] bfececf0
ffff8001[ 374.916701]
[ 374.918186] ecb0 [ 374.919930] 00e4dc28
ffff0000[ 374.922986] bfececf0
ffff8001[ 374.926041] 00e4dc2c
ffff0000[ 374.929096] a04001c5
00000000[ 374.932151]
[ 374.933637] ecd0 [ 374.935381] ae0c0b00
ffff8001[ 374.938437] ae0c0000
ffff8001[ 374.941492] 00000000
00010000[ 374.944548] 08f1a820
ffff0000[ 374.947603]
[ 374.949088] ecf0 [ 374.950832] bfeced50
ffff8001[ 374.953888] 0813690c
ffff0000[ 374.956944] ae16d300
ffff8001[ 374.959999] b7bb5a20
ffff8001[ 374.963054]
[ 374.964539] ed10 [ 374.966283] 00000000
00000000[ 374.969339] bfecede4
ffff8001[ 374.972394] 00000147
00000000[ 374.975450] b7bb5a00
ffff8001[ 374.978505]
[ 374.979990] ed30 [ 374.981735] 095fb1fb
ffff0000[ 374.984790] 08f1a820
ffff0000[ 374.987845] 00000001
00000000[ 374.990901] b8cdc000
ffff8001[ 374.993956]
[ 374.995441] ed50 [ 374.997185] bfecedb0
ffff8001[ 375.000240] 08136a2c
ffff0000[ 375.003295] 094e8000
ffff0000[ 375.006351] b7bb5a20
ffff8001[ 375.009406]
[ 375.010891]
[ 375.012376] Process ksoftirqd/0 (pid: 3, stack limit = 0xffff8001b8cdc028)
[ 375.019246] Stack: (0xffff8001bfececf0 to 0xffff8001b8ce0000)
[ 375.024988] Call trace:
[ 375.027428] Exception stack(0xffff8001bfeceb00 to 0xffff8001bfecec30)
[ 375.033866] eb00: ffff8001ae0c0b00 0001000000000000 0000000004320000 ffff000000e4dc2c
[ 375.041694] eb20: 00000000a04001c5 ffff000008218de4 ffff8001b8c03d00 0000000000000007
[ 375.049520] eb40: ffff800100000000 ffff00001564803e ffff8001bfeceb60 ffff000008218d3c
[ 375.057348] eb60: ffff8001bfecebb0 ffff000008128164 000000000000037d ffff0000094e8000
[ 375.065174] eb80: ffff8001b8cbae00 0000000000000000 ffff8001b8cbae00 0000000000000000
[ 375.073000] eba0: 0000000000000a10 ffff0000089c8200 ffff8001bfececa0 000000000004094e
[ 375.080826] ebc0: 00000000000001c0 0000000000000000 0000000000000000 fffffffffffffe50
[ 375.088652] ebe0: ffff8001b8cbb758 ffff000009d11000 0000000000000002 0000000000000000
[ 375.096478] ec00: ffff0000094e8490 ffff00000952d000 ffff8001b8cdc000 ffff00000a272000
[ 375.104303] ec20: ffff8001b8cbb730 ffff0000095e4f00
[ 375.109191] [] rtl8168_interrupt+0x34/0x360 [r8168]
[ 375.115637] [] __handle_irq_event_percpu+0x5c/0x148
[ 375.122074] [] handle_irq_event_percpu+0x34/0x88
[ 375.128249] [] handle_irq_event+0x48/0x78
[ 375.133819] [] handle_simple_irq+0x9c/0xd0
[ 375.139475] [] generic_handle_irq+0x24/0x38
[ 375.145220] [] dw_handle_msi_irq+0xa4/0xe0
[ 375.150876] [] kirin_pcie_msi_irq_handler+0x10/0x18
[ 375.157313] [] __handle_irq_event_percpu+0x5c/0x148
[ 375.163749] [] handle_irq_event_percpu+0x34/0x88
[ 375.169925] [] handle_irq_event+0x48/0x78
[ 375.175494] [] handle_fasteoi_irq+0xd8/0x1c8
[ 375.181322] [] generic_handle_irq+0x24/0x38
[ 375.187064] [] __handle_domain_irq+0x84/0xf0
[ 375.192895] [] gic_handle_irq+0x54/0xa8
[ 375.198288] Exception stack(0xffff8001b8cdf8d0 to 0xffff8001b8cdfa00)
[ 375.204724] f8c0: 00000000000005dc 0000000000000000
[ 375.212550] f8e0: 00000000000000dc ffff8001acdfd8aa 0000000000000005 ffff8001acdfd896
[ 375.220376] f900: 0000000000000096 02ee6fc7dc1706c5 ffff0000094e8490 ffff00000952d000
[ 375.228203] f920: ffff8001b8cdc000 ffff00000a272000 ffff8001b8cbb730 ffff0000095e4f00
[ 375.236029] f940: 00000000fef34399 ffff8001b828e804 0000000000000623 0000000000000000
[ 375.243855] f960: 0000000000000000 ffff8001ae1eb800 ffff8001acdfd800 ffff0000094e8000
[ 375.251681] f980: ffff8001ae0c0000 ffff0000095e4000 ffff000008f13000 0000000000000096
[ 375.259508] f9a0: ffff8001acdfd896 ffff0000094f6638 ffff0000095fb6ad ffff8001b8cdfa00
[ 375.267334] f9c0: ffff000008a6031c ffff8001b8cdfa00 ffff000008a60410 0000000060400145
[ 375.275160] f9e0: ffff0000095e4000 ffff00000950c000 ffffffffffffffff ffff000008a6031c
[ 375.282988] [] el1_irq+0xe8/0x190
[ 375.287865] [] ip_rcv+0x268/0x590
[ 375.292740] [] __netif_receive_skb_core+0x3d0/0xc78
[ 375.299178] [] __netif_receive_skb+0x14/0x68
[ 375.305008] [] process_backlog+0xd8/0x230
[ 375.310576] [] net_rx_action+0x10c/0x2d8
[ 375.316057] [] __do_softirq+0x168/0x2a0
[ 375.321453] [] run_ksoftirqd+0x3c/0x68
[ 375.326763] [] smpboot_thread_fn+0x1dc/0x2e8
[ 375.332593] [] kthread+0x118/0x120
[ 375.337554] [] ret_from_fork+0x10/0x20
[ 375.342864] Code: f9458038 aa1603e0 95f5dcc9 9100fb17 (794002f9)
[ 375.348960] —[ end trace f9b013c43f2deaed ]—
[ 375.366114] Kernel panic - not syncing: Fatal exception in interrupt
[ 375.372466] SMP: stopping secondary CPUs
[ 375.897219] SMP: failed to stop secondary CPUs 0,2
[ 375.902006] Kernel Offset: disabled
[ 375.905489] Memory Limit: none
[ 375.921070] Rebooting in 5 seconds…
[ 380.924642] SMP: stopping secondary CPUs
[ 381.449392] SMP: failed to stop secondary CPUs 0,2

or rsyslogd (background process responsible for system log) - please check next post which contains the crash log.

Both crashes show that one of the CPUs was running code from r8168 but it has also failed with kirin_pcie_rd_own_conf (pcie-kirin driver).
It seems that these processes are trying to access the common areas (maybe IO or other memory areas) at the same time one of the drivers is running.
However, I still don’t understand how and why (I thought that the drivers were the only common access to the hardware).


#3

Rsyslogd related crash log:

[ 714.291654] Unhandled fault: synchronous external abort (0x96000210) at 0xffff0000121fe06c
[ 714.299944] Internal error: : 96000210 [#1] PREEMPT SMP
[ 714.305171] Modules linked in: r8168 ath10k_pci ath10k_core ath mac80211 cfg80211
[ 714.312719] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G S 4.9.78-147536-g7304d83b6c94-dirty #56
[ 714.322283] Hardware name: HiKey970 (DT)
[ 714.326204] task: ffff8001b8d15c00 task.stack: ffff8001b8d24000
[ 714.332156] PC is at rtl8168_xmii_link_ok+0x24/0x50 [r8168]
[ 714.337746] LR is at rtl8168_xmii_link_ok+0x1c/0x50 [r8168]
[ 714.343316] pc : [] lr : [] pstate: a04001c5
[ 714.350708] sp : ffff8001bff10d80
[ 714.354018] x29: ffff8001bff10d80 x28: 00000000000000a0
[ 714.359341] x27: 0000000000000005 x26: ffff0000094d5180
[ 714.364663] x25: ffff8001b8d24000 x24: 0000000000000140
[ 714.369984] x23: ffff8001ae33ed00 x22: ffff0000121fe000
[ 714.375305] x21: ffff8001ae338bd0 x20: ffff00000c2bdcf8
[ 714.380627] x19: ffff0000121fe06c x18: 0000000000000000
[ 714.385949] x17: 0000000000000000 x16: 0000000000000000
[ 714.391271] x15: 0000000000000000 x14: 0000000000000000
[ 714.396592] x13: 0000000000000000 x12: ffff8001b8d16530
[ 714.401913] x11: ffff00000a272000 x10: ffff8001b8d24000
[ 714.407235] x9 : ffff00000952d000 x8 : ffff0000094e8490
[ 714.412557] x7 : 0000000000000000 x6 : 0000000000000002
[ 714.417878] x5 : ffff000009d11000 x4 : ffff8001b8d16580
[ 714.423200] x3 : fffffffffffffe50 x2 : 0000000000000000
[ 714.428521] x1 : 00000000000001c0 x0 : 00000000000001c0
[ 714.433845]
[ 714.433845] SP: 0xffff8001bff10d00:
[ 714.438806] 0d00 [ 714.440554] 121fe000
ffff0000[ 714.443615] ae33ed00
ffff8001[ 714.446674] 00000140
00000000[ 714.449734] b8d24000
ffff8001[ 714.452794]
[ 714.454283] 0d20 [ 714.456029] 094d5180
ffff0000[ 714.459089] 00000005
00000000[ 714.462149] 000000a0
00000000[ 714.465209] bff10d80
ffff8001[ 714.468268]
[ 714.469757] 0d40 [ 714.471503] 00de821c
ffff0000[ 714.474563] bff10d80
ffff8001[ 714.477623] 00de8224
ffff0000[ 714.480683] a04001c5
00000000[ 714.483742]
[ 714.485230] 0d60 [ 714.486976] 121fe000
ffff0000[ 714.490037] 0c2bdcf8
ffff0000[ 714.493097] 00000000
00010000[ 714.496158] 00e4df8c
ffff0000[ 714.499218]
[ 714.500706] 0d80 [ 714.502452] bff10da0
ffff8001[ 714.505513] 00e4dfa0
ffff0000[ 714.508574] ae338b00
ffff8001[ 714.511635] ae338000
ffff8001[ 714.514695]
[ 714.516183] 0da0 [ 714.517929] bff10e00
ffff8001[ 714.520990] 0814b020
ffff0000[ 714.524051] 094e8000
ffff0000[ 714.527112] b8d24000
ffff8001[ 714.530172]
[ 714.531661] 0dc0 [ 714.533407] 00000101
00000000[ 714.536467] 00e4df58
ffff0000[ 714.539528] ae338000
ffff8001[ 714.542588] 00000200
dead0000[ 714.545648]
[ 714.547137] 0de0 [ 714.548884] b8d24000
ffff8001[ 714.551945] 0814b018
ffff0000[ 714.555005] 094e8000
ffff0000[ 714.558065] b8d24000
ffff8001[ 714.561126]
[ 714.562617]
[ 714.562617] X3: 0xfffffffffffffdd0:
[ 714.567579] fdd0 [ 714.569334] ********
********[ 714.572396] ********
********[ 714.575457] ********
********[ 714.578518] ********
********[ 714.581578]
[ 714.583067] fdf0 [ 714.584813] ********
********[ 714.587874] ********
********[ 714.590935] ********
********[ 714.593996] ********
********[ 714.597056]
[ 714.598546] fe10 [ 714.600292] ********
********[ 714.603353] ********
********[ 714.606415] ********
********[ 714.609476] ********
********[ 714.612536]
[ 714.614024] fe30 [ 714.615771] ********
********[ 714.618833] ********
********[ 714.621893] ********
********[ 714.624955] ********
********[ 714.628015]
[ 714.629503] fe50 [ 714.631250] ********
********[ 714.634312] ********
********[ 714.637373] ********
********[ 714.640433] ********
********[ 714.643493]
[ 714.644982] fe70 [ 714.646728] ********
********[ 714.649790] ********
********[ 714.652851] ********
********[ 714.655912] ********
********[ 714.658972]
[ 714.660460] fe90 [ 714.662208] ********
********[ 714.665269] ********
********[ 714.668330] ********
********[ 714.671390] ********
********[ 714.674450]
[ 714.675939] feb0 [ 714.677685] ********
********[ 714.680746] ********
********[ 714.683807] ********
********[ 714.686867] ********
********[ 714.689927]
[ 714.691417]
[ 714.691417] X4: 0xffff8001b8d16500:
[ 714.696378] 6500 [ 714.698125] 080ceffc
ffff0000[ 714.701186] 080ce770
ffff0000[ 714.704246] 00070c37
00070c36[ 714.707307] 00000000
00000001[ 714.710367]
[ 714.711855] 6520 [ 714.713602] 0c238c42
c03c3119[ 714.716663] 00000003
00000000[ 714.719724] 00000000
00000000[ 714.722785] 0814afa8
ffff0000[ 714.725845]
[ 714.727333] 6540 [ 714.729079] bff10e48
ffff8001[ 714.732140] 00000000
00000000[ 714.735201] 000426a4
00000000[ 714.738262] 571635ba
c47fa862[ 714.741322]
[ 714.742811] 6560 [ 714.744557] 00e4df8c
ffff0000[ 714.747618] ae338be8
ffff8001[ 714.750679] 00000000
00000000[ 714.753739] 000c269e
00000000[ 714.756800]
[ 714.758289] 6580 [ 714.760035] dcecb30a
5c69e25b[ 714.763096] 00de821c
ffff0000[ 714.766157] 0c2bdd10
ffff0000[ 714.769218] 00000000
00000000[ 714.772278]
[ 714.773767] 65a0 [ 714.775513] 000c237d
00000000[ 714.778574] b6a20b42
86a5255e[ 714.781635] 08120fac
ffff0000[ 714.784696] bfee5da0
ffff8001[ 714.787755]
[ 714.789244] 65c0 [ 714.790991] 00000000
00000000[ 714.794052] 000c22c5
00000000[ 714.797113] ca99ee77
9d50f3f8[ 714.800174] 08120f74
ffff0000[ 714.803234]
[ 714.804722] 65e0 [ 714.806468] 0950c0b8
ffff0000[ 714.809529] 00000000
00000000[ 714.812590] 000a2018
00000000[ 714.815651] 3ee874bd
bbf692a7[ 714.818711]
[ 714.820202]
[ 714.820202] X10: 0xffff8001b8d23f80:
[ 714.825250] 3f80 [ 714.826998] b8d20000
ffff8001[ 714.830059] 00000000
00000000[ 714.833118] 00000000
00000000[ 714.836180] 00000000
00000000[ 714.839240]
[ 714.840728] 3fa0 [ 714.842475] 00000000
00000000[ 714.845536] 00000000
00000000[ 714.848597] 00000000
00000000[ 714.851656] 00000000
00000000[ 714.854716]
[ 714.856204] 3fc0 [ 714.857952] 00000000
00000000[ 714.861012] 00bcb1a4
00000000[ 714.864073] 00000000
00000000[ 714.867134] 00000000
00000000[ 714.870193]
[ 714.871681] 3fe0 [ 714.873428] 00000000
00000000[ 714.876488] 00000000
00000000[ 714.879549] 7fffffff
fffff7b7[ 714.882610] 7fffffff
fefffbff[ 714.885669]
[ 714.887157] 4000 [ 714.888903] 00000000
00000000[ 714.891964] ffffffff
ffffffff[ 714.895025] b8d15c00
ffff8001[ 714.898085] 01618000
00000000[ 714.901145]
[ 714.902633] 4020 [ 714.904379] 00000104
00000003[ 714.907440] 57ac6e9d
00000000[ 714.910501] fdffbbfd
ffffffff[ 714.913562] ffffffff
feff7fbf[ 714.916622]
[ 714.918110] 4040 [ 714.919857] feffffff
fffffe7f[ 714.922918] fffffdff
ffd7ffff[ 714.925979] ffffbfff
fefbbf77[ 714.929040] ffffffff
ffffffff[ 714.932101]
[ 714.933589] 4060 [ 714.935336] 7eff7fff
ffffffff[ 714.938396] bfff7f7f
fbf7fdff[ 714.941456] ffffffff
ffffffff[ 714.944518] ffffffff
ffffffdf[ 714.947578]
[ 714.949068]
[ 714.949068] X12: 0xffff8001b8d164b0:
[ 714.954117] 64b0 [ 714.955863] 00000000
00000000[ 714.958924] 00000000
00000000[ 714.961985] 00000000
00000000[ 714.965046] 00000000
00000000[ 714.968106]
[ 714.969595] 64d0 [ 714.971340] 00000000
00000000[ 714.974401] 00070c3b
00000000[ 714.977462] 08bc51e4
ffff0000[ 714.980523] 08bc4f6c
ffff0000[ 714.983584]
[ 714.985072] 64f0 [ 714.986819] 00070c3a
00070c3b[ 714.989879] 00000000
00000000[ 714.992939] 080ceffc
ffff0000[ 714.996000] 080ce770
ffff0000[ 714.999060]
[ 715.000548] 6510 [ 715.002295] 00070c37
00070c36[ 715.005355] 00000000
00000001[ 715.008416] 0c238c42
c03c3119[ 715.011477] 00000003
00000000[ 715.014537]
[ 715.016026] 6530 [ 715.017773] 00000000
00000000[ 715.020834] 0814afa8
ffff0000[ 715.023895] bff10e48
ffff8001[ 715.026955] 00000000
00000000[ 715.030015]
[ 715.031503] 6550 [ 715.033250] 000426a4
00000000[ 715.036311] 571635ba
c47fa862[ 715.039372] 00e4df8c
ffff0000[ 715.042432] ae338be8
ffff8001[ 715.045492]
[ 715.046980] 6570 [ 715.048727] 00000000
00000000[ 715.051788] 000c269e
00000000[ 715.054849] dcecb30a
5c69e25b[ 715.057910] 00de821c
ffff0000[ 715.060970]
[ 715.062459] 6590 [ 715.064205] 0c2bdd10
ffff0000[ 715.067265] 00000000
00000000[ 715.070326] 000c237d
00000000[ 715.073387] b6a20b42
86a5255e[ 715.076447]
[ 715.077938]
[ 715.077938] X21: 0xffff8001ae338b50:
[ 715.082987] 8b50 [ 715.084734] 00000000
00000000[ 715.087795] 00000000
00000000[ 715.090855] 00000000
00000000[ 715.093916] 00000000
00000000[ 715.096975]
[ 715.098464] 8b70 [ 715.100210] 00000000
00000000[ 715.103271] 00000000
00000000[ 715.106332] 00000000
00000000[ 715.109392] 00000000
00000000[ 715.112452]
[ 715.113941] 8b90 [ 715.115687] 00000000
00000000[ 715.118748] 00000000
00000000[ 715.121808] 00000000
00000000[ 715.124870] 00000000
00000000[ 715.127929]
[ 715.129418] 8bb0 [ 715.131164] 00000000
00000000[ 715.134225] 00000000
00000000[ 715.137286] 00000000
00000000[ 715.140346] 00000000
00000000[ 715.143406]
[ 715.144894] 8bd0 [ 715.146641] 41494148
dead4ead[ 715.149701] 00000003
00000000[ 715.152762] b8d15c00
ffff8001[ 715.155823] 00e60620
ffff0000[ 715.158883]
[ 715.160371] 8bf0 [ 715.162117] 09dc4848
ffff0000[ 715.165178] 00000000
00000000[ 715.168238] 00e5ecd8
ffff0000[ 715.171299] 00070007
dead4ead[ 715.174360]
[ 715.175848] 8c10 [ 715.177595] ffffffff
00000000[ 715.180655] ffffffff
ffffffff[ 715.183716] 00e60628
ffff0000[ 715.186776] 09dc4d58
ffff0000[ 715.189837]
[ 715.191325] 8c30 [ 715.193072] 00000000
00000000[ 715.196134] 00e5ecf0
ffff0000[ 715.199195] 0000ffff
40000000[ 715.202255] 80000000
20000000[ 715.205315]
[ 715.206806]
[ 715.206806] X23: 0xffff8001ae33ec80:
[ 715.211854] ec80 [ 715.213602] 00000000
00000000[ 715.216663] 00000000
00000000[ 715.219723] 000005f3
00000000[ 715.222784] 00000000
00000000[ 715.225843]
[ 715.227331] eca0 [ 715.229078] 00000000
00000000[ 715.232139] 00000000
00000000[ 715.235200] 00000000
00000000[ 715.238260] 00000000
00000000[ 715.241320]
[ 715.242809] ecc0 [ 715.244556] 00000000
00000000[ 715.247616] 00000000
00000000[ 715.250677] 00000000
00000000[ 715.253738] 00000000
00000000[ 715.256798]
[ 715.258286] ece0 [ 715.260032] 00000000
00000000[ 715.263093] 00000000
00000000[ 715.266154] 00000000
00000000[ 715.269216] 00000000
00000000[ 715.272276]
[ 715.273764] ed00 [ 715.275510] 00000200
dead0000[ 715.278571] 00000000
00000000[ 715.281631] 00019442
00000001[ 715.284692] 00e4df58
ffff0000[ 715.287751]
[ 715.289240] ed20 [ 715.290986] ae338000
ffff8001[ 715.294047] 12400003
ffffffff[ 715.297108] 00000000
00000000[ 715.300168] 00000000
00000000[ 715.303228]
[ 715.304717] ed40 [ 715.306463] 00000000
00000000[ 715.309524] 00e60618
ffff0000[ 715.312585] 00000000
00000000[ 715.315646] 00000000
00000000[ 715.318706]
[ 715.320195] ed60 [ 715.321941] 00e5eb08
ffff0000[ 715.325002] 00000006
01200001[ 715.328063] 00ff0004
00000000[ 715.331124] 000c0140
00000000[ 715.334184]
[ 715.335674]
[ 715.335674] X25: 0xffff8001b8d23f80:
[ 715.340723] 3f80 [ 715.342469] b8d20000
ffff8001[ 715.345529] 00000000
00000000[ 715.348590] 00000000
00000000[ 715.351651] 00000000
00000000[ 715.354711]
[ 715.356199] 3fa0 [ 715.357946] 00000000
00000000[ 715.361006] 00000000
00000000[ 715.364067] 00000000
00000000[ 715.367128] 00000000
00000000[ 715.370188]
[ 715.371676] 3fc0 [ 715.373423] 00000000
00000000[ 715.376484] 00bcb1a4
00000000[ 715.379544] 00000000
00000000[ 715.382605] 00000000
00000000[ 715.385665]
[ 715.387153] 3fe0 [ 715.388900] 00000000
00000000[ 715.391961] 00000000
00000000[ 715.395021] 7fffffff
fffff7b7[ 715.398082] 7fffffff
fefffbff[ 715.401142]
[ 715.402630] 4000 [ 715.404377] 00000000
00000000[ 715.407437] ffffffff
ffffffff[ 715.410498] b8d15c00
ffff8001[ 715.413559] 01618000
00000000[ 715.416619]
[ 715.418108] 4020 [ 715.419853] 00000104
00000003[ 715.422914] 57ac6e9d
00000000[ 715.425974] fdffbbfd
ffffffff[ 715.429035] ffffffff
feff7fbf[ 715.432095]
[ 715.433583] 4040 [ 715.435329] feffffff
fffffe7f[ 715.438390] fffffdff
ffd7ffff[ 715.441451] ffffbfff
fefbbf77[ 715.444512] ffffffff
ffffffff[ 715.447573]
[ 715.449061] 4060 [ 715.450808] 7eff7fff
ffffffff[ 715.453868] bfff7f7f
fbf7fdff[ 715.456929] ffffffff
ffffffff[ 715.459990] ffffffff
ffffffdf[ 715.463050]
[ 715.464540]
[ 715.464540] X29: 0xffff8001bff10d00:
[ 715.469589] 0d00 [ 715.471336] 121fe000
ffff0000[ 715.474396] ae33ed00
ffff8001[ 715.477457] 00000140
00000000[ 715.480518] b8d24000
ffff8001[ 715.483579]
[ 715.485068] 0d20 [ 715.486814] 094d5180
ffff0000[ 715.489875] 00000005
00000000[ 715.492936] 000000a0
00000000[ 715.495997] bff10d80
ffff8001[ 715.499056]
[ 715.500545] 0d40 [ 715.502292] 00de821c
ffff0000[ 715.505352] bff10d80
ffff8001[ 715.508413] 00de8224
ffff0000[ 715.511474] a04001c5
00000000[ 715.514534]
[ 715.516023] 0d60 [ 715.517769] 121fe000
ffff0000[ 715.520829] 0c2bdcf8
ffff0000[ 715.523891] 00000000
00010000[ 715.526952] 00e4df8c
ffff0000[ 715.530012]
[ 715.531501] 0d80 [ 715.533247] bff10da0
ffff8001[ 715.536308] 00e4dfa0
ffff0000[ 715.539370] ae338b00
ffff8001[ 715.542431] ae338000
ffff8001[ 715.545492]
[ 715.546980] 0da0 [ 715.548727] bff10e00
ffff8001[ 715.551788] 0814b020
ffff0000[ 715.554849] 094e8000
ffff0000[ 715.557910] b8d24000
ffff8001[ 715.560970]
[ 715.562459] 0dc0 [ 715.564205] 00000101
00000000[ 715.567266] 00e4df58
ffff0000[ 715.570327] ae338000
ffff8001[ 715.573387] 00000200
dead0000[ 715.576448]
[ 715.577936] 0de0 [ 715.579683] b8d24000
ffff8001[ 715.582744] 0814b018
ffff0000[ 715.585804] 094e8000
ffff0000[ 715.588865] b8d24000
ffff8001[ 715.591925]
[ 715.593413]
[ 715.594902] Process swapper/3 (pid: 0, stack limit = 0xffff8001b8d24028)
[ 715.601602] Stack: (0xffff8001bff10d80 to 0xffff8001b8d28000)
[ 715.607347] Call trace:
[ 715.609791] Exception stack(0xffff8001bff10b90 to 0xffff8001bff10cc0)
[ 715.616232] 0b80: ffff0000121fe06c 0001000000000000
[ 715.624063] 0ba0: 0000000001618000 ffff000000de8224 00000000a04001c5 00000000000001c0
[ 715.631893] 0bc0: ffff800100000000 0000000000000007 ffff800100000000 ffff0000121fe06c
[ 715.639724] 0be0: ffff8001b8d24000 00000000000001c0 ffff8001bff10c40 ffff000008128164
[ 715.647554] 0c00: 000000000000037d ffff0000094e8000 ffff8001b8d15c00 0000000000000000
[ 715.655384] 0c20: 000000000000069e ffff0000094e8000 ffff8001b8d15c00 0000000000000000
[ 715.663214] 0c40: ffff8001bff10d30 000000000004094e 00000000000001c0 00000000000001c0
[ 715.671045] 0c60: 0000000000000000 fffffffffffffe50 ffff8001b8d16580 ffff000009d11000
[ 715.678875] 0c80: 0000000000000002 0000000000000000 ffff0000094e8490 ffff00000952d000
[ 715.686705] 0ca0: ffff8001b8d24000 ffff00000a272000 ffff8001b8d16530 0000000000000000
[ 715.694555] [] rtl8168_xmii_link_ok+0x24/0x50 [r8168]
[ 715.701187] [] rtl8168_link_timer+0x48/0x740 [r8168]
[ 715.707728] [] call_timer_fn+0x78/0x100
[ 715.713127] [] expire_timers+0xb0/0xd8
[ 715.718438] [] run_timer_softirq+0xfc/0x1a0
[ 715.724189] [] __do_softirq+0x168/0x2a0
[ 715.729591] [] irq_exit+0xa4/0xf8
[ 715.734471] [] handle_IPI+0x138/0x180
[ 715.739695] [] gic_handle_irq+0xa0/0xa8
[ 715.745091] Exception stack(0xffff8001b8d27dd0 to 0xffff8001b8d27f00)
[ 715.751531] 7dc0: ffff8001b8d15c00 0000000000000001
[ 715.759362] 7de0: ffff8001b8d24000 0000000000000000 0000000000000000 0000000000000001
[ 715.767192] 7e00: 0000000000000002 0000000000000000 ffff0000094e8490 ffff00000952d000
[ 715.775022] 7e20: ffff8001b8d24000 ffff00000a272000 ffff8001b8d16530 0000000000000000
[ 715.782852] 7e40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 715.790681] 7e60: 0000000000000000 000000a64b5f4723 ffff8001bff170e0 0000000000000002
[ 715.798513] 7e80: ffff0000095cdc88 0000000000000002 000000a610f7f61c ffff000009039910
[ 715.806343] 7ea0: ffff0000095fb1e3 0000000000000001 ffff0000094e8000 ffff8001b8d27f00
[ 715.814173] 7ec0: ffff00000889be84 ffff8001b8d27f00 ffff00000889be88 0000000060400105
[ 715.822004] 7ee0: 0000000000000002 ffff00000889bf40 ffffffffffffffff ffff00000889be84
[ 715.829834] [] el1_irq+0xe8/0x190
[ 715.834718] [] cpuidle_enter_state+0x148/0x250
[ 715.840724] [] cpuidle_enter+0x18/0x20
[ 715.846044] [] call_cpuidle+0x1c/0x40
[ 715.851270] [] cpu_startup_entry+0x150/0x1e8
[ 715.857102] [] secondary_start_kernel+0x13c/0x1d8
[ 715.863368] [<0000000000bcb1a4>] 0xbcb1a4
[ 715.867381] Code: aa1403e0 95f7734c 9101b273 aa0003e1 (39400273)
[ 715.873566] —[ end trace 8aac137137942685 ]—
[ 715.877230] Bad mode in Error handler detected on CPU2, code 0xbf000002 – SError
[ 715.877235] Internal error: Oops - bad mode: 0 [#2] PREEMPT SMP
[ 715.877252] Modules linked in: r8168 ath10k_pci ath10k_core ath mac80211 cfg80211
[ 715.877259] CPU: 2 PID: 3118 Comm: rs:main Q:Reg Tainted: G S D 4.9.78-147536-g7304d83b6c94-dirty #56
[ 715.877261] Hardware name: HiKey970 (DT)
[ 715.877264] task: ffff8001b2e88000 task.stack: ffff8001b348c000
[ 715.877268] PC is at 0xffffa4848230
[ 715.877271] LR is at 0xffffa4848210
[ 715.877274] pc : [<0000ffffa4848230>] lr : [<0000ffffa4848210>] pstate: 80000000
[ 715.877276] sp : 0000ffffa3ae7400
[ 715.877280] x29: 0000ffffa3ae7400 x28: 0000aaaaefab4d48
[ 715.877286] x27: 0000000000000000 x26: 0000aaaaefab4d70
[ 715.877290] x25: 000000000000023a x24: 0000000000000000
[ 715.877295] x23: 0000ffffa4847f40 x22: 0000000000000474
[ 715.877300] x21: 0000aaaaefab0ca0 x20: 0000ffffa4861000
[ 715.877304] x19: 0000aaaaefab4d58 x18: 0000ffffa4757a70
[ 715.877309] x17: 0000ffffa4847ff8 x16: 0000aaaabd0ec830
[ 715.877313] x15: 0000000000000000 x14: 0000000000000000
[ 715.877318] x13: 54203a726f736e65 x12: 73742e3030303033
[ 715.877323] x11: 0a3030303536203c x10: 0000000000000000
[ 715.877328] x9 : 0000000000000000 x8 : 0000000000000062
[ 715.877333] x7 : 0000000000000001 x6 : 0000000000000000
[ 715.877337] x5 : 0000ffffa3ae8000 x4 : 00000000ffffffbb
[ 715.877341] x3 : 0000000000000000 x2 : 0000000000000000
[ 715.877346] x1 : 0000000000000080 x0 : 0000000000000000
[ 715.877347]
[ 715.877350] Process rs:main Q:Reg (pid: 3118, stack limit = 0xffff8001b348c028)
[ 715.877354] —[ end trace 8aac137137942686 ]—
[ 716.053945] Kernel panic - not syncing: Fatal exception in interrupt
[ 716.060306] SMP: stopping secondary CPUs
[ 716.585060] SMP: failed to stop secondary CPUs 0,3
[ 716.589847] Kernel Offset: disabled
[ 716.593330] Memory Limit: none
[ 716.608864] Rebooting in 5 seconds…
[ 721.612434] SMP: stopping secondary CPUs
[ 722.137184] SMP: failed to stop secondary CPUs 0,3