Recently I have been able to return to my investigations of using kexec on the Dragonboard 820c.
As the time interval from my last post in Kexec hang still not resolved - #14 by kldixon is so long, I thought I should start a new thread.
./Documentation/PCI/pci.rst opens with:
“The world of PCI is vast and full of (mostly unpleasant) surprises.”
I have come to appreciate, all too well, the accuracy of that statement.
I put the following pr_debugs into ./drivers/pci/controller/dwc/pcie-designware-host.c:dw_pcie_access_other_conf
static int dw_pcie_access_other_conf(struct pcie_port *pp, struct pci_bus *bus,
u32 devfn, int where, int size, u32 *val,
bool write)
{
int ret, type;
u32 busdev, cfg_size;
u64 cpu_addr;
void __iomem *va_cfg_base;
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
pr_debug("%s 0 %d %d %d\n", __func__, bus->parent->number, pp->root_bus_nr, where);
busdev = PCIE_ATU_BUS(bus->number) | PCIE_ATU_DEV(PCI_SLOT(devfn)) |
PCIE_ATU_FUNC(PCI_FUNC(devfn));
if (bus->parent->number == pp->root_bus_nr) {
type = PCIE_ATU_TYPE_CFG0;
cpu_addr = pp->cfg0_base;
cfg_size = pp->cfg0_size;
va_cfg_base = pp->va_cfg0_base;
} else {
type = PCIE_ATU_TYPE_CFG1;
cpu_addr = pp->cfg1_base;
cfg_size = pp->cfg1_size;
va_cfg_base = pp->va_cfg1_base;
}
pr_debug("%s 1 %d %LX %x %x %px\n", __func__, type, cpu_addr, busdev, cfg_size, va_cfg_base);
dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX1,
type, cpu_addr,
busdev, cfg_size);
if (write)
ret = dw_pcie_write(va_cfg_base + where, size, *val);
else
ret = dw_pcie_read(va_cfg_base + where, size, val);
pr_debug("%s 2 %x %d %LX %llx %x\n", __func__, *val, pci->num_viewport, pp->io_base, pp->io_bus_addr, pp->io_size);
if (pci->num_viewport <= 2)
dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX1,
PCIE_ATU_TYPE_IO, pp->io_base,
pp->io_bus_addr, pp->io_size);
return ret;
}Note the the value of ‘write’ is false and ‘size’ is 4, as dw_pcie_access_other_conf is called via pci_bus_read_config_dword.
Also note:
./drivers/pci/controller/dwc/pcie-designware.h:
#define PCIE_ATU_TYPE_CFG0 0x4and bus->number = 1 and devfn = 0.
The result was, for the first kernel:
[ 13.788446] dw_pcie_access_other_conf 0 0 0 0
[ 13.792185] dw_pcie_access_other_conf 1 4 D100000 1000000 80000 ffff800013300000
[ 13.796741] dw_pcie_access_other_conf 2 6121b21 2 D200000 d200000 100000
[ 26.610043] dw_pcie_access_other_conf 0 0 0 0
[ 26.613783] dw_pcie_access_other_conf 1 4 E100000 1000000 80000 ffff800013500000
[ 26.618342] dw_pcie_access_other_conf 2 10831969 2 E200000 e200000 100000
[ 46.934350] dw_pcie_access_other_conf 0 0 0 0
[ 46.938093] dw_pcie_access_other_conf 1 4 C100000 1000000 80000 ffff800014880000
[ 46.942660] dw_pcie_access_other_conf 2 3e168c 2 C200000 c200000 100000and for the second, (kexec’d), kernel:
[ 13.583168] dw_pcie_access_other_conf 0 0 0 0
[ 13.586907] dw_pcie_access_other_conf 1 4 D100000 1000000 80000 ffff800013300000
[ 13.591458] dw_pcie_access_other_conf 2 ffffffff 2 D200000 d200000 100000
[ 19.668798] dw_pcie_access_other_conf 0 0 0 0
[ 19.672537] dw_pcie_access_other_conf 1 4 E100000 1000000 80000 ffff800013500000
[ 19.677091] dw_pcie_access_other_conf 2 ffffffff 2 E200000 e200000 100000
[ 30.819566] dw_pcie_access_other_conf 0 0 0 0
[ 30.823305] dw_pcie_access_other_conf 1 4 C100000 1000000 80000 ffff800013e80000
[ 30.827849] dw_pcie_access_other_conf 2 3e168c 2 C200000 c200000 100000with
root@linaro-alip:~# lspci -vvvxxxx
producing the following lines showing the three PCIe endpoints and the first 16 bytes of their corresponding configuration spaces:
...
0000:01:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
...
00: 8c 16 3e 00 06 04 10 00 32 00 80 02 00 00 00 00
...
0001:01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
...
00: 21 1b 12 06 07 04 10 00 01 01 06 01 00 00 00 00
...
0002:01:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
...
00: 69 19 83 10 07 00 10 00 c0 00 00 02 00 00 00 00
...The virtual addresses, va_cfg_base, vary from boot instance to boot instance.
The device tree ‘reg’ entries for the configuration address spaces of the pcie nodes in ./arch/arm64/boot/dts/qcom/msm8998.dtsi are:
...
reg = <0x0c100000 0x100000>;
...
reg = <0x0d100000 0x100000>;
...
reg = <0x0e100000 0x100000>;In ./drivers/pci/controller/dwc/pcie-designware-host.c:dw_pcie_host_init, cfg0_size is set by:
cfg_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
if (cfg_res) {
pp->cfg0_size = resource_size(cfg_res) >> 1;and is then used to set va_cfg0_base:
if (!pp->va_cfg0_base) {
pp->va_cfg0_base = devm_pci_remap_cfgspace(dev,
pp->cfg0_base, pp->cfg0_size);So everything looks ok except for the the value of ffffffff being read from the first four bytes of the configuration space of the ASM1062 and AR8151 in the second kernel.
One difference between the QCA6174 and the other two endpoints is that the configuration space of the QCA6174 is read after ath10k_pci_init, as the probe is deferred, but the configuration spaces of the other endpoints are read before ahci_pci_driver_init and atl1c_driver_init.
Another difference is that the 4k of bridge memory for the QCA6174 is below the endpoint memory but above the endpoint memory for the ASM1062 and AR8151.
And yet another difference is that ‘I/O behind bridge’ reported by lspci is disabled for the QCA6174 but not for the other two endpoints.
Even a fundamental warm reset fails to make any difference.
I tried including the following shutdown method in qcom_pcie_driver:
static void qcom_pcie_shutdown(struct platform_device *pdev)
{
struct qcom_pcie *pcie = platform_get_drvdata(pdev);
struct dw_pcie *pci = pcie->pci;
struct pcie_port *pp = &pci->pp;
int ret;
ret = qcom_pcie_host_init(pp);
}I put pr_debugs in qcom_pcie_host_init, but, although it reported as expected and returned 0, it made no difference.
As I understand it, this gives a fundamental reset to the endpoint and sets up the root complex port via dw_pcie_setup_rc.
I have also tried using ahci_remove_one and atl1c_remove instead of ahci_shutdown_one and atl1c_shutdown as shutdown methods but, again, no difference.
Is it more likely that the ATU is misbehaving? If so, what can I do about it?
Or is the firmware interfering? As far as I can see, LK does not touch PCI but perhaps xbl.elf, tz.mbn, or rpm.mbn does?
I also do not understand why PCIE_ATU_TYPE_IO is being programmed? Why not PCIE_ATU_TYPE_MEM as well?
