PMU registers get reset for 410c with Debian 144

Products Support DragonBoard410c
#1

Hi,

I use PMU counter register to benchmark the performance of some functions. I have a DragonBoard 410C and write a kernel module to set PMUSERENR_EL0 to 0xF for user mode access. I had no problem with Debian release 36.
Recently I got a new 410c board and load Debian release 144. Then the problem shows up - the register PMUSERENR_EL0 got reset to 0 quickly after it is set by the kernel module. So the PMU registers are not accessible from user mode.
Does anyone know what reset PMU registers?

Best regards,
Eugene

#2

I tried a few more things today. I opened a LX terminal to “insmod” the kernel mode to enable user mode access by setting “PMUSERENR_EL0” and “PMCR_EL0”, then use a while loop to check if “PMUSERENR_EL0” is set. If it is set, it will exit the while loop and print the information about how long it takes. Interestingly it never exits the while loop since the “insmod” is always hanging there.

Then, in the meanwhile, I open another LX terminal to run a program to read “pmccentr0_el0”, but it still shows illegal instructions.

Please note I have no problem to run the same thing with Debian release 36.

Can anyone help with this issue?

Best,
Eugene

#3

could you check if the kernel corresponding to Debian release 36 contains a pmu/cpu-pmu node in the device tree like we have now?

see below:
https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/tree/arch/arm64/boot/dts/qcom/msm8916.dtsi?h=release/qcomlt-4.9#n202

#4

if you are not sure you can probably check /proc/device-tree.

Or if you want to do a reverse recompilation of the device tree do:

$ apt-get update
$ apt-get install device-tree-compiler
$ dtc -I fs -O dts /proc/device-tree -o file.dts

Then you can inspect the file.dts with an editor.

#5

Hi Jorge,

Thank you very much for the instructions. I followed them and generated file.dts. It contains only cpu-pmu node in the device tree. Please see below. It is different from what we have now for release 144. Could you please advise how we can get the pmu back to work?

Best regards,
Eugene

cpu-pmu {
	interrupts = <0x1 0x7 0xf00>;
	compatible = "arm,armv8-pmuv3";
};
#6

I am a bit surprised - I was counting on the kernel not to be controlling the PMU so you -as a user- could access its registers from user space instead of using the recommended way (ie perf).

Could you validate perf in your #36 release?
If you are cross-compiling your kernel you could just do

server$ cd $KERNEL_TREE
server$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- tools/perf

then copy tools/perf/perf to your db410c.

Once done please execute as follows:

  • terminal_1: keep perf busy using a high count
    db410c$ perf stat -e cycles dd if=/dev/zero of=/dev/null count=10000000

*terminal_2: check if perf is receiving interrupts
db410c$ cat /proc/interrupts | grep arm-pmu

If perf is indeed working on #36 (which I doubt since I would expect perf to interfere with your module) you should see an “arm-pmu” interrupt registered in terminal 2 (while terminal_1 is busy).

You could then reduce the count given on terminal_1 to get a valid output quicker on terminal 1.


root@linaro-alip:/home/linaro# perf stat -e cycles dd if=/dev/zero of=/dev/null count=10000
10000+0 records in
10000+0 records out
5120000 bytes (5.1 MB) copied, 0.0816867 s, 62.7 MB/s

 Performance counter stats for 'dd if=/dev/zero of=/dev/null count=10000':

          17987875      cycles                                                      

       0.092714040 seconds time elapsed

In any case, maybe you should try to recompile the #144 kernel without PMU support (remove the device tree entry for instance or disable it from the kernel config); I would then expect userspace to have complete access to the hardware and not collide with the kernel PMU initialization.

Either way my advice would be to drop your current performance measurement system and write some macros/functions around trace/perf (a first class Linux subsystem) and then use this tool to access the PMU registers that you need.

#7

Thank you very much, Jorge. I followed your instructions and did see the interrupts from CPU-PMU. The same benchmarking with count=10000 took twice of time in my 410c with debian #36.

My question is do you plan to remove PMU support for next release and when will be the next release?

Following your advice, could you please point me to some sample code to use perf?

Best regards,
Eugene

#8

Sorry I might have confused you.

PMU support will not be removed (it is here to stay); what I suspect is that the support we had on #36 was buggy and this allowed you to run your module.

If you disable cpu-pmu from the new release that you are trying to use I believe your module will work again.

If you are not sure about how to rebuild and re-install the kernel image, there are instructions on the release page on how to do this from sources:
http://builds.96boards.org/releases/dragonboard410c/linaro/debian/16.09/

This is the perf tutorial (you probably know this already)
https://perf.wiki.kernel.org/index.php/Tutorial

And this is an interesting article on how to integrate kernel tracing support in your software.

#9

Thanks again for all the information, Jorge. I really appreciate it!

Could you please give a little more details about why the following pmu code in the device tree of 144# could cause pmu registers inaccessible from user mode?

pmu {
	compatible = "arm,armv8-pmuv3";
	interrupts = <GIC_PPI 7 GIC_CPU_MASK_SIMPLE(4)>;
};

As you said in one previous post, this code should not prevent user mode from accessing these registers. So could there be a bug somewhere in this part? Or it should be defined as cpu-pmu as in #36?

#10

Hi Eugene,

I am sorry, maybe I wasn’t clear.

I don’t know how your module is controlling/accessing the PMU but what I suspect there is an interaction with the kernel’s PERF framework.

If your module uses the kernel’s PMU support, then try to disable perf from your kernel configuration file (so the perf framework doesn’t touch any PMU registers).
The configuration knob is CONFIG_PERF_EVENTS under General Setup. Unselect this option, recompile, install the new kernel and test.

If your module doesn’t use the kernel’s PMU support, then remove the pmu-cpu node from the device tree, recompile, install and test.

does this make sense?

#11

Thank you very much for the information, Jorge!

Today I followed the steps to generate the file.dts for #144, which surprisingly has the same settings for cpu-pmu as that for #36. If I run the same test with perf, it also shows the interrupts from cpu-pmu. So I think perf and cpu-pmu might be ok for #144?

What could be other things which prevent the access of pmu cycle counter registers, or reset the “PMUSERENR_EL0” register?

#12

To use your own performance tools to access the PMU, disable perf from the kernel.

  1. configure the kernel (make menuconfig …) disabling CONFIG_PERF_EVENTS.
  2. then build and re-install.

16.09 instructions on building from source.
http://builds.96boards.org/releases/dragonboard410c/linaro/debian/16.09/

When you test, make sure you installed the right kernel with the right configs

ie:

zcat /proc/config.gz | grep PERF

should NOT show:
CONFIG_PERF_EVENTS=y

#13

FWIW, I’m also successfully accessing the PMU registers from userspace with a Debian 83 based install. So whatever changed was somewhere between Debian 83 and Debian 114, that is, between debian-qcom-dragonboard410c-16.04 and debian-qcom-dragonboard410c-16.09.

I didn’t see any conclusion on this thread here, but I ran into the same issue on another board though, where I’ve debugged the issue a bit further and reached some sort of solution at least.

I tested disabling CONFIG_PERF_EVENTS, but that didn’t have any effect on the PMU register accessibility at all.

In https://devtalk.nvidia.com/default/topic/955554/jetson-tx1/performance-counters-reset-itself/ it was suggested that CONFIG_CPU_IDLE could be related to this. By disabling that config, I’m able to set pmuserenr_el0 via a kernel module just as on the Dragonboard. And in that setup, profiling with perf works just fine as well - I just have to reenable the counters (which can be done all in userspace by setting pmcr_el0 and pmcntenset_el0) after profiling with perf.

The Dragonboard 410c Debian 83 kernel also does have CONFIG_CPU_IDLE enabled though, so I don’t really know what differs between that and the newer Debian 114 kernel (or my newer devboard for that matter) though, why it didn’t clobber pmuserenr_el0 before but it does now.

As another remark, I tried editing reset_pmuserenr_el0 which is called from cpu_do_resume (in arch/arm64/mm/proc.S) to enable it again on resume, in order not to have to disable CONFIG_CPU_IDLE, by resetting the right bits in pmuserenr_el0 instead of clearing them. With that, I was able to access the registers from userspace, but I wasn’t able to do any benchmarking in userspace as the counter always ended up disabled all the time.