The Android SPI driver is very different from the Debian SPI Driver, the issues in the other thread only apply to the Debian builds. I haven't tried out the Android SPI drivers so I am not really familiar with their operation, but I have a guess.
I suspect your problem is related to the fact that the you have a very light processing load and the CPU(s) are being clocked very slowly. Hence it takes a long time to get in and out of the kernel and through all of the layers of device driver.
Have you tried speeding up the CPUs? The Processor Governor Guide will give you some hints on how to speed up the CPUs. https://developer.qualcomm.com/download/db410c/processor-governor-guide-qualcomm-snapdragon-410.pdf