some drivers are built in, and some drivers are built as modules. the kernel will fail to load modules which aren't built from the same tree / version.
the first experiment you've done is expected. if you built a different kernel, you need to build the modules as well, and deploy them in the rootfs in /lib/modules. Unfortunately the modules can't go into the boot image, so it makes the installation a bit more cumbersome.
after compiling the kernel , compile the modules as well:
then to copy the modules from host PC to the board, one option is to use a USB stick, on the PC
make modules_install INSTALL_MOD_PATH=<path to USB drive>
then boot the board and mount the USB driver, and copy the modules from USB into /lib/modules.. then reboot
You can use SD card as well..
For the second experiment, it should have worked. I am tempted to believe that you did not set "KERNELRELEASE=4.4.23-linaro-lt-qcom" when compiling the kernel, could that be it?