hi @sabjorn, what you want to do is then not possible. You can't mix and match official Debian binary packages and OE images. You really need to pick the distro you want to use and stick to it. Both Debian or OE based distro have their own pros and cons..
However if managing the kernel is the only issue you have with Debian , then you might want to investigate how to get your own Debian image with your custom kernel. It is not that difficult to do. Our own Debian images use a custom kernel (the LT kernel), and the process of building a Debian image with a custom kernel can be automated, for example, this is how we build our images:
I admit it is not trivial, but it is not that difficult (and we can help you if needed). I know for sure others have been able to setup their own build.
For development/local build, you don't need to recreate the entire image each time, I personally build the .deb package using the in-kernel Makefiles (make bindeb-pkg) and deploy the .deb on the board.