This is more of a note to myself, as I keep forgetting this. The propriety NVIDIA drivers, provided by the x11-drivers/nvidia-drivers dislikes alternatives. It will refuse to build against a kernel with the rivafb (CONFIG_FB_RIVA) and nvidiafb (CONFIG_FB_NVIDIA) built in or built as modules. Both can be found (and unset) under:
Device Drivers
-> Graphics support
-> nVidia Framebuffer Support
-> nVidia Riva support