Check what changes in lspci command between not having the GPU connected vs. Having it connected.

I am suspecting that your PCI-E bandwidth is getting exhausted once the kernel activates your GPU.

Edit: Although I could be wrong about this. So makes sense to try passing “nomodeset” to your kernel parameters and see if that changes anything.