As far as the physical you can either get a riser or just Dremel the plastic back stop off (though you'd still need to figure out physical support for a larger device). Many short PCIe slots come with an open back out of the box just for this purpose, I'm surprised the Pi (for all it's hacking purpose) doesn't. Power is also going to be another thing, PCIe is supposed to provide up to 75 Watts. There are solutions for this as well, particularly if you went with a "bitcoin mining" type riser in the earlier step.
Plenty of people have done GPUs on https://www.pine64.org/rockpro64/