*Fitting the GPU*
A normal GPU, mounted in the normal way (perpendicular to the motherboard), will not fit in a 2U case.
It looks like Jeff's solution was to buy a half-height GPU. Neat! I didn't actually realize those exist. But it looks like NVidia's regular consumer-oriented GPUs don't come in this form factor; Jeff instead went with an RTX A2000, which is part of NVidia's "professional" GPU line. The problem with this is that it's way more expensive to achieve the same gaming performance.
In my previous house I solved the problem by using 3U chassis. This provides just enough space to mount the GPU perpendicular. However, it still had problems. Modern GPUs require supplementary power direct from the PSU. The socket for this usually points outward, away from the motherboard. In a 3U chassis, this socket is right up against the top plate so there is no room to plug in the power. Sometimes you can find a GPU that has the power socket pointing in a different direction, but they have become rare. I actually had to physically mod ten GTX 1060's by hand, cutting away the plastic socket housing and directly attaching each power wire to the corresponding pin. Big PITA but it worked.
In the new house I'm going with 2U, but using a riser card. So, the GPU is mounted parallel to the motherboard. This seems to work OK. The big problem is ensuring the chassis and the motherboard both have their expansion slots aligned right for this. Most 2U chassis are designed for perpendicular, half-height expansion cards. A smaller fraction support full-height parallel cards on a riser. The motherboard also has to have a PCIe slot in the correct place for the riser. This is usually the "top" slot, closest to the CPU, and I've noticed a lot of motherboards these days are opting to omit that slot to make more space around the CPU.
The chassis I'll probably go with is the iStarUSA D-200: https://www.istarusa.com/en/istarusa/products.php?model=D-20...
(I'm not too excited about it though, and would be happy for other recommendations.)
At least one motherboard that worked when I tried it was the ASUS PRIME B660-PLUS, but I haven't made a final motherborad selection yet.
A further problem which might be specific to this chasis is that the total horizontal space between the riser card and the power supply is tight. Some GPUs are excessively "tall" and won't fit. (The same problem probably applies to 3U chassis with perpendicular mounting.) I was just barely able to fit the EVGA GeForce RTX 3070 Ti 08G-P5-3785-K into the space.
*RAM orientation*
Servers orient their RAM modules parallel to the direction of air current, that is, perpendicular to the back of the chasis. For reasons I don't understand, consumer / gaming motherboards almost never do this, they orient the RAM parallel to the back. Maybe this is because desktop chassis are not usually optimized for efficient airflow anyway so it doesn't matter? And there's more space on the board if the RAM can be mounted this way? I dunno.
Anyway, in my experiments so far this has just not caused a problem. Yeah, the RAM has non-ideal orientation, but there haven't been any issues with overheating. ::shrug::
*PSU*
Servers use a different form factor of PSU. This is infuriating because all the server PSUs available retail seem to be terrible compared to desktop PSUs! They tend not to support higher-end wattage (needed for power-hungry GPUs) and they are never "modular" (you get a bundle of cords sticking out of the PSU and there's no way to remove the ones you don't need except with scissors).
There are some 2U chassis that actually allow you to mount a desktop PSU. However, in practice they are not actually compatible with any modern desktop PSU, because all such PSUs these days are designed with their intake fan on the top or bottom, not on the front/back. In a 2U chassis, that intake will be totally blocked by the top/bottom of the chassis.
I am going to have to live with crappy server PSUs I guess.
*Retail server market is awkward*
More generally, I get the general feeling that the server-oriented components which are available via normal retail channels like NewEgg are... not what big server farms actually use. Everything seems kinda crappy? Maybe I just feel this way because they don't candy-coat everything the way the consumer market does. But I have the funny feeling that people building out datacenters generally order mass quantities of chassis, PSUs, and even motherboards tailored specifically to their needs, and as a result the retail market for server components may be surprisingly small and poorly served. But that's just my suspicion based on my experiences here.