On the easiest side of the spectrum, Apple could take the previous-generation desktop case (I believe Mac users lovingly call it the "cheese grater") and install a modern motherboard, CPU, memory, disks, and GPUs. There's virtually no design effort needed. Maybe a SuperMicro Xeon workstation board, a couple Xeon E5s, 64 or 128 GB of memory, a bunch of SSD and disk options, whatever GPUs work well with macOS. (These parts are roughly what I've used in my personal workstation with great success; but I don't run macOS.)
Hackers do this every day that a Hackintosh is booted for its first time; but such hackers don't have the luxury of battle-testing device drivers for their grab-bag hardware. So they nervously worry about operating system updates. Obviously Apple would spend a month or two ensuring the drivers are bulletproof and boom, there's a new high-specification Mac Pro.
If reusing an old design is simply too uncouth to contemplate, they could design a new nice, minimalist case sized for industry-standard components. Dozens of PC case designs would pass muster. I suspect most people who actually want a Mac Pro don't care if it's ultra compact or a mid-tower that sits on the floor. Why invest so much engineering effort in making it compact? Take away the size constraint and the required engineering effort essentially evaporates.