It's doable, but you need to spend time to optimize the noise factor:
- If you need many cores but can live with lower clock speeds, you can underclock.
- 1U servers will be noisy no matter what. 2U is already more manageable. 4Us are the best.
- GPUs will be noisy, no real solution around that. High heat production, and they hinder the air flow, so the front fans will have to spin faster. I physically remove GPUs from chassis when I'm not actively using them.
- There are small-size soundproof server cabinets that can easily solve this problem. They're very expensive though.
- Depending on the server, you can directly control fan speed at the firmware level. This needs careful control if you don't want to destroy your hardware and/or burn down your place.
- The pandemic makes all this much harder. I used to keep things running lightly during the night and full blast during the day when everyone was out of the house. I saved tons of money, compared to what I would have paid on any cloud platform. Now that we're home 24/7 it's not so worth it anymore.
Also this is all worth it if you actually enjoy playing with the hardware, in that case the money you put into this brings you both fun and cheap compute. If the setup is a chore for you, don't do it, it probably won't be worth your hourly rate.
Fun fact: the noise/space/aesthetics factors are referred to in the community as the "WAF" - Wife Acceptance Factor ;-)