Granted, the RPi2 had a slow CPU, but I am finding the RPi4 is snappy enough for "serve a website out of a docker container" workloads. I am hitting SD card I/O limits faster than I hit CPU limits.

Power supply: It's USB powered. For a home server, I find it convenient to plug it into a USB port on the router, to which it is also connected for ethernet.

SDCard: It was set up to only use the sdcard to boot and use a USB stick as root, which gets around all of the sdcard issues. This is not a totally plug-and-play experience, though. My understanding is that the newer firmwares finally allow direct boot from USB, though.

Case: You can DIY it easily. The case for my home server was basically a folded piece of cardstock on which I drew a picture of a once-bitten pear. Other times I've used legos or dremel tooled altoid tins.

Heat sink: Unnecessary prior to the Pi4. (which also ramped up the cpu and power usage quite a bit...) They keep producing the older models, so it's mainly a matter of picking where you want to live on the compute vs heat management scale.

Heatsink and case were totally optional in my experience (or a case would make thermals worse). Power supply maybe, if your phone charger did not already work. SD Card or flashdrive; I have many of these lying around.

There is something to be said for having keyboard, pointing device and screen included with a used laptop, but the stuff you list it seems you just don't actually need.