You can build a fanless x64 PC quite easily: 4-core Intel Silvermont/Goldmont or AMD Ryzen Embedded V and R series can be passively cooled. Many of those ITX boards come with a 19V DC-Jack power-in so you can use a laptop power brick and eliminate the PSU fan as well. Fujitsu used to have some nice designs, I think now it's owned by Kontron. Asrock also has a good selection for more retail-oriented products.

Of course you can also passively cool a full-fledged desktop CPU if you want but that's a different category in terms of price and performance so I am not talking about that.

As for power, I have Puma-based x64 4-core boards that idle at 5W and consume 12W at full load (measured at the wall). Goldmont should be similarly efficient. RPi4 is about 7W at load I think. Not a huge difference.