You can hear the fan at full load, especially on the M4 Pro. I really wish Apple went with a larger case and fan for that chip, which would allow quieter cooling.
Also, many units are affected by idle (power supply) buzzing: https://discussions.apple.com/thread/255853533?sortBy=rank
The Mac Mini is quieter than a typical PC, but it's not literally silent like, say, a smartphone.
My Mac Mini M2 never does any noise, even when I run FFMpeg the fans don’t spike. It just gets slightly warmer. Still, unless I’m doing these high CPU bound activities, every time I touch it it’s cold as if it was turned off, which is very different than my previous Intel one that was always either warm or super hot.