The Mac mini 3.5mm jack advertises "advanced support for high-impedance headphones" so you might not even need an amp unless you have some really crazy cans.
My M1 MacBook Pro (the first one in the new chassis) can easily drive my 250 Ohm DT 990 Pro to uncomfortable loudness so it should indeed be fine for 99% of headphones. I've been told it also drives 600 Ohm headphones just fine.