I've not found anything better than StaxRip if you want access to the deep guts of encoder options. It can use nVidia's nvEnc API either directly, or through ffmpeg; same for Intel's QuickSync and AMD's VCEEnc.

https://staxrip.readthedocs.io/generated/screenshots.html

Or Permute on MAS/Setapp

Yes. Last I checked handbrake used FFmpeg in the background.

handbrake is such an underrated tool! I've been using it for over a decade to compress and convert videos, and nothing comes close (except ffmpeg ofcourse)