Yup, using rem for as many things as possible has always been good to me.
If the user is on a phone and has a larger default font size due to vision difficulties, making padding scale with the font size takes screen real estate away from the larger text the user needs.
I want to use those as references to fix my UI (on increased font-sizes on small screens) before releasing an app I've spent 4+ months on.
Also, not having ridiculously oversized margins, like so many 2019-2022 websites trying to look "modern" used to use.
old.reddit.com is one example where the paddings still look good when magnification is set to 150-170% (which I have to do because of the tiny default font size). I think doing it that way but with better readability at 100% zoom, would be a decent solution.