This is the conclusion I've come to as well, but I'm still not 100% happy with my setup. First off, many cheaper audio interfaces might have a hardware monitoring toggle but not allow monitor mixing. I found these devices to be totally unusable since many times the monitor level ended up being way too low to be effective.
Even with monitor mixing, however, I'm still annoyed that I have to dial in the mix and adjust the volume every time I connect with someone depending on how loud or soft their microphone is. Are you aware of any audio interfaces that allow for adjusting the absolute monitor level vs. just the relative mix level?
Like, I always want to hear myself at a specific level and I'd love to be able to adjust the output volume so I can hear the other person properly without affecting my monitor level. Currently I'm using an Elgato Wave XLR interface with an Audio-Technica BPHS1 broadcast headset and it works perfectly except for this one issue.
Edit: Forgot to mention that I've also found mic mute buttons to be another often overlooked necessity! The capacitive mute button on the Wave XLR is probably my favorite feature. Totally seamless/silent muting and unmuting.