After hand-rolling my own setups and working with a few libraries, I have found https://mediasoup.org/ v3 to be the easiest library to use that still gives me the freedom to work with the architecture I want. This of course you're not using WebRTC for its p2p capabilities and are willing to scale via SFUs which is a common approach these days.