Youtube music app allows this. I don’t think the website does.
This does what you want.
For a specific example, it has a nice toggle above every track, where you can flip between video and audio-only. It works great, except that I cannot find a way to force it to stay on audio-only. As soon as the current track ends, the toggle resets back to "video".
And this kind of stuff is all over the app.