Yes, this is what I do. It would be nice to use the volume rocker to do that.
Honestly, the most intuitive option for me would the audio controls should never touch the ringer, bury that behind a menu, rather than switching based on context.
You can always adjust the ringer (something people rarely do -- for quite times there's the "silent" mode anyway) from the UI, and always use the volume buttons for the system's (music, video etc) audio volume.
You can touch the screen to drop the selector, click on which volume you want, and go ahead with using the rocker for that volume selector.
To silence app-audio, I use just: volume-rocker, click on down arrow, click on music icon to mute. That's it.