Settings -> Sounds & Haptics -> Change With Buttons: OFF. Now the volume controls will only ever change the audio volume and never the ringer volume.