Or don't try to make buttons tangible/physical objects. If your hand (or pizza paddle hand) is inside a button, highlight the button and then use trigger to "press" button. It is hard to press buttons you can't really touch
But also, to your suggested solution, having a button need to be interacted with in VR and a physical button is error prone and annoying. I know I use a button like that daily. I would rather just reach over and stab most of these buttons.