I can select very small text on my phone with my bigger fingers. And also with a touch screen you may have more options in a smaller space.
Sure, that may suck for critical things, but there probably are a lot of settings and so forth that could be hidden that are not critical?
But I understand what you meant, I prefer physical buttons as well.