Also on slower hardware - zooming is a drag, and it can be slow.
The browsers do allow for customisaton, and I think this is probably the easiest thing for an end user. I use a minimal font size of 16px, and that works for me. But it does break some sites - Facebook's layout has been broken over the last month like this for example. Which is fine if you kind of understand the reason for it, but could confuse other users. Just as some navigation breaks, think of horizontal nav, that falls onto another line when the text size increases - I'm thinking hover states, and drop down menus.
If he were reading a magazine and you said "do you find the text to be too small", you think he might turn to you and say "what do you mean by text, I'm reading a magazine here, there is no text."