It just depends. I live at sea level in the bay area but regularly spend a week at a time at 8000ft near tahoe. I've never thought twice about it.
When we went to Peru, we flew in to Cusco which is around 11,000ft which seems like "not that much higher"; however, the decrease in pressure and O2 content is exponential, the the transition between "no problems" and "some problems" may be abrupt. My wife is fitter than I am, but seemed to have had more issues on our first few days, despite us both taking Acetazolamide (although we didn't bother until after arrival).
Another factor is that speed of ascent factors into onset of sickness; flying in to a high altitude can be worse than ascending over a period of days, as it takes time to acclimate.
Physical fitness doesn't seem to have a lot to do with how a given person will respond to altitude. I believe women and older people are slightly more likely to respond poorly, but it's kind of a crapshoot.
We hiked over 15,000ft but it definitely sucked at times -- tingly hands and feet related to the drugs, I believe.
Funny thing was, after > 1 week at that altitude, we both stopped taking our drugs the morning of our departure. Our flight was delayed by a few hours and by 1pm I started feeling the onset of symptoms again, wishing I had taken my dose that morning.