This is really program dependent. The NSF actually publishes statistics on all kinds of aspects of degree programs. Life Sciences (which doesn't have the reputation as a super speedy science program) has a median time in PhD program of around 5.5 years. Obviously the distribution is going to be skewed, but some fields have had a push towards really reducing the time to graduation.
I'm all for doing serious introspection before (and during!) a PhD, but the comment is way too short to set off many alarm bells for me.