Pattern recognition and the ability to construct, destruct and handle abstract structures in your mind IS the essence of IQ. Just have a look at the usual tests.
What people don't like, or even question, about this concept is the correlation between IQ and "intelligence" (whatever this is).
IQ seems to be mostly a genetic trait. Test results kind of normalize in grown up humans. Childhood experience gets less and less significant with age, which points to a strong genetic influence. (According to Wikipedia).
I think a lot of people actually don't like the implication arising here, as it points in the direction that you need to be born "smart" to be "smart" — and that you can't change mostly anything about that significantly no matter what you do.
Nevertheless there seems to be a difference how fast, or even if, one can reach "full potential".
IQ is just a trait, and without the right stimulus it won't develop quickly. This seems no different to other genetic traits after all.
I'm of course not going into the question whether IQ is a good, or even adequate, measure of "intelligence" (whatever this is), as this is a can of worms… At least it's quite sure not the full picture. One can for example excel at arts or sports without a high IQ.