If brain structure stayed static throughout your lifetime I would be inclined to agree. But as the structure is constantly in flux, and proven so, no so much. Granted there is disagreement on how much your iq can change over time.
For example, some IQ tests ask you "What's the next number in this series...". If you aren't very good with numbers, you might do very poorly on this section. However, if you sit down and learn about a lot of the patterns that show up frequently in those types of questions, then you may increase your score dramatically on that section, and thus improve your IQ.