But even if they were the latter, that's no guarantee that they would be a good way of measuring intelligence.
If I keep you up and prevent you from sleeping for 5 days straight, and then give you an IQ test, I didn't test your intelligence in its actual potential. And if I pitted you against a well rested person, the IQ difference doesn't necessarily measure any real difference in intelligence.
Similarly, if one kid A is brought up in a nurturing family, loved, well fed, well treated medically, educated, surrounded by people reading books etc, compared to a kid B from a poor family, with absent parents working all the time, starving occasionally from lack of money, with no intellectual stimulation and role models, etc -- measuring their respective IQ doesn't tell you whether kid A is inherently smarter than B, nor does it tell that kid B is poorer because it has less IQ...