> Now, let’s assume that there’s something that it truly doesn’t know anything about. I can’t think of an example right now but let’s say we go back two centuries when airplanes weren’t a thing. All it’s given is the raw data like 1) Birds can fly 2) Trains have engines that produce mechanical power from chemical/thermal energy 3) Bernoulli’s principle and fluid dynamics are fairly well-known at that point. Would you say that today’s AI would have been able to truly “reason” that building machines that fly are possible? Could they be capable of going from “zero to one”?
Yes.
AI has, multiple times already over multiple different era's definitions of what counts as "AI", been used to invent new things that humans have not been able to.
> Another interesting thought experiment: Could AI come up with the theories of relativity or other similar discoveries/inventions that didn’t just conform to the available knowledge but required genuine human imagination?
E.g. resolving multiple open maths problems over a few months that humans struggled with for multiple decades?
Right now AI's problem isn't that their best isn't good enough, it's that their worst isn't good enough. They can be superhuman one moment and amateurish-to-childlike in the next.