Had the same thought sometime back about AI discovering theory of relativity with only the data before 1905. It would give a definite answer about whether any reasoning involved in the LLM output.