This is what expert systems did, and why they fell apart. The cost of doing this, ongoing, forever never justified the results. It likely still would not even at minimum wage, and maybe more so because LLM's require so much more data.
> All these problems should be assumed to be within the training data.
And yet most models are going to fall flat on their face with these. "In the data" isnt enough for it to make the leaps to a solution.
The reality is that "language" is just a representation of knowledge. The idea that we're going to gather enough examples and jump to intelligence is a mighty large assumption. I dont see an underlying commutative property at work in any of the LLM's we have today. The sooner we get to an understand that there is no (a)I coming, the sooner we can get down to building out LLM's to their full (if limited) potential.