https://www.gnod.com/search/ai#q=Two%20cars%20have%20a%20100...
I gave it 3 tries and each time, Yi picked one of the cars as the winner.
I've been watching for many months now, how LLMs got better and better at solving it. Many still struggle with it, but the top ones nowadays mostly get it right.
His answer was "Neither win" and it took him 1 minute and 24 sec using no pre-defined algorithm or heuristic.
He said his process of thoughts was:
"I figured it would take 10 hours for car A to finish 100 miles and it would take twice that long for car B. Since Car B is already halfway there when car A starts, then they would arrive together"
I as 40 year old man, approached it intentionally naively (eg. I did not go looking for an optimal solver first) by making a drawing and attempting to derive the algorithm. It took me ~3 minutes to come to the same conclusion but at the end I had a series of equations, but no algebraic proofs.[1]
So now you have a human child reference metric if you want it.
[1]https://twitter.com/AndrewKemendo/status/1766872572300235022
GPT-4 correctly solved the problem when it was reworded to: "There is a 100 mile race with two participants: car A and car B. Car A travels at 10 miles per hour but does not begin driving immediately. Car B travels at 5 miles per hour and is given a 10 hour head-start. After 10 hours, car A begins to move as well. Who wins the race?"
On the other hand, it does feel fun that the top ones appear to solve it, and I understand why it feels cool to have a computer that appears to be capable of solving these puzzles. But really, I think this is just specificity in training. There is no theoretical or empirical basis for LLMs having any reasoning capability. The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence.
The LLMs lay out how to go about figuring out the answer, do a series of calculation steps and then come up with an answer.
If you add "Please answer in just one short sentence." to the prompt, even the top ones get it wrong.
Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance:
Deep learning models are specifically designed for automatic pattern recognition. That includes patterns of reasoning and problem solving.
> The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence.
That's not how deep learning works, and not how machine learning works in general. The models can automatically recognize patterns of reasoning then apply those methods to problems it has never seen before.
> The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle.
This is a fundamental misunderstanding of how it works. The large deep learning models have 100+ layers, modelling extremely abstract features of the data, which include abstract patterns of problem solving and reasoning. They are not simply regurgitating training examples.
Yes there is. Learning to predict the next token implies a lot of things, among which is also logical reasoning. The chain-of-thought approach shows that when you stimulate this behavior, you get higher accuracies.
Geoffrey Hinton - Mapping Part-Whole Hierarchies into Connectionist Networks (1990)
https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf
"The paper, titled "Mapping Part-Whole Hierarchies into Connectionist Networks" (1990), demonstrated how neural networks can learn to represent conceptual hierarchies and reason about relations like family trees.
Specifically, Hinton showed that by training a neural network on examples of family relationships (parent-child, grandparent-grandchild, etc.), the network was able to accurately model the inherent logical patterns and reason about new family tree instances it had not encountered during training.
This pioneering work highlighted that instead of just memorizing specific training examples, neural networks can extract the underlying logical rules and reasoning patterns governing the data. The learned representations captured abstract concepts like "parent" that enabled generalizing to reason about entirely new family tree configurations."