I gave it 3 tries and each time, Yi picked one of the cars as the winner.
I've been watching for many months now, how LLMs got better and better at solving it. Many still struggle with it, but the top ones nowadays mostly get it right.
His answer was "Neither win" and it took him 1 minute and 24 sec using no pre-defined algorithm or heuristic.
He said his process of thoughts was:
"I figured it would take 10 hours for car A to finish 100 miles and it would take twice that long for car B. Since Car B is already halfway there when car A starts, then they would arrive together"
I as 40 year old man, approached it intentionally naively (eg. I did not go looking for an optimal solver first) by making a drawing and attempting to derive the algorithm. It took me ~3 minutes to come to the same conclusion but at the end I had a series of equations, but no algebraic proofs.[1]
So now you have a human child reference metric if you want it.
the "son" model might just be the future of LLMs!
Please release "son" and the weights used to train him it on github (with a permissive license, if possible)
Interestingly, GPT-4 also fails to correctly solve this prompt, choosing car A each time after multiple tries for me. I tend to find that models struggle with such logic puzzles when using less common phrasing (e.g., two cars "having" a race instead of participating in one, "headstart" instead of "head-start", etc).
GPT-4 correctly solved the problem when it was reworded to: "There is a 100 mile race with two participants: car A and car B. Car A travels at 10 miles per hour but does not begin driving immediately. Car B travels at 5 miles per hour and is given a 10 hour head-start. After 10 hours, car A begins to move as well. Who wins the race?"
You can tell ChatGPT that it’s brilliant at reasoning, ask it to rephrase the problem in its own words and then solve it avoiding any traps. I have special instruction sets for inducing these chain of thought behaviors. There is more output in the end, and it helps the model think better before coming to a conclusion.
On one hand, I don’t really understand why anyone would expect an LLM to solve logic puzzles. The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle.
On the other hand, it does feel fun that the top ones appear to solve it, and I understand why it feels cool to have a computer that appears to be capable of solving these puzzles. But really, I think this is just specificity in training. There is no theoretical or empirical basis for LLMs having any reasoning capability. The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence.
There might be no reasoning in a single pass which outputs a single token. But in the loop where the output of the LLM repeatedly gets fed back into its input, reasoning is clearly happening:
The LLMs lay out how to go about figuring out the answer, do a series of calculation steps and then come up with an answer.
If you add "Please answer in just one short sentence." to the prompt, even the top ones get it wrong.
Yep, humans too have to think before answering most non-trivial questions, and especially the ones that include calculations.
So it seems "obvious" that we should try to to give LLMs too some time to think before answering, for example with the popular methods of asking for step-by-step thinking, thinking out loud, and only giving the final answer at the end, and also asking it to proofread and correct it's answers at the end all can help with that.
Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance:
> There is no theoretical or empirical basis for LLMs having any reasoning capability.
Deep learning models are specifically designed for automatic pattern recognition. That includes patterns of reasoning and problem solving.
> The only reason it can solve it is because side the creators of these top models specifically trained the models on problems like this to give the appearance of intelligence.
That's not how deep learning works, and not how machine learning works in general. The models can automatically recognize patterns of reasoning then apply those methods to problems it has never seen before.
> The only way it can do so is not through reasoning, but by having been trained on a structurally similar puzzle.
This is a fundamental misunderstanding of how it works. The large deep learning models have 100+ layers, modelling extremely abstract features of the data, which include abstract patterns of problem solving and reasoning. They are not simply regurgitating training examples.
> There is no theoretical or empirical basis for LLMs having any reasoning capability.
Yes there is. Learning to predict the next token implies a lot of things, among which is also logical reasoning. The chain-of-thought approach shows that when you stimulate this behavior, you get higher accuracies.
"The paper, titled "Mapping Part-Whole Hierarchies into
Connectionist Networks" (1990), demonstrated how neural networks can learn to represent conceptual hierarchies and reason about relations like family trees.
Specifically, Hinton showed that by training a neural network on examples of family relationships (parent-child, grandparent-grandchild, etc.), the network was able to accurately model the inherent logical patterns and reason about new family tree instances it had not encountered during training.
This pioneering work highlighted that instead of just memorizing specific training examples, neural networks can extract the underlying logical rules and reasoning patterns governing the data. The learned representations captured abstract concepts like "parent" that enabled generalizing to reason about entirely new family tree configurations."
https://www.gnod.com/search/ai#q=Two%20cars%20have%20a%20100...
I gave it 3 tries and each time, Yi picked one of the cars as the winner.
I've been watching for many months now, how LLMs got better and better at solving it. Many still struggle with it, but the top ones nowadays mostly get it right.