Hacker News Reader

Comment by mg | Hacker News Reader

I tried my standard testprompt:

    Two cars have a 100 mile race. Car A drives 10
    miles per hour. Car B drives 5 miles per hour,
    but gets a 10 hour headstart. Who wins?

And the Liquid-40B model failed with a long explanation why car B wins the race.

Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a headstart. Because every time the fast runner gets to the point where he saw the slow runner, the slow runner is already a bit further along the track.

View on HN

To be fair, this article does say:

>What are Language LFMs not good at today:

>Zero-shot code tasks

>Precise numerical calculations

>Time-sensitive information

>Counting r's in the word "Strawberry"!

>Human preference optimization techniques have not been applied extensively to our models yet.

Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.

But they also say

> What are Language LFMs good at today:

> General and expert knowledge

> Mathematics and logical reasoning

"Our LLM is good at mathematics but bad at computation" doesn't really cut the mustard here. What they mean is "good at well-known math benchmarks but bad at simple math problems that it hasn't been trained on." The idea that this constitutes "mathematics and logical reasoning" is a testament to AI companies' poor scientific standards.

I got this >>

To determine the winner, we need to calculate the time it takes for each car to complete the 100-mile race.

For Car A: Speed = 10 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 10 miles per hour = 10 hours

For Car B: Speed = 5 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 5 miles per hour = 20 hours

However, Car B has a 10-hour headstart, so we need to subtract that from its total time: Adjusted Time for Car B = 20 hours - 10 hours = 10 hours

Both cars take 10 hours to complete the race, so they tie.

What about acceleration times? :-)

0-5 vs 0-10 should be slightly different

Thanks! I am collecting all "test prompts" which appear at HN and Reddit. I plan to create a hugging face dataset. I will soon publish "Vojta-1B", which is gonna pass all of them.

Reply on news.ycombinator.com