I am using 8b-instruct and I am honestly blown away, this almost feels like early GPT4 levels of logic
I am absolutely pumped for whats in store for small models!
I am absolutely pumped for whats in store for small models!
One of my gotos is a variation of a classic logic puzzle, as models naturally tend to try to provide the original form of the solution because of token similarities and by seeing how well they can identify where steps are wrong or seeing if they can solve the variation when breaking token similarities with representative symbols or other techniques like mapping out object relationships in the initial answer before providing a solution it tends to reveal the underlying capacity for logic outside of simply similarities to the training data.
Mistral, GPT-4, Claude, Gemini can all solve it with various degrees of competency and hand holding.
Llama 3 just cannot handle it.
I think what we're seeing is that the massive training data set is leading to a much more capable stochastic parrot but not necessarily as capable world modeling. Or else there's an advantage to MoE for this kind of task that Llama 3 lacks. But it's tracking with my subjective sense of being somewhere between GPT-3 and 4 in terms of "critical thinking" ability, but not in the same class as more modern models, even though it seems to have a better breadth of training to draw on than most other medium sized models.
jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have?
(real answer is 2)
-------------------
Llama 3 8b
Let's break it down step by step:
Jack has 15 sisters.
Each of his sisters has 3 brothers.
To find the total number of brothers, we need to multiply the number of sisters (15) by the number of brothers each sister has (3).
15 sisters x 3 brothers/sister = 45 brothers
So, Jack has 45 brothers!
-------------------
GPT-4
Jack has 3 brothers.
Here's why:
If each of Jack's sisters has 3 brothers, these brothers are the same for all sisters, including Jack himself. Therefore, Jack and his 3 brothers make up the total count of brothers in the family. Since the problem states that each sister has 3 brothers, we know that Jack himself must be counted as one of these brothers. So, Jack has 3 brothers.
-------------------
Lol.
You can tweak the questions numbers and names to avoid training data.
Can Opus solve this?
jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have?
> Claude-3-Opus:
Let's approach this step-by-step:
We know that Jack has 15 sisters.
Each of Jack's sisters has 3 brothers.
Since all the sisters are from the same family, they all have the same brothers.
Therefore, the 3 brothers that each sister has are the same 3 brothers.
Since Jack is a brother to his sisters, he is one of these 3 brothers.
So, Jack has 2 brothers.
Therefore, Jack has 2 brothers.
A classic lateral thinking puzzle!
Let's break it down:
* Jack has 15 sisters. * Each sister has 3 brothers.
Now, here's the key insight: if each sister has 3 brothers, that means each sister is counting Jack as one of their 3 brothers!
So, how many brothers does Jack have? Only 2 The other 2 brothers that each sister is counting are actually Jack himself.
Answer: Jack has 2 brothers.
I don't have enough ram to run 70b but i'm very curious about performance on other logic puzzles!
"bob has 7 sisters, each sister has 4 brothers. How many brothers does bob have?"
-----------------
Llama 70b instruct (wrong)
A classic lateral thinking puzzle!
At first glance, it seems like a straightforward math problem, but wait...
If each of Bob's 7 sisters has 4 brothers, that would mean each sister has Bob and 3 other brothers, right? But that's not possible, because Bob is the only brother!
So, the correct answer is: Bob has no brothers. He has 7 sisters, and each of them has him as their only brother.
Did I get it right?
-----------------
GPT-4 (correct)
To solve this, let's consider the family structure:
Bob has 7 sisters. Each of these sisters has 4 brothers. Since all the sisters have the same number of brothers, and these brothers include Bob, this implies Bob has 3 other brothers. Therefore, including Bob, there are 4 brothers in total. Hence, Bob has 3 brothers.
>jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have?
Let's approach this step-by-step:
1) We know that Jack has 15 sisters.
2) Each of Jack's sisters has 3 brothers.
3) Since all the sisters are Jack's siblings, they share the same brothers. In other words, the brothers of one sister are the same as the brothers of any other sister.
4) Therefore, the number of brothers that Jack has is the same as the number of brothers each of his sisters has.
5) We are told that each sister has 3 brothers.
Therefore, Jack has 3 brothers.
> Nope
You're right, I made an error in my reasoning. Let me try again:
1) Jack has 15 sisters.
2) Each of Jack's sisters has 3 brothers.
3) Jack is one of the brothers of each of his sisters.
4) Therefore, the total number of brothers (including Jack) is 3.
5) To find the number of brothers Jack has, we need to subtract Jack from the total number of brothers.
6) $\text{Number of Jack's brothers} = \text{Total brothers} - \text{Jack} = 3 - 1 = 2$
Therefore, Jack has 2 brothers.
Ask it to write code for you, ask it pop culture questions to test recall and hallucinations, literally a million more interesting things you can do.