Tried out 3B on ollama, asking questions in optics, bio, and rust.
It's super fast with a lot of knowledge, a large context and great understanding. Really impressive model.
It's super fast with a lot of knowledge, a large context and great understanding. Really impressive model.
3B models are great for text manipulation, but I've found them to be pretty bad at having a broad understanding of pragmatics or any given subject. The larger models encode a lot more than just language in those 70B+ parameters.
I'm pretty sure the AI guys are well aware of which types of models they want to produce. Models that can intake knowledge and intelligently manipulate it would mean general intelligence.
Models that can intake knowledge and only produce subsets of it's training data have a use but wouldn't be general intelligence.
Usually the problem is much simpler with small models: they have less factual information, period.
So they'll do great at manipulating text, like extraction and summarization... but they'll get factual questions wrong.
And to add to the concern above, the more coherent the smaller models are, the more likely they very competently tell you wrong information. Without the usual telltale degraded output of a smaller model it might be harder to pick out the inaccuracies.
Though I wouldn't treat it as a domain expert on anything. For example when I asked about the safety advantages of Rust over Python it oversold Rust a bit and claimed Python had issues it doesn't actually have
So exactly like a human
Imo we should be testing reasoning for these models by presenting things or situations that neither the human or machine has seen or experienced.
Think; how often do humans have a truly new experience with no basis on past ones? Very rarely - even learning to ride a bike it could be presumed that it has a link to walking/running and movement in general.
Even human "creativity" (much ado about nothing) is creating drama in the AI space...but I find this a super interesting topic as essentially 99.9999% of all human "creativity" is just us rehashing and borrowing heavily from stuff we've seen or encountered in nature. What are elves, dwarves, etc than people with slightly unusual features. Even aliens we create are based on: humans/bipedal, squid/sea creature, dragon/reptile, etc. How often does human creativity really, _really_ come up with something novel? Almost never!
Edit: I think my overarching point is that we need to come up with better exercises to test these models, but it's almost impossible for us to do this because most of us are incapable of creating purely novel concepts and ideas. AGI perhaps isn't that far off given that humans have been the stochastic parrots all along.
Its not a perfect comparison and Llama does a lot more than English, but I would say 6.5GB of data can certainly contain a lot of knowledge.
For Ancient Greek I just asked it (in German) to translate its previous answer to Ancient Greek, and the answer looks like Greek and according to google translate is a serviceable translation. However Llama did add a cheeky "Πηγή: Google Translate" at the end (Πηγή means source). I know little about the differences between ancient and modern Greek, but it did struggle to translate modern terms like "climate change" or "Hawaii" and added them as annotations in brackets. So I'll assume it at least tried to use Ancient Greek.
However it doesn't like switching language mid-conversation. If you start a conversation in German and after a couple messages switch to English it will understand you but answer in German. Most models switch to answering in English in that situation
Your findings are Amazing! I have used ChatGPT to proofread compositions in German and French lately, but it would have never occurred to me that I should have tested ability to understand idioms, which are the cherry on the cake. I’ll have it a go
As for Ancient Greek or Latin, ChatGPT has provided consistent translations and great explanations but its compositions had errors that prevented me from using it in the classroom.
All in all, chatGPT is a great multilingual and polyglot dictionary and I’d be glad if I could even use it offline for more autonomy
That’s interesting; could this be an indicator that someone is running content through GT and training on the results?
Yeah, chatting more, it's confusing Spanish and Greek. Half the words are Spanish, half are Greek, but the words are more or less the correct ones, if you speak both languages.
EDIT: Now it's doing Portuguese:
> Εντάξει, πού ξεκίνησα? Εγώ είναι ένα κigneurnative πρόγραμμα ονομάζεται "Chatbot" ή "Μάquina Γλωσσής", που δέχθηκε να μοιράσει τη βραδύτητα με σένα. Φυσικά, não sono um essere humano, así que não tengo sentimentos ou emoções como vocês.