Hacker News Reader

Comment by mrkramer | Hacker News Reader

Right now, but how long does it take for OpenAI, Facebook and Google's Bard/Gemini to catch up?

Google is trying and just can't? They are pretending they aren't behind despite setting a pile of cash on fire by spending exorbitant amounts on exaggerating their capabilities in basically fake videos.

Meanwhile you see Mistral casually dropping magnet links to weights with barely any instructions on how to use them on a Friday afternoon and you see reports of some very happy people finetuning on a civilian 4090 and getting GPT-4-quality performance in blind tests on Saturday morning.

Google is trying to catch up with OpenAI, not with open source 7B models.

Open Source 7B models have already caught up with OpenAI.

The current top of the open model 7B leaderboard beats GPT-3.5 and Bard while running on a laptop, smartphone, raspberry pi, or 10 year old graphics card.

Tim Dettmers just released code to get Mixtral 8x7b running in 4GB of RAM, the same amount required by Mistral 7b.

We are quite clearly a matter of weeks from Open Source being on par with GPT-4, likely before Google's Gemini Ultra is even released.

This is all to say nothing of the multi-model LLama-3 120B coming in 2 months according to insider leaks.

I’m very skeptical of LLM benchmarks because I’ve been testing models through Ollama and none of the open source 7b models are at even close to GPT4 (or Open AI’s GPT 3.5 implementation) in daily usage, despite what benchmarks say. LLM results are actually really hard to measure objectively because it depends on what kinds of questions you ask and what answers you expect.

I’ve just been finding lots of little errors and reasoning mistakes, and even in terms of producing useful results, they fall short of GPT4.

Considering Google been lying in their marketing videos about their AI capabilities, OpenAI seems to be falling apart from the inside and Facebook refuses to release open (not commercially-locked "open") models, there seems to be space for at least one honest and open AI company.

Reply on news.ycombinator.com