Hacker News Reader

Comment by _fw | Hacker News Reader

I have a Gemini Pro trial, have been paying for ChatGPT since premium launched and as of last month started paying for Claude Opus.

Claude’s context window for Opus is remarkable and I’d say that GPT4 is vastly inferior as a result.

The second Anthropic ship Claude with a sandbox that can use python (or equiv) to take care of tasks, I’d happily end my ChatGPT subscription.

GPT4 has been here for over a year and it shows. But until other assistants can manipulate Excel data as easily… I’m stuck paying for it.

Claude is amazing for summarisation, copywriting and reasoned thinking. I love it.

View on HN

I got into the habit of running every query I do through all of the SOTA models so I can directly compare the results.

This was originally GPT-4, Claude 3 Opus, and Gemini Advanced. I recently added Meta AI when they launched.

Right now I've sent 486 queries through the first three systems.

The clearest pattern to emerge is that Gemini is terrible, not on par with the other two. There hasn't been a single query that it was the only model who did well. Around 1/4 of the time it gives a clearly inferior answer to the others.

But between GPT-4 and Claude it's less clear. 31 of the 486 queries Claude provided a significantly better answer than the other two but 20 times GPT-4 provided the significantly better answer.

I do think that Claude is a slightly better model but right now it's not a clear enough advantage that I'd recommend it generally. I will say you can probably cancel you Gemini subscription if you're using it though.

One thing I would say about Opus over ChatGPT is that it can return code without placeholders whereas ChatGPT will always have placeholders. It wasn't always that way though and it seems like cutting corners.

interesting, this was a problem around the turn of the year for me but chatgpt has stopped doing it now

Why would you compare Gemini pro with the other two - they are not in the same category. Did you mean to say advanced?

Why isn’t the model offered by Google in the same category? Or did I misunderstand their latest naming fiasco and “Gemini Pro” is not actually their latest model?

EDIT: It looks like Gemini Ultra 1.0 > Gemini Pro 1.5 > Gemini Pro 1.0… is that true? What a horrible versioning scheme.

Yes I do mean Gemini Advanced.

> running every query I do through all of the SOTA models so I can directly compare the results

For everyday non-dev users, Msty for MacOS lets you run split chats and compare each model's answers.

https://msty.app

You might want to check out perplexity.AI. It has access to many of these models, and it’s trivial to switch models mid conversation and ask it to repeat itself, so that you can see the differences.

That’s interesting, I found Opus to be underwhelming compared to GPT4. Hallucinations occurred as often, if not slightly more, and overall I found Opus to be not as “creative” when writing revisions. That and the rate limit resetting at midnight as compared to GPT4 being something like an hour or two made me return to old faithful.

Maybe I should give it another try.

I enjoy chatting with Claude (opus) more, but still find GPT4 is better for coding.

I also frankly will not extensively use a model that I can't turn off data storage for.

Kagi (paid privacy focused search engine) allows access to all of these for $25 a month.

Only downside I’ve found is it doesn’t save chat history for privacy reasons, but you can download them.

https://help.kagi.com/kagi/ai/assistant.html

Is the context window reduced?

Yes, input context window is reduced to 16k characters (not tokens).

https://help.kagi.com/kagi/ai/assistant.html#tips-and-tricks...

Reply on news.ycombinator.com