Hacker News Reader

Comment by random3 | Hacker News Reader

I'm curious about how the benchmark is done. I suspect it can be improved in order to represent user's / usability expectations.

I gave Bard a go, after seeing Jeff Dean's tweet.

It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off.

I asked it to generate a chart and 3 times it came back with "here's a chart" with no chart, finally saying it doesn't have that ability.

View on HN

My favourite by miles was asking bard for some more mathematical examples after explaining quantum field theory quite well in words, to which it said "Ok here are some examples of mathematics: 2+2=4"

https://bard.google.com/share/ff90b8239ebe found it

It’s based on users blind rating the output of two LLMs given the same prompt.

Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it.

Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it.

And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.

Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.

You’re probably using the old Bard model. You can try the new one, bard-jan-24-gemini-pro, by clicking the Direct Chat tab on https://chat.lmsys.org.

Thank you! Quick feedback - lmsys will introduce bias because it lacks rendering support for things like [ x_2 = \frac{1}{2} \left( 1.5 + \frac{2}{1.5} \right) = \frac{1}{2} \left( 1.5 + \frac{4}{3} \right) = \frac{1}{2} \left( \frac{9}{6} + \frac{8}{6} \right) = \frac{17}{12} \approx 1.4166666666666667 ]

I tried Bard today, it did improve surprisingly

Disclaimers: I don’t work for Google

https://i.imgur.com/b0fIGyS.png

that makes it even worse for saying 2 times "here's your chart" and the third one appologizing for not having the capability, when in fact is has it?

Here you go https://imgur.com/a/MT04viM

That is indeed weird.

Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison.

Also what a wild comparison, because afaik chat gpt can’t make charts either.

> Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison.

Wait so if Google suddenly started charging for Bard, it would be instantly better?

It could be. Bard is clearly not the best model Google has and the only reason not to serve the better models is inference cost.

Google is fighting for its life here. They are not worried about cost.

Google is one of the most cost conscious companies in tech when it comes to compute costs. Sure they are blind to other kinds of cost like reputation damage due to stupid leadership decisions. But in terms of their tech, they run their servers and their network to very high utilization, definitely exceeding competitors like Amazon.

You mention cost, and then proceed to describe efficiency. They are different things.

They can spend billions giving away compute (cost) and be extremely efficient (high utilization) in doing so.

Then what's your explanation for them not deploying Gemini?

It's either to expensive or they just don't think it's that important to have a competitive, accessible model out there (which is a valid stance imo).

I don’t really know tbh. I don’t think their current position is because of penny pinching. They invented the transformer and slept on it while OAI ate their lunch.

They would not charge for Bard, we are providing free training data and free test hours for them.

It absolutely can. Try asking ChatGPT4 something like "Please draw a chart of the population of Switzerland over the past 20 years".

It will write Python code to use matplotlib and the resulting chart will appear in the chat. I'd show an example but their sharing feature doesn't work with images for some reason.

> Try asking ChatGPT4 something like "Please draw a chart of the population of Switzerland over the past 20 years".

I tried this on the free version you can get to without paying anything, I didn't realize the advanced version supported this.

Are we talking about a capability or pricing benchmark?

> It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off.

If this aint anecdotal evidence, I dont know what is. You need to assume a whole lot to draw these conclusions from prompt output comparison.

Heck, just compare a couple outcomes from ChatGPT and you should conclude the same thing, assuming rationality of course.

Reply on news.ycombinator.com