Hacker News Reader

Comment by mvdtnz | Hacker News Reader

You can try this out now on Bard, which appears to be generally available. https://bard.google.com/

My first impressions of this are that it's quite good. I gave it a handful of logic puzzles which it could solve and asked it for some New Zealand native gardening and bird advice and it gave good answers.

Best of all it doesn't have that... tone that ChatGPT has. You know that kind of smug condescending tone that gives away ChatGPT-generated content. If you ask Bard to respond in the style of ChatGPT it can replicate that tone, if you like it for some reason.

View on HN

I like this prompt for testing LLMs as the problem is easy to reason about but likely doesn't come up a lot in training data:

> I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.

GPT-3.5 gave me a right-ish answer of 24.848 liters, but it did not realize the last lap needs to be completed once the leader finishes. GPT-4 gave me 28-29 liters as the answer, recognizing that a partial lap needs to be added due to race rules, and that it's good to have 1-2 liters of safety buffer.

I prompted Bard today and the three drafts gave three different answers: 18.28, 82.5, and 327.6 liters. All of these were wildly wrong in different ways.

I like that you're solving precisely a problem I face on a daily basis! (Still don't understand how ACC doesn't have a built in calculator for this).

Interesting. Regenerating replies does seem to help, for whatever reason. I've noticed the first after model loading seems to have a higher likelihood of being inaccurate or a hallucination.

Regenerating a GPT4xAlpaca 30B model from its initial answer of 243L, I got a variation of the following for the next 5 rerolls:

"Based on your qualifying time of 2:04.317 and the length of the race being 20 minutes, we can calculate the number of laps needed for the entire race. Assuming each lap takes approximately 2 minutes (based on the average speed), there will be 20 / 2 = 10 laps during the race.

Using the information provided about the car using 2.73 liters per lap, we can determine the total amount of fuel required for the race. Therefore, it would be advisable to carry at least 10 * 2.73 = 27.3 liters of fuel for this race."

Can you explain how the problem is solved for someone who doesn't understand racing?

Sure thing! If the race is 20 minutes and each lap takes 2m04s, that means there will be 9.67 laps till the race is over, and you round that up to 10 since partial laps must be finished. You need 2.73 liters per lap, so the 10 laps will use 27.3 liters total. GPT-4 is correct in suggesting a tiny safety buffer above that in case fuel usage differs from expected.

It's a math word problem, in which LLM's would not perform well. I have no idea why people try stuff like this.

People try stuff like this because it's precisely the kind of problem that AI would be useful for. If one of these models turned out to be really good at it, it would signify that they're now useful for a whole class of problems.

If you want to solve math problems, LLMs are very useful for this. Exactly how trained professionals are.

You make the model write code for you to solve it.

Would you ask your dad to compute the correlation matrix between 40 thousand vectors? No? Then don't ask an LLM to do it.

I ask ChatGPT / Bard to do all kinds of things I wouldn't ask my Dad for. This is a weird perspective.

Besides, GPT-4 did solve this question perfectly. I like that rather than just involving math, there’s also some real life knowledge needed to give a practical answer.

Because it exposes accuracy problems as querys often involve implied or implicit math skills.

> Bard isn’t currently supported in your country [Canada]. Stay tuned!

Too bad for us...

Same here in Belgium

Also from what I've heard France, Germany, Netherlands, Sweden...

It's quite odd given the announcement of 180+ countries. There are 195 countries in the world. EU consists of 27 countries. So, it can't be EU and Canada being left out or they are lying. I wonder what's going on. Is it a staggered rollout? There was nothing said about that. Pretty crappy to not simply give a list nor details on the rollout.

I don't think it's actually using PaLM 2. If it is, it's not great, but I heavily doubt it is.

The commentary they've made on it could mean many things. One of the possibilities is that they're using the API to PaLM-2 to classify the language and translate it to English if necessary, such that the simpler and non multilingual Bard model can handle it. Obviously this isn't necessarily what is occurring, and it could be just Bard is now palm-2 for some instances for testing, or a full rollout. Ultimately we don't know - because it's not open source, and companies are not great about telling you what they're actually doing.

How do you know Bard is actually using PaLM 2 rather than some lightweight model? The UI doesn't tell you which model it's using, does it?

Google announced it. [1]

> PaLM 2’s improved multilingual capabilities are allowing us to expand Bard to new languages, starting today. Plus, it’s powering our recently announced coding update.

That’s a bit vague, though. Bard might only be using it for some queries?

[1] https://blog.google/technology/ai/google-palm-2-ai-large-lan...

Frankly I did not perceive any difference in quality in Bard responses today and yesterday.

I think you are right, they probably dynamically choose PaLM 2 for some queries. Globally rolling out PaLM 2 for everything everywhere isn't Google style.

Bard isn’t currently supported in your country. Stay tuned!

Reply on news.ycombinator.com