Comment by gorbypark | Hacker News Reader

Comment by gorbypark | Hacker News Reader

Parent Full thread

That's pretty surprising it can't do the months in alphabetical order. I just tried with LLaMA 30B. This is running on my MacBook Air with llama.cpp, quantized to 4 bits, and is a "raw" model (not alpaca / instruction tuned at all). Obviously some models will be better at some things (and will have different outputs given the same inputs) and worse at other, but you'd think Bard would be able to do that pretty simple task.

> Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.

User: Hello, Bob.

Bob: Hello. How may I help you today?

User: Please tell me the largest city in Europe.

Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.

User:Can you list all the months in alphabetical order?

Bob: Yes. January February March April May June July August September October November December

edit Sorry for this useless post...months are in order, not alphabetical order.

> Bob: Certainly. January, February, March, April, May, June, July, August, September, October, November, December.

> User: That's not in alphabetical order. Redo.

> Bob: Sorry about that. Here are the months again, this time in alphabetical order: April, August, December, February, July, June, March, May, November, October, September.

(Also LLaMA, 30B, llama.cpp on x86 Processor)

At least LLaMA sees its mistake and doesn't gaslight you (in this case). "I've been a good LLaMA." ;P

er... "You missed out January, try again"?

Ha, I didn't see that! :D

> User: Almost, but you missed January. Try again.

> Bob: Okay. Let's see...January, February, March, April, May, June, July, August, September, October, November, December.

Very brittle.

Both answers are wrong though?

gpt4:

months in alphabetical order

Here are the months in alphabetical order:

April August December February January July June March May November October September

I've never seen the months listed in alphabetical order before.

And it's absolutely breaking my brain. Which is a reaction I did not expect.

It looks so wrong.

Same here

I wonder if that's why it didn't want to put them in alphabetical order. There is only one valid order for the months.

It's a bit like the Stroop test.

https://en.wikipedia.org/wiki/Stroop_effect

A google search returns about 10,200 results for this, gpt4 probably picked up this sequence as well during the training.

Seems like it. I asked it to sort by their second letter and it failed horribly:

Sorting months by the second letter, we get the following order:

1. April 2. January 3. June 4. July 5. March 6. May 7. February 8. December 9. November 10. September 11. August 12. October

Months are not in alphabetical order.

And largest city is an ambiguous answer: per area, moskow is, by population its istanbul.

Oh man I am crazy, it spit out the months in order and not alphabetical order. Wow. I'll go home now.

Reply on news.ycombinator.com