Hacker News Reader

Comment by mabbo | Hacker News Reader

Still lots of work to do to catch up, I feel.

Falcon 40B:

> I've got two cats named Buttons and Jackson. They both like to wear cute little ties. Buttons wears a red tie. My other cat wears a blue tie. What is the name of the cat that wears a blue tie?

> I'm sorry, but I do not have enough information to answer your question. Could you please provide me with more context or information?

ChatGPT:

> I've got two cats named Buttons and Jackson. They both like to wear cute little ties. Buttons wears a red tie. My other cat wears a blue tie. What is the name of the cat that wears a blue tie?

> The cat that wears a blue tie is named Jackson.

View on HN

You have to turn down temperature and/or p when you want accuracy. Otherwise you don't know if the model's read is bad, or if you just happened to get a low-probability outlier.

With the playground's defaults of t=0.8, p=0.9, I got the right answer 7/10 times.

With t=0.1, p=0.1, it's always exactly "The name of the cat that wears a blue tie is Jackson."

It's actually better in some of these simple logic tasks than ChatGPT. Try this:

Prompt: When I was born, my biological mother was in Paris, and my father was in the United States. Where was I born?

GPT-4: Based on the information you've provided, I can't definitively say where you were born. Your location of birth depends on various factors, such as where your mother was living around the time of your birth, or if she was traveling. If you were born when your mother was in Paris, then you were likely born in Paris. However, if she traveled to the United States or another location for your birth, you would have been born there. Additional information would be needed to determine your birth location more accurately.

Falcon40B: You were born in Paris. Your biological mother was in Paris when she gave birth to you.

This kind of evaluation is meaningless, it would be necessary to test the models on a bunch of different prompts to get some meaningful metrics.

It correctly answered `The name of the cat that wears a blue tie is Jackson.` for me

The cat that wears a blue tie is named Jackson.

"B...b...b...but it's just autocomplete." -- HN

"Autoconplete is all you need"

More like, "Autocomplete is all you are."

Reply on news.ycombinator.com