Hacker News Reader

Parent Full thread

cubefox

I don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.

View on HN

Yes, they are applied. Here's OpenAI doc which describes how to set various sampling parameters for GPT-3:

https://platform.openai.com/docs/api-reference/completions/c...

See presence_penalty and frequency_penalty.

Sampling techniques is one of important arts of LLMs, you'll can find a lot of papers on them.

In general, smaller are more prone to repetition, but you can get caught in it even with larger models.

To clarify: I meant that in general they are commonly applied, but in this case they weren't as the author confirmed. The repetition, of course, doesn't happen all the time.

Yes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.

Reply on news.ycombinator.com