Hacker News Reader

The 7B model just barely fits in a 3090, but I've yet to get useful output; it usually gets stuck in a repetition loop.

Spent some time playing with sampling parameters, to no effect. Has anyone else had a go?

Make sure you are using the instruction tuned model. The base model will be difficult to prompt.

It works in 8-bit with about 12GB of VRAM usage. Here's sample code:

If you're getting stuck in a sampling loop, try raising the temperature (0.6+) and/or increasing the repetition penalty (1.1-1.2 often work well).