Hacker News Reader

Wow that's weird. I have it running sometimes at least 8 hours straight on a single prompt and no weirdness.

Could it be something weird like you're hitting your compaction limit?

No compaction happened. (That is set near 256k.) It seems to really not like my domain but I've seen it for various things as well (especially high-entropy tokens). Dropping random characters or various things. I've tried various flash attention and context quantization settings but nothing seemed to resolve it completely.

Reply on news.ycombinator.com