Hacker News Reader

Possibly something to do with other providers using the it to train their own models?

The only "secret" there is a very basic instruction that the model receives, like "summarize current state and upcoming work" before compaction - same model that was just running your inference, with same cache, only server side, with no extra tools or capabilities. Then the fresh context gets the output from that as an encrypted blob + codex then injects up to 64k tokens of previous conversation, the latter part is visible in source code.

There's nothing to gain from this, really. Perhaps they're preparing for something in the future, where they could give the model server-side tools that improves summarization, but right now, it's just a simple prompt.

The compaction prompt doesn't seem like the valuable thing here. I suspect they're protecting the compaction result itself. If you're trying to distill a model, collecting lots of examples on how a large conversation gets compacted to a smaller summary is particularly useful data.

No, you can give the model same prompt and it will give you a similar compaction result. On the backend, that's precisely what happens. There's nothing else going on in that encrypted blob, it's just summary of what model responds with when prompted "summarize current state and upcoming work".

Reply on news.ycombinator.com