Potential session/cache leakage between workspace instances or consumer accounts

Every time you multiplex requests from multiple clients onto one upstream connection, you are probably vulnerable to this, because (despite its superficial simplicity) HTTP is just too complex to reliably match the requests and responses to upstream.

For example a desync can be triggered in some systems by having more than one Content-Length header, by mixing Content-Length with chunked encoding, or by passing an HTTP/2 header called Content-Length that doesn't match the actual content length.

Here's a DEF CON talk (6 years ago) on this topic: https://www.youtube.com/watch?v=w-eJM2Pc0KI

The same attack has been applied to SMTP by messing up the line endings surrounding the end-of-message delimiter, where it's called SMTP smuggling. It may also apply to other protocols.

Every emergent behavior from these actors - whose claim to positive moral values is barely plausible - should be reported, discussed, dissected and critiqued early and often.

It's unfortunate that there is so little transparency that even if they deny there was a leak we will never know for certain.

> Same thing just happened on a Claude Mobile session in same Enterprise account. Common theme in both is Sonnet 5, first response after more than 5 minutes (cache miss).

All that said, it doesn’t require cross session leakage, it could just be training data or like those nightingale (probably the wrong bird*) data generations where they just prompt an LLM with nothing and it starts spitting out conversations.

I see a bunch of downstream comments about caching, sounds like maybe there’s an error where it loads nothing instead of the cache and so starts spitting out random generations.

* edit: it’s magpie. Worth looking at the concept, I’m not sure people realize they LLMs generate random conversations when prompted with nothing, this seems at least as likely as sessions leaking: https://github.com/magpie-align/magpie

If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.

That is when it bothers to respond instead of just sending back an 1099 error code

I wonder if there could be a large security situation playing out behind the scenes right now.

I’ve been working on using AI to assist me in writing meta parsing grammars. Fortunately I have not launched most of them yet. I know for a fact that the next generation of models represent a major step change in basic vulnerability identification and exploitation, especially if you know where to point them. They’ve found several bugs and at least one exploit in my parsing tools so far, I can’t imagine how many there still are waiting to be discovered across the entire modern tech ecosystem.

---

Note that the author did have a minecraft.py file. So not quite 100% random.

I’ve known some brilliant engineers who would also just randomly bring up Minecraft (more likely Factorio these days) so this makes sense.

Having an LLM session with very long context occasionally go off on a tangent is not uncommon. The people who expect absolute perfection out of every LLM interaction see this as some total indictment of the entire technology, but the people who use these tools daily have learned to treat the output as partially stochastic and to avoid extremely long context, even if the model offers it. It’s best to compact strategically or summarize next steps to hand off to a new session. Using sub-sessions can also reduce context pollution at the cost of additional token expenditure to summarize and transfer data to and from the sub-session.

I haven't had much issue with Codex, but seems Claude Code has major issues being reported nearly on the daily.

They also happen to be the most boastful about not reading or looking at the code.

LLMs are very capable, but not nearly to the level they seem to be messaging.

(We've actually moved on from vibe-coding to having the LLM vibe code itself in a loop)

* There's an enormous amount of very expensive shared state (context cache) which you do not want to duplicate when you can avoid it.

* Memory locality is crucially important for performance.

* Hardware is extremely over-subscribed.

* Hardware is extremely expensive.

These factors all make hardware or even traditional memory-space (hypervisor/VM/hardware assisted virtualization) isolation a non-starter for most workloads and customers, which forces all isolation to the software layer. This already makes things way harder than they are in commodity SaaS.

Moving beyond that, the tools, frameworks, and hardware which the system runs on (GPU) wasn't designed for task isolation and building this isolation is even moreso an emergent research field than it is in x86 CPU hardware-sharing (which has required a huge amount of effort over the past 30+ years to get where we are today).

And, the ratio of usage/sensitivity to maturity is also just poor overall; these are young companies with rapid development and enormous delivery pressure under incredible customer workload requirements, too.

I can't tell if the original post is a real issue or not, but I'm surprised there aren't more like this overall; the whole thing really is a house of cards in this sense.