jcgrillo

Yeah despite the conceptual statelessness, there is quite a bit of state that hangs around though--KV cache and context. I still haven't been able to find anything concrete in docs about how these are isolated. In any case it's clearly a different class of issue than the one from the article. Not endemic to how LLMs work, just normal web session stuff, modulo some GPU memory handling.

undefined

Isn't everything these days? It's all gotten so gross.

I realize the context is where all the retained information is, I guess given how insecure the attempts at preventing injections appear to be I (maybe unfairly) assumed the efforts to keep contexts isolated are similarly lacking. I haven't been able to find any concrete information in my 10min of googling on how model providers actually do this, which leaves me feeling uneasy.

Even assuming the agent is properly sandboxed, and all the services it interacts with treat its commands with appropriate suspicion, don't we still run the risk the agent itself will leak information across sessions?

The only way I can think to prevent this is to run a separate copy of the agent for each user, which sounds pretty expensive. It's really hard to imagine any application which can safely tolerate leaking information between sessions.

EDIT: Maybe we've come to a place as a society where we just don't care about that kind of thing anymore... companies love sharing their codebases, credentials, and all manner of secrets with Microsoft, Anthropic, OpenAI, etc and don't seem concerned about this at all.

Yes, sorry I should have been more specific. A classification task seems totally safe, and like it plays well to LLMs strengths. You also have all kinds of options if it goes wrong, and bounded consequences.

What I'm talking about is something like a customer support agent. If that thing can take any consequential action other than simply parroting publicly available documentation back to users, that's unsafe, or at least likely to cause problems. If you believe me that it would probably be a bad idea for a customer support agent to, say, be able to twiddle RBAC entitlements then probably we can't replace our support staff with an AI agent. OK, so maybe the AI agent can be sort of a front-line filter. Now we need some way for this front-line filter to bubble tasks up to the second line. This fits with how many support orgs work, seems sensible right? But how might this be abused, and what can an attacker do? Potential consequences include DoSing your entire support org, flooding your jira/salesforce/whatever instance with garbage, etc.

So even the most limited, almost useless application is kind of dangerous.

EDIT: one thing people really seem to like the idea of is "natural language queries" in data intensive products. Personally I believe this idea is misguided--query languages exist for a reason, they're really useful tools for thinking about queries. But giving these people the benefit of that doubt, I still can't think of any way to do this safely unless every user gets their own sandboxed model instance. Otherwise it seems likely someone will be able to exfil another user's queries. This is of course assuming there's sufficient security between the LLM and the database that's actually _running_ the queries, which is not trivial.

Information density is one part, precision is another. Papers are often presenting work at the frontier of the field, which is by nature not well understood yet, and competitive. To have something worthy of publication is to have something that is new, and that often requires a degree of precision to communicate that we don't use casually. I think it's pretty gross to denigrate "academic writing" as obfuscatory, just like it's gross to make broad sweeping generalizations about journalists.

The security products industry has a long and storied history of using national security folks in marketing or marketing-adjacent ways (putting them on boards to lend an air of legitimacy, etc). I don't doubt that such people know a lot about national security, but I don't think they know shit about computers, generally.

NSA and others employ thousands of people who actually are super good at computers. If this message was coming from those people--actual credible experts--I'd be a lot more interested. Instead, what it seems like is happening here is the same organizational brain rot that's happening everywhere. Out of touch leadership, a skewed information environment, and bad incentives align to produce utterly insane conclusions divorced from reality. In short, psychotic delusions.

This is, probably, just marketing noise. Nothing to worry about, except that the people spouting it have been psychologically compromised. So if their job function is essential, that's a problem.

If that's the case then user-facing products that can take any useful action are strictly off the table.

100%. Anyone who is feeding unsanitized input to an LLM is doing it wrong. It'd be just like letting users craft their own SQL queries. I think the security aspect raises an interesting (if awkward) question:

How do you sanitize inputs to an LLM? Like how can you even make a secure user-facing product with this thing?

Maybe I'm lacking imagination, but it seems to me all the great "natural language interface" solutions this is supposed to enable are pretty badly hobbled by this issue.

It's more nuanced than either extreme. But regardless of the root cause, if you have engineers duplicating work left and right something has gone wrong. Their labor is not being used efficiently.

EDIT: LLM or not, this is still true. If you have LLMs pumping out tons of duplicate code you're wasting tokens, and probably more importantly wasting engineer hours reviewing duplicate code.

In some cases it might be a fair trade, in moderation. In general it's certainly wrong.

> engineers engineer around them with their own solutions

When that happens there's a major engineering leadership failure currently in progress, even if engineering leadership isn't aware of it.

I don't have a good intuition about the what makes a problem sensitive to tuning, is it possible there are a large number of control problems where proper tuning doesn't matter much? Or the deleterious effects of improper tuning (like oscillations or overshoot) are masked somehow?

EDIT: I guess intuitively, big (lots of inertia) damped systems are probably pretty safe--you can do all kinds of crazy things with the control input and it won't really have much effect. The only way you could go wrong is drift.. Anything that is inherently stable seems like it should be "easy"--like a high-wing monoplane with lots of dihedral angle, you release all control inputs and it defaults to straight and level flight.

I saw a crazy stat somewhere like some large fraction of the industrial PID controllers that are actually deployed in plants are completely untuned--that is, just running default settings. I think it was a book about tuning PID controllers, but I don't have the exact citation handy..

PSA: do not forget to properly format the cover of your TPS reports

It's endlessly fascinating to me how long it's taking for GPUs to be adopted for this kind of workload. I remember all the excitement around the GPGPU era, OpenCL, and eventually CUDA. That was like 15yr ago! Yes, GPUs can do a fantastic amount of computing. But it's really hard to make them do it efficiently. I think maybe the implicit assumption at the time was that something would come along that would make it easier. Despite continuous advances over the last decade+ it's still really hard.

I feel like we're about to learn a similar lesson with generative AI. Things don't always get easier/better/faster.

I feel like based on the amount of power these people wield in the world I should be afraid of them, but then I read this:

> registrants returned again and again to the same theme: that AI will reorder work, war, education, and belief within a few years.

lmao. nvm. they're idiots.

I just wonder how bad it will have to get before they decide we need to be reigned in. Going by how the 20th Century went, if it's anything like that we're in for a rough time before we get there--like, to a place where software engineers have to act like, well, engineers.

The computer does not see things like you see things ;)

It's shocking. I can't understand why people repeatedly fall for it.

Made my day lol. Brilliant.

FaaS are really interesting, because the tradeoffs seem "jagged" (to use a word I learned recently). Like, IIUC this is more or less what's happening when a cloud function is invoked:

1. Some event triggers the function (HTTP request hits load balancer or whatever)

2. A VM is loaded from NVMe (possibly on another computer, and transferred over the network) into RAM

3. That VM boots up

4. The function code runs

5. The VM is terminated

Naively, this probably takes a lot more RAM (not to mention all the other computing resources) than to just have another HTTP handler sitting idle on some computer somewhere.

So this is only a reasonable choice if doing this intensive computation for a short amount of time, and using a large amount of resources for that short amount of time, is somehow better than doing a much smaller amount (maybe none at all) of computing over a longer amount of time. It's kind of hard to tell what the actual conditions are that merit one solution over the other. The other layer is the cloud pricing model, but that's a whole other can of worms..

I have yet to encounter a system that didn't have at least some computers (or cloud instances) that always need to be running. Those computers are almost always a bit overprovisioned (not using 100% of their CPU or RAM or IO all the time). So it has always seemed to me that instead of running a cloud function, it would be cheaper in terms of compute resources to execute the code from step 4 above on one of those machines.

Obviously we have cloud functions for a reason, people like them and find them useful. But is there ever a regime where they're optimal?

EDIT: I guess I've implicitly assumed the code in step 4 is doing a small amount of work. Maybe there's a goldilocks zone of work-per-invocation where this model is more optimal? But then of course there's always the standard executor pool model, and that way you don't have to spin up a VM per task invocation...

I remember reading somewhere about an HPC "rule of thumb" that was like 1GB RAM/1GHz of processing is about the right amount for most applications.. Don't recall where that came from or what the exact reasoning was. That kind of aligns with 2GB/core.. Was that a thing?

I swapped the engine on my lawn mower ~3yr ago because it no longer had any compression and was very hard to start. The new engine has an electric starter. I didn't have time to fabricate a proper battery tray so I ratchet strapped a 2x4 to the gas tank and strapped the battery to that. It stayed that way until last month when I finally had time to weld up a battery tray.

Sometimes when some janky p.o.s. solution works well enough, it truly is good enough.

Machine telemetry data (logs, metrics, traces) are shipped around as highly redundant utf-8 strings. I'm sure there are plenty of other examples of 2+ orders of magnitude improvements that aren't even low hanging fruit, they're basically groundfall. But picking them up may nonetheless require quite a bit of effort, and a major reevaluation of corporate values. Maybe the RAM crunch will be sufficient incentive? Idk.

EDIT: It's also possible that the era of hypergrowth might end. Most of the world's addressable eyeballs are already being monetized. There's no new coal seam to mine. If tech becomes more about doing the same with less, and less about explosively growing at all costs, we'll probably see more efficient systems. And fewer people building them. I think this kind of larger systemic change would be more likely to drive efficiency improvements than the (probably more temporary) RAM crunch.

> The main place RAM usage is going to get optimized is on the server side

Agree, but it's going to be a long, hard, potentially ultimately unsuccessful uphill slog. I think the main obstacle is basically 2 decades of inertia. Approximately everyone believes that server side software needs to scale horizontally. So we optimize systems for this property--I can make it work better by just throwing more computers at it. This was a lot more important ca. 2004 (MapReduce) than it is ca. 2026. For resiliency and redundancy, how many nodes does your service actually need? If you're running more than that number, do you have some kind of justification as to why? This is the mindset we need to try to move towards, but it's very different from how we currently do it.

EDIT: Another good question to ask, which we don't do enough: "does it really need to be a separate service?" Using data in RAM that's already been allocated is probably better than allocating more and shipping data over the network...

> The analysis that I want is on a studied cost/value basis somehow.

Absolutely. The AI ROI question is the entire ballgame. Earlier today I heard the term "jagged technology" for the first time, and it's an extremely good one. Given how jagged AI appears to be, anecdotal reports are all over the place. It also doesn't help that it appears heavily using this technology can affect people psychologically. The information environment for someone who wants to know what this tool is good for and what it isn't is.. treacherous. Some clear, objective grounding would really help.

I stand corrected! That's very interesting.

Out of touch leadership and management combined with a culture of self-congratulatory "innovation" where companies just plagiarize eachother in an endless loop... It doesn't surprise me at all, tbh.

> Only when you implement by hand can you feel if the implementation of your design is painful or not, and only this signal can tell you if your design if truly good or not.

I think this touches obliquely on a point I keep coming back to, that one of the most important things a codebase does is to communicate ideas about how a process should work. Yes, it also produces some binary that runs on a bunch of servers or whatever, but that's a really temporary, ephemeral artifact. The lasting thing is the idea. Making your ideas (expressed in code) easy to understand, easy to work with, and easy to evolve in time is the art of software engineering. I 100% agree, from my own experimentation with LLMs, glancing at something a model has produced and checking that it has some test coverage isn't enough to know whether it's well-engineered. You'll only find out later when you try to work with the code.