31,384 karma · joined September 24, 2008
I suspect a SQL injection attack, a XSS attack, and a prompt injection attack are not viewed as legally distinct matters. Though of course, this is not a matter of case law... yet ;)
Take the article's example "send today’s schedule to my boss John Doe" where the product isn't entirely guarded by the Plan-Then-Execute model (injections can still mutate email body).
But if you combine it with the symbolic data store that is blind, it becomes more like:
"send today's schedule to my boss John Doe" -->
$var1 = find_contact("John Doe")
$var2 = summarize_schedule("today")
send_email(recipient: $var1, body: $var2)
It simply invokes the final tool, which is deterministic and just reads from the shared var store. In this case you're pretty decently protected from prompt injection.
I suppose though this isn't that different from the "Code-Then-Execute" pattern later on...
In the early stages of LLMs yes ("get me all my calendar events for next week and output in JSON format" and pray the format it picks is sane), but nowadays there are specific model features that guarantee output constrained to the schema. The term of art here is "constrained decoding".
The structuring is also a bit of a dark art - overall system performance can improve/degrade depending on the shape of the data structure you constrain to. Sometimes you want the LLM to output into an intermediate and more expressive data structure before converting to a less expressive final data structure that your deterministic piece expects.
> "Are there "mediator" agents with some reliability AND some flexibility?"
Pretty much, and this is basically where "agentic" stuff is at the moment. What mediates the LLM's outputs? Is it some deterministic system? Is it a probabilistic system? Is it kind of both? Is it a machine? Is it a human?
Specifically with coding tools, there seems like the mediator(s) are some mixture of sticklers (compiles, tests) and loosey-goosey components (other LLMs, the same LLM).
This gets a bit wilder with multimodal models too: think about a workflow step like "The user asked me to make a web page that looks like [insert user input here], here is my work, including a screenshot of the rendered page. Hey mediator, does this look like what the user asked for? If not, give me specific feedback on what's wrong."
And then feed that back into codegen. There has been some surprisingly good results from the mediator being a multimodal LLM.
More or less, though the agent doesn't have to be deterministic. There's a sliding scale of how much determinism you want in the "overseer" part of the system. This is a huge area of active development with not a lot of settled stances.
There's a lot of work being put into making the overseer/agent a LLM also. The neat thing is that it doesn't have to be the same LLM, it can be something fine-tuned to specifically oversee this task. For example, "After code generation and build/test has finished, send the output to CodeReviewerBot. Incorporate its feedback into the next round of code generation." - where CodeReviewerBot is a different probabilistic model trained for the task.
You could even put a human in as part of the agent: "do this stuff, then upload it for review, and continue only after the review has been approved" is a totally reasonable system where (part of) the agent is literal people.
> "And there's a asymmetry in strictness, i.e. LLM --> agent funnels probabilistic output into 1+ structured fields, whereas agent --> LLM can be more freeform (stderr plaintext). Is that right?"
Yes, though some flexibility exists here. If LLM --> deterministic agent, then you'd want to squeeze the output into structured fields. But if the agent is itself probabilistic/a LLM, then you can also just dump unstructured data at it.
It's kind of the wild west right now in this whole area. There's not a lot of common wisdom besides "it works better if I do it this way".
> "1) how does the LLM know where to put output tokens given more than one structured field options?"
Prompt engineering and a bit of praying. The trick is that there are methods for ensuring the LLM doesn't hallucinate things that break the schema (fields that don't exist for example), but output quality within the schema is highly variable!
For example, you can force the LLM to output a schema that references a previous commit ID... but it might hallucinate a non-existent ID. You can make it output a list of desired code reviewers, and it'll respect the format... but hallucinate non-existent reviewers.
Smart prompt engineering can reduce the chances of this kind of undesired behavior, but given that it's a giant ball of probabilities, performance is never truly guaranteed. Remember also that this is a language model - so it's sensitive to the schema itself. Obtuse naming within the schema itself will negatively impact reliability.
This is actually part of the role of the agent. "This code reviewer doesn't exist. Try again. The valid reviewers are: ..." is a big part of why these systems work at all.
> "2) Is this loop effective for projects from scratch? How good is it at proper design (understanding tradeoffs in algorithms, etc)?"
This is where the quality of the initial prompt and the structure of the agent comes into play. I don't have a great answer for here besides that making these agents better at decomposing higher-level tasks (including understanding tradeoffs) is a lot of what's at the bleeding edge.
1 - The LLM is in charge and at the top of the stack. The deterministic bits are exposed to the LLM as tools, but you instruct the LLM specifically to use them in a particular way. For example: "Generate this code, and then run the build and tests. Do not proceed with more code generation until build and tests successfully pass. Fix any errors reported at the build and test step before continuing." This mostly works fine, but of course subject to the LLM not following instructions reliably (worse as context gets longer).
2 - A deterministic system is at the top, and uses LLMs in an otherwise-scripted program. This potentially works better when the domain the LLM is meant to solve is narrow and well-understood. In this case the structure of the system is more like a traditional program, but one that calls out to LLMs as-needed to fulfill certain tasks.
> "I’m not understanding how a probabilistic machine output can reliably map onto a strict input schema."
So there are two tricks to this:
1 - You can actually force the machine output into strict schemas. Basically all of the large model providers now support outputting in defined schemas - heck, Apple just announced their on-device LLM which can do that as well. If you want the LLM to output in a specified schema with guarantees of correctness, this is trivial to do today! This is fundamental to tool-calling.
2 - But often you don't actually want to force the LLM into strict schemas. For the coding tool example above where the LLM runs build/tests, it's often much more productive to directly expose stdout/stderr to the LLM. If the program crashed on a test, it's often very productive to just dump the stack trace as plaintext at the LLM, rather than try to coerce the data into a stronger structure and then show it to the LLM.
How much structure vs. freeform is very much domain-specific, but the important realization is that more structure isn't always good.
To make the example concrete, an example would be something like:
[LLM generates a bunch of code, in a structured format that your IDE understands and can convert into a diff]
[LLM issues the `build_and_test` tool call at your IDE. Your IDE executes the build and tests.]
[Build and tests (deterministic) complete, IDE returns the output to the LLM. This can be unstructured or structured.]
[LLM does the next thing]
IMO the defining feature of an agent is that the LLM's behavior is being constrained or steered by some other logical component. Some of these things are deterministic while others are also ML-powered (including LLMs).
Which is to say, the LLM is being programmed in some way.
For example, prompting the LLM to build and run tests after code edits is a great way to get better performance out of it. But the idea is that you're designing a system where a deterministic layer (your tests) is nudging the LLM to do more useful things.
Likewise many "agentic reasoning" systems deliberately force the LLM to write out a plan before execution. Sometimes these plans can even be validated deterministically, and the LLM forced to re-gen if plan is no good.
The idea that the LLM is feeding itself isn't inaccurate, but misses IMO the defining way these systems are useful: they're being intentionally guided along the way by various other components that oversee the LLM's behavior.
1 - Using coding tools in a context/language/framework you're already familiar with.
This one I have been having a lot of fun with. I am in a good position to review the AI-generated code, and also examine its implementation plan to see if it's reasonable. I am also able to decompose tasks in a way that the AI is better at handling vs. giving it vague instructions that it then does poorly on.
I feel more in control, and it feels like the AI is stripping away drudgery. For example, for a side project I've been using Claude Code with an iOS app, a domain I've spent many years in. It's a treat - it's able to compose a lot of boilerplate and do light integrations that I can easily write myself, but find annoying.
2 - Using coding tools in a context/language/framework you don't actually know.
I know next to nothing about web frontend frameworks, but for various side projects wanted to stand up some simple web frontends, and this is where AI code tools have been a frustration.
I don't know what exactly I want from the AI, because I don't know these frameworks. I am poorly equipped to review the code that it writes. When it fails (and it fails a lot) I have trouble diagnosing the underlying issues and fixing it myself - so I have to re-prompt the LLM with symptoms, leading to frustrating loops that feel like two cave-dwellers trying to figure out a crashed spaceship.
I've been able to stand up a lot of stuff that I otherwise would never have been able to, but I'm 99% sure the code is utter shit, but I also am not in a position to really quantify or understand the shit in any way.
I suppose if I were properly "vibe coding" I shouldn't care about the fact that the AI produced a katamari ball of code held together by bubble gum. But I do care.
Anyway, for use case #1 I'm a big fan of these tools, but it's really not the "get out of learning your shit" card that it's sometimes hyped up to be.
Which may be a desirable policy outcome for state rail agencies - but this is a private venture!
I think cost is an under-appreciated aspect of this. You're carrying 5-10x fewer passengers per-train, at greater cost (the cost of turning over a stateroom is many times higher than cleaning a coach seat, along with linens, food, etc.), on very expensive custom equipment that isn't suitable for other uses.
There seem to be two "major" (really heavy scare quotes here) players in the US private sleeper service scene. Dreamstar IMO is the more promising of the two (heavy caveat that this is relative to each other, not absolute odds) by realizing the only way to make the economics work is the ability to charge $$$$$ for tickets.
The other (Lunatrain) IMO is just out to lunch, with a claimed focus on affordability. None of the above leads to affordable tickets.
For example, I had occasion to chat with a relative who's still in high school recently, and was curious what the situation was in their classrooms re: AI.
tl;dr: LLM use is basically universal, but ChatGPT is not the favored tool. The favored tools are LLMs/apps specifically marketed as study/homework aids.
It seems like the market is fine with seeking specific LLMs for specific kinds of tasks, as opposed to some omni-LLM one-stop-shop that does everything. The market has already and rapidly moved beyond from ChatGPT.
Not to mention I am willing to bet that Gemini has radically more usage than OpenAI's models simply by virtue of being plugged into Google Search. There are distribution effects, I just don't think OpenAI has the strongest position!
I think OpenAI has some first-mover advantage, I just don't think it's anywhere near as durable (nor as large) as you're making it out to be.
The more interesting, novel, and useful work you wrap the LLM in the more defensible your pricing will be.
That said I think this can describe a lot of agentic code tools - the entire point is that you're not just talking to the raw LLM itself, you're being intermediated by a bunch of useful things that are non-trivial.
I see this with Anthropic most - they seem to have multiple arms in multiple lines of business that go up the abstraction ladder - Claude Code is just one of them. They seem to also be in the customer service automation business as well.
[edit] I think a general trend we're going to see is that "pure" LLM providers are going to try to go up the abstraction ladder as just generating tokens proves unprofitable, colliding immediately with their own customers. There's going to be a LOT of Sherlocking, and the LLM providers are going to have a home field advantage (paying less for inference, existing capability to fine-tune and retrain, and looooooots of VC funding).
If you forget to dump the key (or if the deletion is not clean) then you've got an absolute whopper of a privacy breach.
Also worth noting that you can't dump the key until the computation is complete, so you'd need to persist the key in some way which opens up another failure surface. Again, if it can't be avoided that's one thing, but if it can you'd rather not have the key persist at all.
Because the other side may not be listening when the compute is done, and you don't want to cache the result of the computation because of privacy.
The sequence of events is:
1. Phone fires off a request to the backend. 2. Phone waits for response from backend.
The gap between 1 and 2 cannot be long because the phone is burning battery the entire time while it's waiting, so there are limits to how long you can reasonably expect the device to wait before it hangs up.
In a less privacy-sensitive architecture you could:
1. Phone fires off request to the backend. Gets a token for response lookup later. 2. Phone checks for a response later with the token.
But that requires the backend to hold onto the response, which for privacy-sensitive applications you don't want!
Sure, but I would argue that the UX is the product, and that has radically improved in the past 6-12 months.
Yes, you could have produced similar results before, manually prompting the model each time, copy and pasting code, re-prompting the model as needed. I would strenuously argue that the structuring and automation of these tasks is what has made these models broadly usable and powerful.
In the same way that Apple didn't event mobile phones nor touchscreens nor OSes, but the specific combination of these things resulted in a product that was different in kind than what came before, and took over the world.
Likewise, the "putting the LLM into a structured box of validation and automated re-prompting" is huge! It changed the product radically, even if its constituent pieces existed already.
[edit] More generally I would argue that 95% of the useful applications of LLMs aren't about advancing the SOTA model capabilities and more about what kind of structured interaction environment we shove them into.
The pitch is that if you do this bucketization, the overall orchestrator can intelligently pick the bucket to use, but the idea is that at any moment the LLM is only exposed to a limited set of tools.
As opposed to the more pie-in-the-sky idea that given N tools (where N is very very large) the LLM can still accurately tool-select without any developer intervention. This seems pretty far off at this point.
Mostly this seems like an end-run around tool calling scalability limits. Model performance degrades heavily if the field of possible tools gets too large, so you insert a component into the system that figures out what tools should be in-scope, and make only those available, to get reliability higher.
In terms of "why outsource this" it seems like the idea is that their orchestration agent would be better than a cruder task state machine that you would implement yourself. Time will tell if this assertion is true!
I would contest this statement very vigorously. Siri has been a disappointment but nearly every single Apple product has a load-bearing reliance on an AI/ML feature.
I'm continually frustrated by the (relatively recent, since LLMs) tic where just because a ML model isn't conversing with you or otherwise making its presence impossible to ignore, that it isn't delivering a massive amount of value. It is frustrating from laypeople, it is extra frustrating from industry insiders.
Just a few examples off the top of my head:
- AirPods with ANC and spatial audio. Both headline features of these products that are 100% AI/ML.
- Watch with heart arrhythmia detection, automatic workout detection, fall detection, etc. All are headline features (some literally life-saving) and are 100% AI/ML.
- iPhones have high-profile features that are 100% AI/ML: automatic car crash detection. Others are more subtle but IMO substantial differentiators - such as automatic image enhancements out-of-camera.
Again, I know in the age of ChatGPT we seem to have twisted ideas of what ML is, but "AI/ML" is not synonymous with LLM.
Generally agreed. I'm generally very bearish on large-scale rewrites for this reason + political/managerial reasons.
The trick with any organization that wants to remain employed is demonstrating progress. "Go away for 3 years while we completely overhaul this." is a recipe for getting shut down halfway through and reassigned... or worse.
A rewrite, however necessarily, must always be structured as multiple individual replacements, each one delivering a tangible benefit to the company. The only way to stay alive in a long-term project is to get on a cadence of delivering visible benefit.
Importantly doing this also improves your odds of the rewrite going well - forcing yourself to productionize parts of the rewrite at a a time validates that you're on the right track.
I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.
The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on.
Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!
Specifically, data is only worth as much as the buyer is able to exploit it to make money. Meta and Google have figured out how to immensely profit off of personal data - but that's because they own the largest ad networks on the planet.
Few other players can leverage personal data to this financial extent. More importantly, the players that can leverage personal data for immense profit also collect their own data, and so externally-sourced data is only of marginal utility.
And you see this in the discourse around this issue. We theorize for example that some grocery chain would "pay a lot" for personal data. But to what end? So they can offer just-in-time enticements to go to the store?
A lot of these supposed "applications" of personal data seem pretty flimsy, and even where they are genuinely useful, their monetary worth is marginal.
Data is worth something, but it isn't worth nearly as much as widely believed.
The main problem with Uber/taxi quality is that the responsibility for cleanliness is on individual drivers, with widely varying results as a natural outcome. In fact a lot of problems with Ubers and taxis is downstream from the ownership and responsibility model.
The advantage of something like Waymo is that the responsibility is now on Waymo itself.
Worth noting that this is a problem in other lines of business: it's harder to ensure quality in franchises vs. stores operated directly by the brand. The more independent parties you have in the mix the more incentives become misaligned and the fewer levers you have available to ensure compliance to some standard.
This also isn't impossible to solve with human drivers, because ultimately this isn't a technological problem but an organizational one. Livery car services where drivers are employees (as opposed to independent owner-operators contractors) can centralize cleaning and training, and have more means to ensure compliance to a standard.
The "downside" of such a model is that there are many more laws to ensure you can't shovel your own expenses onto the employee.
Overall I feel like our industry has lost the plot to a large degree. Hype has always to some degree exceeded the merits of the technology-of-the-month, but the last few cycles have been truly extraordinary in terms of the gap between the breathless hype and the reality of the tech. It's LLMs now but before it was crypto.
It just seems like we're stagnating as an industry, and rather focus our efforts on the hard R&D needed to reach the next Big Thing, we've decided it's much easier just to focus on cults of personality combined with vast over-hypedness.
The overwhelming goal of a company's legal department is "don't get sued", followed by "if sued, lose as little money/leverage as possible".
In general the lawyer in the room is going to be far more risk-averse than the engineers, product people, sales people, or marketers.
The trick is that outside of some limited circumstances the legal department at companies are not the final say. Many lawyers who "go in house" (i.e., quit a private outside firm and go work directly for a company) find this frustrating. They come into a room, say "don't do that", and then a few weeks/months/years later someone did it and now they have to prepare for a lawsuit.
- "You have already received benefit X for your investment in our team, and X has been well worth it. Continued investment in us will yield benefit Y."
- "Even if the project is terminated early, or later stages become blocked due to business or technical impracticalities, the company will have gained a tangible benefit already in the form of X. There is a payout, even if it is not the full thing we wanted."
- "The intermediate products validate the technology/product/business model, such that every incremental deliverable means an overall reduction in risk to the final goal."
This applies also to ML - it applies to all tech projects, though yeah, it's harder in ML. But not figuring out the intermediate products is not an option though - your stuff will get killed prematurely if you don't.
The trick with ML is not to promise "98% precision and 92% recall by Q4", it's to figure out what kind of product is shippable with lower precision and recall. Or perhaps a stepping-stone model that allows some simpler use case, but gives you progress towards the greater goal.
It's always case-specific, but as a ML team you do need to figure out what your intermediate checkpoints are. You need to demonstrate not only progress, but that your progress is contributing to the company's goals.
> "That did introduce tension for our team because we were supposed to be taking experimental bets for the platform’s future. These bets couldn’t be baked into product without hacks or shortcuts in the typical quarter as was the expectation."
If I can pump one learning into engineers' and PMs' heads it's this: intermediate deliverables are not optional no matter how cutting-edge your team is.
You will never succeed if your pitch to leadership is "give us a budget for the next N years and expect no shippable products until the end of N years". Even if you get approved somehow at the beginning, there's a 99.5% chance your team/project will be killed before you get to N years.
Again, once again for the audience in the back: there is no such thing as a multi-year project without convincing, meaningful intermediate deliverables.
To clarify, that doesn't mean "don't have multi-year roadmaps", it means "your multi-year roadmaps must deliver wins at a consistent cadence".
Understanding this will carry you a lot further in the industry.
As a fairly cutting-edge R&D team part of your job is to figure out what slice of this is shippable (and worth shipping). If you're coming up empty you are not ready to pitch this to execs.
"Shopping with your voice" never took off despite many attempts. The contribution towards subscription services like Audible and Amazon Music was not substantial enough to warrant the massive R&D investment. The business unit never found any other sources of convincing revenue.
Every other decision is downstream from that unresolved tension.