I do think there's something to be said for the big difference in the current abstraction layer jump - the loss of determinism. It is meaningfully different.
I think I agree, but could you expand?
My point is humans have made progress on understanding systems without clean determinism. Genetics and cosmology is technically deterministic, practically speaking, but I don’t think that’s accessible to pre-modern societies. Instead, trends and tendencies were observed, acted and capitalized on.
On the other hand, an LLM "knows" what a benchmark is, and is capable of detecting benchmark-like scenarios. A prompt that appears reliable in testing may fail unexpected in the real world. In this way, an LLM is like a stock market (also composed of intelligent agents): the act of experimenting on it changes its properties. Such systems are never reliably predictable.
Computing has never really been deterministic all the way down. Storage gets corrupted, RAM has soft errors, networks drop/reorder packets, schedulers race, caches go stale, distributed systems partition, query planners change plans, ...
Obviously those become tolerable because we've developed layers of contracts and understanding the bounds around them. Error correcting codes, checksums, retries, consensus, idempotency, false-positive rates, SLAs, etc.
If the abstraction is “delegate this task to a junior engineer, analyst, lawyer, designer, or support rep,” you dont expect deterministic behavior. There's review, constraints, escalation, checklists, tests, and accountability.
To be fair, this isn’t a bad description of human vision.
Determinism is way undervalued.
So now instead of improving the tools of biology so we can actually understand it deeply - we increase complexity of IT so we have to rely on muddy, side-effecty tools of biology to try to infer some of the properties of the systems we made. That's depressing.
I do not understand the need to argue that monkeys are better than screwdrivers at screwing. Just let the monkey be the best version of a monkey.
I would argue that management is a better discipline to pull from for employing LLMs; It is better equipped to deal with non-determinism and going completely off the rails.
Non-determinism combined with bugginess?
That's a terrible combination. It is impossible to gradient-descent your way into a working prompt.
I would say it is unambiguously defined by the problem you're solving. For example a spellchecker is inherently probabilistic so you shouldn't need hard coded rules here like we did for years. A "bug" then would be considered "weak accuracy" not "crash" or "incorrect behavior", as it might be the case in any other layer.
They should add a term for it, maybe "embarrassingly nondeterministic"
Sure. You gave an example where it can work. Another example is something like a PCB or chip layout engine.
That particular domain (layout) is NP-complete. You'll never have an exhaustive brute force search for the optimal layout, but...
You can subsequently easily check whether the produced layout meets all your acceptance criteria or not.
Another example that successfully utilizes non-determinism for good outcomes is the application of genetic algorithms to things like antenna design.
This works because (a) you have a relatively cheap fitness test; and (b) as with real evolution, the mix of combining working designs and randomly introducing mutations often eventually produces outstanding results.
Presumably, if you applied genetic algorithm techniques to, e.g., creating your LLM prompts, you could also get good results, but that would probably quickly get expensive in terms of tokens.
So we're left with people just semi-randomly modifying prompts in order to try to tweak results.
When it works, it can be amazing. When it doesn't work, it's like a brick wall.
I like your "embarrassingly nondeterministic" term, but I somewhat disagree with:
> A "bug" then would be considered "weak accuracy" not "crash" or "incorrect behavior",
When a lawyer asks an LLM for citations of cases that support his position, he is arguably doing something stupid, because embedding an assertion such as "Show me cases which support X" is just asking for hallucinatory trouble with many current LLMs.
Nonetheless, I submit that hallucinations are, by definition, "incorrect behavior" and not merely "weak accuracy."
Now, nondeterminism could certainly be useful to the lawyer, in that it could help an LLM make connections that LexisNexis might not have in their database. So asking an LLM for help with legal issues is theoretically not an insane thing to do, but the results need to be checked very carefully.
Chip design, etc would go to the latter category which is actually a huge space for innovation regardless of how you do it, while I was focusing more on the former which I think a lot of discussion seems to be concerned about.
Now if you are "running" your "generated code" that's where people will have strong opinions because it conflates the two completely different processes in the worst way possible.
For example, the layout software vendors provide a solution that both uses non-determinism in their final product, and also checks the results against whatever criteria you give it.
If it tells you it will work, it's probably not lying, and if it tells you it won't work, it's almost certainly not lying.
So in your own creative process of building a board, you can rely on that.
Many LLMs don't have this property at all, and honestly, it really doesn't matter whether you are using the LLM to craft a legal document or to craft code; if you don't check the output yourself, you're possibly in for a world of hurt.
> Now if you are "running" your "generated code" that's where people will have strong opinions because it conflates the two completely different processes in the worst way possible.
I think a primary driver of the attractive nuisance here is that, because the execution of the resultant code is itself a deterministic process (even if the process that produced that code it isn't) people think that they can tame the process, by some combination of automated testing and looped automated reprompting.
At some level, they may not be wrong -- computer chips themselves are built on top of stochastic atomic processes, and google famously proved in its early days that it could have reliable processes on top of shitty commodity hardware.
But one key difference is that the nondeterminism in the atomic processes or in the non-ECC memory of the white box computers is uncorrelated.
And another key difference is that, unlike the layout software vendors, so far the LLM vendors seem unable or unwilling to properly self-check their own outputs.
To me, this is kind of the canary in the coal mine. You would think that the LLM vendors have every incentive to weed out bad results, that they, more than anybody else, have the understanding of root causes and probabilities that a particular output might be bad, and have more than sufficient resources to fix this, if it's not an intractable problem.
So Occam's Razor says that (currently) this is still an intractable problem.