Building AI products in the probabilistic era
giansegato.com
giansegato.com
The central analogy of the article is entirely bogus.
This article does not rise to the level of being wrong.
Amazing
Don't you mean from determinism to nondeterminism?
> There is not one physicist on earth that would ever exhort you to use quantum theory where classical theory will do.
That's being practical.
No, I mean exactly what I said. Given a system's state, one evolves the state using wave equation du jour, nondeterminism does not occur.
“Think about it: we’ve built a special kind of function F' that for all we know can now accept anything — compose poetry, translate messages, even debug code! — and we expect it to always reply with something reasonable.”
This forms the axiom from which the rest of this article builds its case. At each step further fuzzy reasoning is used. Take this for example:
“Can we solve hallucination? Well, we could train perfect systems to always try to reply correctly, but some questions simply don't have "correct" answers. What even is the "correct" when the question is "should I leave him?".”
Yes of course relationship questions don’t have a “correct” answer. But physics questions do. Code vulnerability questions do. Math questions do. I mean seriously?
The most disturbing part of my tech career has been witnessing the ability that many highly intelligent and accomplished people have to apparently fool themselves with faulty yet complex reasoning. The fact that this article is written in defense of chatbots that ALSO have complex and flawed reasoning just drives home my point. We’re throwing away determinism just like that? I’m not saying future computing won’t be probabilistic but to say that LLMs are probabilistic, so they are the future of computing can only be said by someone with an incredibly strong prior on LLMs.
I’d recommend Baudrillards work on hyperreality. This AI conversation could not be a better example of the loss of meaning. I hope this dark age doesn’t last as long as the last one. I mean just read this conclusion:
“It's ontologically different. We're moving away from deterministic mechanicism, a world of perfect information and perfect knowledge, and walking into one made of emergent unknown behaviors, where instead of planning and engineering we observe and hypothesize.”
I don’t actually think the above paragraph makes any sense, does anyone disagree with me? “Instead of planning we observe and hypothesize”?
That’s called the scientific method. Which is a PRECURSOR to planning and engineering. That’s how we built the technology we have today. I’ll stop now because I need to keep my blood pressure low.
But as per Gödel's incompleteness theorem and the Halting Problem, math questions (and consequently physics and CS questions) don't always have an answer.
> Yes of course relationship questions don’t have a “correct” answer. But physics questions do. Code vulnerability questions do. Math questions do. I mean seriously?
Can you say more? It seems to me the article says the same thing you are.
> I don’t actually think the above paragraph makes any sense, does anyone disagree with me? “Instead of planning we observe and hypothesize”?
I think the author is drawing a connection to the world of science, specifically quantum mechanics, where the best way to make progress has been to describe and test theories (as opposed to math where we have proofs). Though it's not a great analog since LLMs are not probabilistic in the same way quantum mechanics is.
In any case, I appreciated the article because it talks through a shift from deterministic to probabilistic systems that I've been seeing in my work.
Yes, LLMs are a bit of a new beast in terms of the use of stochastic processes as producers—but we do know how to deal with these systems. Half the "novelty" is just people either forgetting past work or being ignorant of it in the first place.
We also see this in cryptocurrencies. The path to forgetting is greased by the presence of money and fame, and at some later time they are eventually forced to "discover" the same ancient problems they insisted couldn't possibly apply.
Any specific piece of writing you can recommend? I tried reading Simulacra and Simulation (English translation) a while ago and I found it difficult to follow.
Current and past computing has always been probabilistic in part, doesn't mean it will become 100% so. Almost all of the implementationof LLMs is deterministic except the part that is randomized. Its output is used in the same way. Humans combine the two approaches as well. Even reality is a combination of quantum uncertainty at a low level and very deterministic physics everywhere else.
> We're moving away from deterministic mechanicism, a world of perfect information and perfect knowledge, and walking into one made of emergent unknown behaviors, where instead of planning and engineering we observe and hypothesize.
The hype machine always involves pseudo-scientific babble and this is a particularly cringey example. The idea that seems to be promoted, that AI will be god like and therein we'll find all truth and knowledge is beyond delusional.
It a tool, like all other tools. Just like we see faces in everything we're also very susceptible to language (especially our own, consumed and regurgitated back to us) from a very neat chatbot.
AI hype is borderline mass hysteria at this point.
Thanks for confirming. As crazy as the chatbot fanatics are, hearing them talk makes ME feel crazy.
Plenty of front-running companies have hired plenty of…not-solid or excessively imaginative data scientists
From what the other comments say, this one seems to lack a grounding in science itself, which frankly is par for the course depending on their background
I don't think the author is arguing that all computing is going to become probabilistic. I don't get that message at all - in fact they point out many times that LLMs can't be trusted for problems with definite answers ("if you need to add 1+1 use a calculator"). Their opening paragraph was literally about not blindly trusting LLM output.
> I don’t actually think the above paragraph makes any sense, does anyone disagree with me?
Yes - it makes perfect sense to me. Working with LLMs requires a shift in perspective. There isn't a formal semantics you can use to understand what they are likely to do (unlike programming languages). You really do need to resort to observation and hypothesis testing, which yes, the scientific method is a good philosophy for! Two things can be true.
> the use of formal mathematical notation just adds insult to injury here
I don't get your issue with the use of a function symbol and an arrow. I'm a published mathematician - it seems fine to me? There's clearly no serious mathematics here, it's just an analogy.
> This AI conversation could not be a better example of the loss of meaning.
The "meaningless" sentence you quote after this is perfectly fine to me. It's heavy on philosophy jargon, but that's more a taste thing no? Words like "ontology" aren't that complicated or nonsensical - in this case it just refers to a set of concepts being used for some purpose (like understanding the behaviour of some code).
Harmless enough if you are just making images for fun. But probably not an ideal workflow for real work.
Hmm. Ideal is rarely an option, so I have to assume you are being careful about phrasing.
Still, despite it being a black box, one can still tip the odds on one's favor, so the real question is what is considered 'real work'? I personally would define that as whatever they you are being paid to do. If that premise is accepted, then the tool is not the issue, despite its obvious handicaps.
It can be, and usually is by default. If you set the seeds to deterministic numbers, and everything else remains the same, you'll get deterministic output. A slot machine implies you keep putting in the same thing and get random good/bad outcomes, that's not really true for Stable Diffusion.
You yourself acknowledge someone can better than another on getting good results from Stable Diffusion, how is that in any way similar to slot machine or rolling the dice? The point of those analogies is precisely that it doesn't matter what skill/knowledge you have, you'll get a random outcome. The same is very much not true for Stable Diffusion usage, something you seem to know yourself too.
Even if the assumption is correct, forcing a probabilistic system on a strongly deterministic society won't end well. Maybe for society, but mostly for the companies drumming up their probabilistic systems and their investors.
Also, anyone who wants to make money probabilistically is better off going to the casino. Baccarat is a good one. European Roulette also has a better house margin than chatGPT's error margin.
Thank you for saying that!
I read it as: "our product is fickle and unreliable and you have to get used to it and love it because we tell you that is the future".
But it's not the future, it's just one of many possible futures, and not one I and a large part of society wants to be a part of. These "leaders" are talking and talking but they are just salemen, trying to frame what they are seling you as good or inevitable. It's not.
Look for example, at the Ex CEO of Github and his clownish statements:
- 2nd of August: Developers, either embrace AI or leave the industry https://www.businessinsider.com/github-ceo-developers-embrac...
- 11th of August: Resigns. https://www.techradar.com/pro/github-ceo-resigns-is-this-the...
Tell me this is not pitiful, tell me this is the person I gotta believe in and who knows what's the future of tech?
Tell me I gotta believe Sama when he tells me for the 10th time that AGI is nearly there when his latest features were "study mode", "announcing OpenAI office suite" and ChatGPT5 (aka ChatGPT4.01).
Or Musk and his full self driving cars which he promised since 2019? The guy who bought Twitter, tanked its value in half so he can win the election for Trump and then got himself kicked out of the government? The guy making Nazi salutes?
Are those the guys telling what's the future and why are we even listening to them?
Elon failed his boring tunnel
Elon is failing his rocket ventures so much that he had to try to gut NASA so he remains competitive
Elon didn't achieve full self driving cars and is now sued, hopefully to oblivion, for the fatal car crashes his half baked tech created.
Zuck didnt achieve the Metaverse.
Zuck didn't achieve widespread VR adoption.
It's extremely funny to see these pitiful billionaires speak with confidence of a future they want and pour billions in, but then fail to achieve. Their failures are multiplied by how much money, resources, time and people they threw at the problem and still failed. Watching these men-chileren unravel in front of our very eyes is infinite comedy, we are talking going Nazi, getting drug addicted, going Bro?, MMA training, one can't make this stuff up, their actions read like made up satire in the Onion.
They are society's laughing stock, because despite all their vast resources they seemingly don't want to do anything good for society and instead prefer to race to see who can fly their pitiful rockets to Mars first or sell more crap to people who don't need it, or yet influence another election and elect the next Mussolini wannabe. Those guys are the ultimate clowns.
The key is building systems that are transparent about their confidence levels and gracefully handle edge cases.
The companies that will win in AI aren't the ones with perfect algorithms - they're the ones who design for human understanding and real-world messiness.
It is possible (and often desirable) to use different WOFF fonts for italics, and they can look quite different from the standard font.
Ask ChatGPT to fix it.
ps. On a side note, I love that vibe has come back as a word. Feels like the 60s.
in the real world we expect consistent behavior from engineered systems, and building everything on top of a gpu powered bullshit engine isn't going to give you that
its a nice narrative though, maybe you should pick up writing sci-fi
The entire article is a business pseudo-philosophy word salad equivalent of tiktok 'I'm not like other (men/women)' posts.
The rigidness and near-perfect reliability of computer software is the unusual thing in human history, an outlier we’ve gotten used to.
Ordered approximately by recency:
Banking? Clocks? Roman aqueducts? Mayan calendars? The sun rising every day? Predictable rainy and dry season?
How is software the outlier here?
People on HN regularly claim that LLMs are useless if they aren’t 100% accurate all the time. I don’t think this is true. We work around that kind of thing every day.
With your examples:
- Before computers, fraud and human error was common in the banking system. We designed a system that was resilient against this and mostly worked, most of the time, well enough for most purposes even though it was built on an imperfect foundation.
- Highly precise clocks are a recent invention. For regular people 200 years ago, one person’s clock would often be 5-10 minutes off from someone else’s. People managed to get things done anyway.
I’ll grant you that Roman aqueducts, seasons and the sun are much more reliable than computers (as are all the laws of nature).
A lot of the stuff the author says resonates deeply, but like, the whole deterministism thing is why I liked programming and computers in the first place. They are complicated but simple; they run on straightforward, man-made rules. As the article says:
> Any good engineer will know how the Internet works: we designed it! We know how packets of data move around, we know how bytes behave, even in uncertain environments like faulty connections.
I've always loved this aspect of it. We humans built the entire system, from protocols down to transistors (and the electronics/physics is so abstracted away it doesn't matter). If one wants to understand or tweak some aspect of it, with enough documentation or reverse engineering, there is nothing stopping you. Everything makes sense.
The author is spot on; every time I've worked with ML it feels more like you're supposed to be a scientist than an engineer, running trials and collecting statistics and tweaking the black box until it works. And I hate that. Props to those who can handle real fields like biology or chemistry, right, but I never wanted to be involved with that kind of stuff. But it seems like that's the direction we're inevitably going.
Modern ML is at this hellish intersection of underexplored math, twisted neurobiology and applied demon summoning. An engineer works with known laws of nature - but the laws of machine learning are still being written. You have to be at least a little bit of a scientist to navigate this landscape.
Unfortunately, the nature of intelligence doesn't seem to yield itself to simple, straightforward, human-understandable systems. But machine intelligence is desirable. So we're building AIs anyway.
People want others to think this tech is mysterious. It's not. We've known the theory of these systems since the mid 1900s, we just didn't fully work out the resource arrangements to make them tractable until recently. Yes, there are some unknowns and the end product is a black box insofar as you cannot simply inspect source code, but this description of the situation is pure fantasy.
There's a reason why so many "laws" of ML are empirical - curves fitted to experimental observation data. If we had a solid mathematical backing for ML, we'd be able to derive those laws from math. If we had solid theoretical backing for ML, we'd be able to calculate whether a training run would fail without actually running it.
People say this tech is mysterious because it is mysterious. It's a field where practical applications are running far ahead of theory. We build systems that work, and we don't know how or why.
We know it’s not the correct answer, but better something close than nothing. (close can be awfully far, which is worse than nothing)
You can predict how some simple narrow edge case neural networks will converge, but this doesn't go all the way to frontier training runs, or even the kind of runs you can do at home on a single GPU. And that's one of the better covered areas.
Today's mathematical background applied to frontier systems is a bit like trying to understand how a web browser works from knowing how a transistor works. The mismatch is palpable.
Sure, if you descend to a low enough level, you wouldn't find any magic fairy dust - it's transistors as far as eye can see. But "knowing how a transistor works" doesn't come close to capturing the sheer complexity. Low level knowledge does not automatically translate to high level knowledge.
Nah, it's just a very very big and fancy autocomplete with probabilistic tokenization and some extra tricks thrown in to minimize the shortcomings of the approach.
> Unfortunately, the nature of intelligence doesn't seem to yield itself to simple, straightforward, human-understandable systems.
LLMs are maybe artificial but they are not intelligence unless you have overloaded the term intelligence to mean something much less and more trivial. A crow and even a cat is intelligent. An LLM is not.
The proper name for it is "AI effect", but the word "copium" captures the essence perfectly.
Humans want to feel special, and a lot of them feel like intelligence is what makes them special. So whenever a new AI system shows a new capability that was thought to require intelligence? A capability that was once exclusive to humans? That doesn't mean it's "intelligent" in any way. Surely it just means that this capability was stupid and unimportant and didn't require any intelligence in the first place!
Writing a simple short story? Solving a college level math problem? Putting together a Bash script from a text description of what it should do? No intelligence required for any of that!
Copium is one hell of a drug.
What is the word for creating an account 12 days ago and exclusively defending the LLMs because they can't defend themselves?
> Writing a simple short story
Ah, allow me to introduce you to the Infinite Monkey theorem.
https://en.wikipedia.org/wiki/Infinite_monkey_theorem
In the case of LLMs it's just the monkey's hand is artificially guided by all the peanut-guided trainings it was trained on but it still didn't use a single ounce of thought or intelligence. Sorry that you get impressed by simple tricks and confuse them for magic.
We know that. The value of intelligence is being able to outperform that random chance.
LLMs already outperform a typewriter monkey, a keyboard cat, and a non-insignificant amount of humans on a very diverse range of tasks.
The trouble is that many problems simply cannot be solved using rules. They're too complex and undefinable and vague.
We need systems that are more flexible and expressive to handle the open-ended complexity of the real world.
Even amidst the pressure of "client X needs feature Y yesterday, get it done with maximum tech debt!" it felt like there were still chances to carve out niches of craft, care and quality.
The rise of "probabilistic" software feels like it is a gift to "close enough is good enough" and "ship first, ask questions later".
If only we all became scientists. We'll more likely become closer to tabloid journalists, producing something that sounds just truthy enough to get clicks.
I think he has a lot of good points there. Yes he treads on the thin ice with the very big statements and generalisations, but those we don't have to "sign under with blood".
I do like the simple formula concept. It does make sense. It's not an ultimate representation of everything, but it's a nice idea of how to frame the differences between the logics we are dealing with.
I choose to not commit the whole message of the article to my core beliefs, but I'll borrow thoughts, ideas for the debates and work ahead.
“We have a class of products with deterministic cost and stochastic outputs: a built-in unresolved tension. Users insert the coin with certainty, but will be uncertain of whether they'll get back what they expect. This fundamental mismatch between deterministic mental models and probabilistic reality produces frustration — a gap the industry hasn't yet learned to bridge.”
And all the news today around AI being a bubble -
We’re still learning what we can do with these models and how to evaluate them but industry and capitalism forces our hand into building sellable products rapidly
Anyone who thinks humans are reliable must have never met one.
The problem is that the tech industry has devolved into a late-capitalist clown market supported on pure wild speculation and absurd "everything machine" marketing. This not only leads to active damage (see people falling into delusional spirals thanks to chat bots) but also inhibits us from figuring out what the actual good uses are and investing into the right applications.
Radical technology leads to behavior change. Smart phones led to behavior change, but you didn't have to beg people to buy them. LLMs are leading to behavior change but only because they are being forcibly shoved into everyone's faces and people are co-gaslighting each other into the collective hysteria that to not participate is to miss out on something big, but they can never articulate what that big thing actually is.
By probabilistic pattern matching. Next.
> Every model update doesn’t necessarily mean a complete rewrite every time, but it does force you to fundamentally rethink your assumptions each time, making a rewrite a perfectly plausible hypothesis.
Translation: our crap is unreliable and will introduce regressions that you have to manage. It can regress so much that anything you built on this has to be redone. Next.
> I strongly believe that when it comes to AI, something is happening. This time it does feel different.
Said the guy working for the company that is selling you this stuff. Next.
it doesn't matter how much jargon and mathematical notation you layer on top of your black box next token generator, it will still be unreliable and inconsistent because fundamentally the output is an approximation of an answer and has no basis in reality
This is not a limitation you can build around, its a basic limitation of the underlying models.
Bonus points if you are relying on an LLM for orchestration or agentic state, its not going to work, just move on to a problem you can actually solve.
And you'd be half-right: humans are extremely unreliable, and it takes a lot of safeguards and automated testing and PR reviews etc to get reliable software out of humans.
(Just to be clear, I agree that current models aren't exactly reliable. But I'm fairly sure with enough resources thrown at the problem, we could get reasonably reliable systems out of them.)
Well, it took about 3 years of non-stop AI hype from the industry and press (and constant ignoring of actual experts) until finally the perception seems to have shifted in recognising it as another bubble. So I wouldn't say any lessons were learned. Get ready for the next bubble when the crypto grifters that moved to "AI" will soon move on the to the NEXT-BIG-THING!
Who recognized this exactly? The MIT article? Give me a break. NVDA was $90 this year, was that the world recognizing AI was a bubble? No one is privy to anything when it comes to this. Everyone is just going to get blindsided again and again every time they sleep on this stuff.
Clearly, a lot of people very desperately want AI tech to fail. And if there is such a strong demand for "tell me that AI tech will fail", then there will be hacks willing to supply. I trust most of those "experts" as far as I can throw them.
Two propositions that can both be true, and which I believe ARE both true:
1. AI is going to change the world and eat many industries to an even greater degree than software did
2. Today, AI is often over-hyped and some combo of grifters, the naive, and gamblers are driving a bubble that will pop at some point
https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
Parallel with the bitter lesson being that general purpose algorithms that use search and learning leveraged by increasing computational capacity tend to beat out specialized methods that exploit intuition about how cognitive processes work.
The bitter lesson is about model level performance improvements and the futility of scaffolding in the face of search and scaling.
And we will over time have even more capability in the model (that is more general purpose) than our deterministic scaffolds...
Who is "we" in that case? Are you the one building the model? Do you have the compute and data capacity to test every corner case that matters?
In a deterministic system you can review the code and determine what it does under certain conditions. How do you know the ones that do build non-deterministic systems (because, let's face it, you will use but not build those systems) haven't rigged it for their benefit and not yours?
> And we will over time have even more capability in the model (that is more general purpose) than our deterministic scaffolds...
Our deterministic scaffolds sounds so dramatic, it sounds like you think of them like the chains that keep holding you, if only those chains were removed, you'd be able to fly. But it's not you who'd be able to fly, it's the ones building the model and having the compute to build it. And because of its non-deterministic nature, a backdoor for their benefit is now simply plausible deniability. Who is we. You are a user of those models, you will not be adding anything to it, maybe only circumstantially by your prompts being mined. You are not we.
I have been using OSS libraries for the last 20+ years where there are signatures and one can actually check the source. I think the industry mostly moved away from using magic binaries?
With LLMs a lot of what you get out is based on the data they are trained on (black box), a dash of extra refinement and fine tuning (black box) and if it's not running on your device, maybe even shadow prompts. I dont want to build anything with LLMs for the same reason I don't want an Alexa in my home.
To me, GenAI slop is the ultimate dumbing down of society and the ultimate control mechanism. People don't wanna bother read 5 lines of text, bam, summarize it for them into a one liner. People don't wanna bother to learn to draw: bam, some shitty model generates a picture of a cat for you. People don't wanna learn some syntax and think about their codebase: bam, Claude to the rescue.
We are collectively dumbing down an already progressively dumb society using increasingly sophisticated tools atrophying our already atrophied brains (doomscrolling and brainrot didn't help).
If experience has shown anything, it's that when our reliance of these tools increases, thats when the people who control them will start forcing ads and change public opinion with them, along with continuous surveillance. Why are we doing this to ourselves, just so we can create a webapp 5 min faster that won't be maintainable in 2 months?