Hacker News Reader

Comment by coldtea | Hacker News Reader

>Those little black boxes of AI can be significantly demystified by, for example, watching a bunch of videos (https://karpathy.ai/zero-to-hero.html) and spending at least 40 hours of hard cognitive effort learning about it yourself.

That's like saying you can understand humans by watching some physics or biology videos.

View on HN

No it’s not

Nobody has built a human so we don’t know how they work

We know exactly how LLM technology works

We know _how_ it works but even Anthropic routinely does research on its own models and gets surprised

> We were often surprised by what we saw in the model

https://www.anthropic.com/research/tracing-thoughts-language...

Which is…true of all technologies since forever

Except it's not. Traditional algorithms are well understood because they're deterministic formulas. We know what the output is if we know the input. The surprises that happen with traditional algorithms are when they're applied in non-traditional scenarios as an experiment.

Whereas with LLMs, we get surprised even when using them in an expected way. This is why so much research happens investigating how these models work even after they've been released to the public. And it's also why prompt engineering can feel like black magic.

I don’t know what to tell you other than to say that the concept of determinism in engineering is extremely new

Everything you said right now holds equally true for chemical engineering and biomedical engineering so like you need get some experience

That’s some epic goal post shifting going on there!!

We’re talking about software algorithms. Chemical and biomedical engineering are entirely different fields. As are psychology, gardening, and morris dancing

I said all technologies

Yeah. Which any normal person would take to mean “all technologies in software engineering” because talking about any other unrelated field would just be silly.

I think the historical record pushes back pretty strongly on the idea that determinism in engineering is new. Early computing basically depended on it. Take the Apollo guidance software in the 60s. Those engineers absolutely could not afford "surprising" runtime behavior. They designed systems where the same inputs reliably produced the same outputs because human lives depended on it.

That doesn't mean complex systems never behaved unexpectedly, but the engineering goal was explicit determinism wherever possible: predictable execution, bounded failure modes, reproducible debugging. That tradition carried through operating systems, compilers, finance software, avionics, etc.

What is newer is our comfort with probabilistic or emergent systems, especially in AI/ML. LLMs are deterministic mathematically, but in practice they behave probabilistically from a user perspective, which makes them feel different from classical algorithms.

So I'd frame it less as "determinism is new" and more as "we're now building more systems where strict determinism isn't always the primary goal."

Going back to the original point, getting educated on LLMs will help you demystify some of the non-determinism but as I mentioned in a previous comment, even the people who literally built the LLMs get surprised by the behavior of their own software.

I refuse to believe you sincerely think this is a salient point. Determinism was one of the fundamental axioms of software engineering.

We know why they work, but not how. SotA models are an empirical goldmine, we are learning a lot about how information and intelligence organize themselves under various constraints. This is why there are new papers published every single day which further explore the capabilities and inner-workings of these models.

You can look at the weights and traces all you like with telemetry and tracing

If you don’t own the model then you have a problem that has nothing to do with technology

Ok, but the art and science of understanding what we're even looking at is actively being developed. What I said stands, we are still learning the how. Things like circuits, dependencies, grokking, etc.

Reply on news.ycombinator.com