Hacker News Reader

Comment by wrs | Hacker News Reader

My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)

View on HN

I think there are people who know exactly how it works, they know how neural networks work, they know how the transformer architecture works, how attention works, embeddings, tokenization, etc. We just can’t define the weights of the connections between the neurons.

That's a bit like saying knowing how a pipe works is enough to explain a combustion engine. You're just listing part of how LLMs work.

Those mechanisms only explain next word prediction, not LLM reasoning.

That's an emergent property that no person, as far as I understand it, can explain past hand waving.

Happy to be corrected here.

There's no magic involved, the LLM creators can go anywhere and rebuild an LLM with pretty much the same outcome, if they have the same training data. With unlimited time you could even reproduce the output of an LLM manually as it is just a lot of mathematics. Including reasoning, as that is mostly adding words in the context that will steer the word predication to include reasoning. As this is a useful LLM behavior this context is now part of the training data, so it becomes part of the neural network weights.

Yes, but we can't inspect, reproduce or explain the emergent property independently. We can't pick out the "math reasoning" part or the "programming" part, or inspect how it's working, or selectively change any of it. You can't turn any dials or twiddle any knobs. You can't replace one part with another, or pick out components. You can't peek inside and say:

"hey it's got an irrational preference for naming its variables after famous viking warriors, lets change that!"

But worse, it's not that you can't change it, you just don't know! All you can do is test it and guess its biases.

Is it racist, is it homophobic, is it misogynistic? There was an article here the other day about AI in recruitment and the hidden biases. And there was a recruitment AI that only picked men for a role. The job spec was entirely gender neutral. And they hadn't noticed until a researcher looked at it.

It's a black box. So if it does something incorrectly, all they can do is retrain and hope.

Again, this is my present understanding of how it all works right now.

It’s not clear that those “emergent properties” even exist.

So just because all the parts are collaborating to get the desired outcome, and specific aspects of that outcome cannot be contributed to specific parts of the llm you think we don’t understand LLMs? With mixture of expert systems we’re introducing dedicated subsystems into the llm responsible for specific aspect of the llm, so we’re partially moving in that direction.

But overall in my opinion if devs are able to rebuild it from scratch with a predefined outcome, and even know how to improve the system to improve certain aspects of it, we do understand how it works.

Being able to make it and knowing how it works are just not the same thing. I can make bread of consistent quality, and know how to improve certain aspects of it. To know how it works, I’d need to get a doctorate in biochemistry and then still have a fairly patchy understanding. I know plenty of people who can drive a car very successfully but would never claim they know how it works.

But with LLMs is there really more to understand? They’re just large functions that take numerical input and transform it into numerical output based on trained weights. There is nothing behind the scenes doing things we don’t understand. The magic is in the weights, and we know how to create these based on training data.

Regarding the car, if you know how to build a car, you understand how a car works. A driver is more like someone using and llm, not a developer able to create an llm.

I’ve never seen the word “just” take on more load than it did in that second sentence!

Sorry to inform everybody doing their Ph.D. on LLM interpretability that they’re just wasting their time.

> But with LLMs is there really more to understand?

Yes! loads! (: I want to be able to say statements like "this model will never ask the user to kill themselves" and be confident, but I can't do that today, and we don't know how. Note that we do know how to prove similar statements for regular software.

> With mixture of expert systems we’re introducing dedicated subsystems into the llm responsible for specific aspect of the llm

Common misconception, MoEs do have different "experts", but the model learns when to send input to different experts, and the model does not cleanly send coding tasks to the coding agent, physics tasks to the physics agent, etc. It's quite messy, and not nearly as intepretable as we'd want it to be.

I think you’re confusing knowing how a system works with being able to predict how that system will perform.

In a non-linear system the former is often easier than the latter. For example we know how planets “work” from the laws of motion. But planetary orbits involving > 2 bodies are non-linear, and predicting their motion far into the future is surprisingly difficult.

Neural networks are the same. They’re actually quite simple, it’s all undergraduate maths and statistics. But because they’re non-linear systems, predicting their behaviour is practically impossible.

Claude just analyzed the emotional content of a piece of music for me -- quite accurately -- by just looking at an uploaded PDF of the score. How does that work? "It's a nonlinear system" or "it's a bunch of matrix multiplication" is in no useful way an explanation. That's way down at the bottom of an explanatory abstraction hierarchy that we have only begun to make tools to begin to explore. It's like asking how humans work and getting the answer "it's just undergraduate chemistry!"

The study of LLMs is much closer to biology than engineering.

Did it show you the reasoning? Did it recognize the notes, the scale, tempo and determine the emotion effect of these or did it use some other reasoning?

I don't know about the reasoning, but it gave "evidence" for its observations in much the same way a human composer might. In terms of "understanding", it seemed in the ballpark of what I get when I ask it to explain some code.

I don't want to paste in the whole giant thing, but if you're curious: [0]

[0] https://drive.google.com/file/d/1D5yICywmkp24YajboKHdYFcBej0...

Impressive, it clearly is able to read the score, see patterns, timing, chords, and apply music theory to it. Would be interesting to give it editing and playback capabilities, e.g., by connecting it to something like strudel.

This article describes how Belgian supermarkets are replacing music played in stores by AI music to save costs, but you can easily imagine that the ai could also generate music to play to the emotions of customers to maybe influence their buying behavior: https://www.nu.nl/economie/6372535/veel-belgische-supermarkt...

How is it possible that nobody knows how it works - it’s running on hardware we have complete control over and perfect observability into, is it not? At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred - we have perfect knowledge of the source code, the compiler, whatever components you prefer to break software down into -

What is it that we don’t understand?

The source code is not the LLM. The LLM is billioms of random floating point numbers that somehow encode everything the model knows and can do.

The ML field has a good understanding of the algorithms that produce these floating point numbers and lots of techniques that seem to produce “better” numbers in experiments. However, there is little to no understanding of what the numbers represent or how they do the things they do.

We know how each of the "parts" work, but there is a gazillion of parts (especially since you need to take the model weights into account, which are way larger in size than the code that generates them or uses them to generate stuff), and we found out that together they do something that we do not really understand why they do it.

And inspecting each part is not enough to understand how, together, they achieve what they achieve. We would need to understand the entire system in a much more abstract way, and currently we have nothing more than ideas of how it _might_ work.

Normally, with software, we do not have this problem, as we start on the abstract level with a fully understood design and construct the concrete parts thereafter. Obviously we have a much better understanding of how the entire system of concrete parts works together to perform some complex task.

With AI, we took the other way: concrete parts were assembled with vague ideas on the abstract level of how they might do some cool stuff when put together. From there it was basically trial-and-error, iteration to the current state, but always with nothing more than vague ideas of how all of the parts work together on the abstract level. And even if we just stopped the development now and tried to gain a full, thorough understanding of the abstract level of a current LLM, we would fail, as they already reached a complexity that no human can understand anymore, even when devoting their entire lifetime to it.

However, while this is a clear difference to most other software (though one has to get careful when it comes to the biggest projects like Chromium, Windows, Linux, ... since even though these were constructed abstract-first, they have been in development for such a long time and have gained so many moving parts in the meantime that someone trying to understand them fully on the abstract level will probably start to face the difficulty of limited lifetime as well), it is not an uncommon thing per se: we also do not "really" understand how economy works, how money works, how capitalism works. Very much like with LLMs, humanity has somehow developed these systems through interaction of billions of humans over a long time, there was never an architect designing them on an abstract level from scratch, and they have shown emergent capabilities and behaviors that we don't fully understand. Still, we obviously try to use them to our advantage every day, and nobody would say that modern economies are useless or should be abandoned because they're not fully understood.

> At any frame we can pause, examine the state, then step forward, examine the state, and observe what changes have occurred

This example from software doesn't meaningfully hold for neural networks. It's a bit like trying to watch an individual COVID virus duplicate and then attempting to predict the pandemic. It's incredibly complicated and we haven't yet built the tools to help us understand

We don’t know how to build humans from scratch (other than letting nature do it) so that’s not really a relevant example. We only know how to fix certain aspects of defect humans, and how to minimize the chance of future defects, but that’s far from building a human from scratch.

We don't know how to build LLMs from scratch either, in that sense. We know how to build a machine that you run trillions of tokens through and hope something good happens. What happens, we only know at a vague and low level. You just end up with a bag of hundreds of billions of floating point numbers with some amazing emergent behaviors.

Similarly, two adult humans know what to do to start the process that makes another human, and we know a few of the very low-level details about what happens, but that is a far cry from knowing how adult humans do what they do.

Well we are happy to work with other humans without knowing how they work.

Kinda, but if another human repeatedly showed signs of being dishonest or untrustworthy, we wouldn't be happy to work with them. Suspicion of unknown entities is good.

I thought I could do just this, but alas, [some people][1] are very convinced that we know how these things work.

[1]: https://www.reddit.com/r/slatestarcodex/comments/1o6n5ne/why...

Nobody knows (full scope and on every level) how human brains work. Still bosses rely on their employees' brains all the time.

> Nobody knows (full scope and on every level) how human brains work.

That is what the parent meant.

And my point is, that it does not matter. That we still rely on all kinds of things we do not fully grasp.

If I need to manage an AI as I would manage an employee's brain, I'm going to need quite a few non-technical resources to actually achieve that: time, willpower to babysit it, ability to motivate it, leverage in the form of incentives (and reprimands), to name a few.

AI sits at a weird place where it can't be analyzed as software, and it can't be managed as a person.

My current mental model is that AGI can only be achieved when a machine experiences pleasure, pain, and "bodily functions". Otherwise there's no way to manage it.

Bosses rely on their employees brains, but only after multiple rounds of interviews and reference checks to ensure the brain they're getting is reliable enough for the job. No boss relies on arbitrary brains taken off the street.

Reply on news.ycombinator.com