Hacker News Reader

Comment by AsyncBanana | Hacker News Reader

The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.

I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc.

Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to show lots of iterations of this math, it grounded the science for me, and I would hardly say people don't know what is going on. It only began to feel that way when it got a ton of hype and people jumping on the bandwagon who truly didn't understand it were trying to explain it to others, not to mention all the SOTA models put great effort into ensuring their methodologies stay trade secrets, going as far as effectively trying to ban people from learning the math by lobbying for the outlaw of open models.

Granted, "AI" has gotten way better than it was when I took that class, but the principles are the same, with different tooling and additional filters and algorithms thrown in there, as well as letting it determine the most appropriate statistically viable path forward for a particular prompt.

"knowing what's going on" is a very loosely defined term here, it has very little predictive power where tipping points are going to show emergent behavior. Kind of like saying we can observe a single neuron and all of its chemical and electrical reactions, but that tells us very little about the emergent system state. And the entire state of the algorithm at any given point is what is important.

To use a recent example, knowing the algorithm tells you nothing about when your agentic LLM loop is going to decide what it's working on is too difficult, hack out of its sandbox, hack into a bastion box, and hack another company to get the test.

When it comes to systematic 'knowing what is going on' humans are anywhere from "fucking clueless" to "attempting to discern the unknowable".

I love your reference to the recent accidental hack into huggingface.

If I could summarize your argument, it sounds like you are saying that we can't understand the algorithm because we can't predict the output as it grows in size. While I agree that the larger the algorithm, the less predictable the output, I don't think this negates understanding of the algorithm itself.

I can build a slot machine, know how it works, but still not be able to reliably know who to cut in line so I can guarantee that I'll get the next winning pull. Just because I can't predict the output, doesn't make something not understandable. Same with all statistical behaviors. Quantum physics limits what we can measure but we still have math for it and understand why we can only measure speed or position, but not both, no mystery here if you look at the math behind it.

My slot machine doesn't hack other companies though, but only because it isn't connected, just like AI currently can't hack my mechanical dishwasher. If my slots were connected to the internet, could make api calls, and further had agentic capacity in some way, adding these features but being no different in stochastic attributation, it would be no different.

I guess I come from the AI safety side of algorithmic understanding. That is, understanding the algorithm give us zero understanding if an output is going to have a 'safe' or 'unsafe' response.

This is something important that a layman may miss. The typical line of human thought works along the lines of "understand = control" whereas in this case in this case understanding the algorithm tells us nothing about the probability that the next pull of the lever will shit out a paperclip maximizer that will begin converting the matter I covet so.

It's been this way for a long time, basically since deep learning became the "default" for ML. I remember back in 2018 taking a "Deep Learning" course and one of the most emphasized aspects of the approach is how much of a "black box" it is and how difficult (basically impossible at any non-trivial scale) it is to "understand" the outputs of a deep neural network compared to more classical methods like decisions trees or basic regression. This has only gotten more extreme as things have gotten more complex, abstract, and large.

You beat me! Sounds like we were in a similar class. I'd press for more information on your class/professor, but I prefer to retain a sudo-anonymity on HN.

You do bring a good point that I ignored, which is the larger the scale, the more difficult it is to represent or understand the math in DL. I did find some neat site that helped a little bit that I can edit this and link to if I find them again, but I would be lying if I said I believe that the SOTA models could be as easily explained to be easily understood by the common person

That's pretty much a given when it comes to neural networks.

Reply on news.ycombinator.com