Hacker News Reader

Comment by PheonixPharts | Hacker News Reader

I find this result absolutely fascinating, and is exactly the type of research into neural networks we should be expanding.

We've rapidly engineered our way to some very impressive models this past decade, and yet gap in our real understanding of what's going on has widened. There's a large list of very basic questions about LLMs that we haven't answered (or in some cases, really asked). This is not a failing of people researching in this area, it's only that things move so quickly there's not enough time to ponder things like this.

At the same time, the result, unless I'm really misunderstanding, gives me the impression that anything other than grid search hyper parameter optimization is a fools errand. This would give credence to the notion that hyper parameter tuning really is akin to just re-rolling a character sheet until you get one that is over powered.

View on HN

>exactly the type of research into neural networks we should be expanding.

While it certainly makes for some nice visualizations, the technical insight of this is pretty limited. First of all, this fractal structure emerges at learning rates that are far higher than those used in training actual neural networks nowadays. It's interesting that the training still converges for some combinations and that the (expected) hit-and-miss procedure yields a fractal structure. But if you look closely at the images, you'll see the best hyperparameters are, while close, not at the border. So even if you want to follow the meta-learning approach outlined in the post, your gradient descent has already screwed up before if it ever ends up in this fractal boundary region.

There is an expanding field of study looking at machine learning with statistical physics tools. While there is still a lot of work to do in this area, it yields interesting insights on neural networks, e.g. linking their training with the evolution of spin glasses (a typical statistical physics problem). We can even talk about phase transition and universal exponents.

Most of the research is done with simpler models though (because mainly math people do it, and it's hard to prove anything on something as complex as a transformer).

> gives me the impression that anything other than grid search hyper parameter optimization is a fools errand. This would give credence to the notion that hyper parameter tuning really is akin to just re-rolling a character sheet until you get one that is over powered.

The visualizations only show that at the Border there are a lot of fractals, not in every part of the space. (Although the highest performance is often achieved close to the border.). I would not state hparam search as bad as that..

what was the name of the Google app (Dreaming or some such) that would iterate frames similar to this that would find lizard/snakes/dogs/eyes and got super trippy the longer it ran? The demos would start with RGB noise, and within a few iterations, it was a full on psychedelic trip. It was the best visual of AI "hallucinating" I've seen yet

https://en.wikipedia.org/wiki/DeepDream

winner winner chicken dinner. thanks!

always thought we now know what the android's dreams were like

DeepDream is essentially what started the current image-creating generative AI craze.

Yes. It all started with a leaked image on Reddit. Even though at that point it was just a rumour that it was generated by a neural net, it was so strikingly unlike anything else that it caused a huge stir. Wish I could find that image again, it was a fairly noisy "slugdog" thing.

Reply on news.ycombinator.com