Hacker News Reader

Comment by mark_l_watson | Hacker News Reader

First, thanks to the publisher and authors for making this freely available!

I retired recently after using neural networks since the 1980s. I still spend at least 10 hours a week keeping up with DL, RL, etc. it seems like the roof has blown off the field recently, progress increases exponentially. I like material that makes me think of NNs with different intuitions. I am working on a CC licensed book consisting my experiments, in Jupiter notebook/Colab form - expect me to be shamelessly plugging that in a few months.

In the book, I especially loved this quote:

“You can hide a lot in a large-N matrix.” – Steve Shenker – John McGreevy

View on HN

How do you keep up with the deep learning community? There seems to be a deluge of new papers, and most of them aren't worth our time. What do you do?

I follow about a dozen DL people on Twitter, and use their recommendations in tweets. It is a deluge, and it is impossible to keep up, really.

I'm always looking for new people to follow, could you write up the accounts you find most useful/novel/interesting?

You train a DNN to classify outliers and look at those first.

That’s such a fascinating background to have! It must be strange to retire and, far from the cliché of your skills having been made obsolete and irrelevant, instead you’re an expert on what’s probably the forefront of modern technology.

If you can spare a moment to answer: is there any knowledge from the 80s neural net ‘summer’ which you think has been forgotten now? As someone who’s concerned by both poor performance and overfitting (strongly correlated), I thought it was a shame that lots of the research around pruning (optimal brain damage, optimal brain surgeon, etc) has been forgotten[0].

[0] It feels to me as though the ML community has convinced itself - in defiance of information theory - that highly overparameterised models are totally OK, and can even successfully extrapolate with greater than random accuracy in the general case (‘double descent’). I worry that lots of these models are effectively succeeding only at interpolation problems, and only by virtue of the massive hardware advances that let us memorise the entire training set in these enormous models - basically Runge’s phenomenon writ large. People seem to be convincing themselves of magical things that are not mathematically sound.

Pruning ( both structured and unstructured) is an extremely active area of research and development. Since the Ampere generation, Nvidia GPU’s have HW support for sparsified networks.

WRT to overparameterized models, even the largest models such as Google’s PALM 540b model has several orders of magnitude fewer connections than the human brain. Remembering a set of observations doesn’t forbid deducing a general pattern. Imagine trying to understand why the Pythagorean Theorem is true without first memorizing what it states! Children often benefit from seeing and even memorizing many concrete examples before being taught the abstract principles which explain them all. Kepler deduced his Three Laws of Planetary Motion by analyzing four decades of careful astronomical observations compiled by Tycho Brahe. Implicit and explicit regularizations bias modern learning algorithms to seek generally simpler solutions in “wide flat basins of the loss surface.” For instance, the humble skip connection acts to help smooth out the loss surface.

It's really neat to find people on HN who've been working on those structures for such a long time. If you can indulge me, what is a lesser known or obscure book on neural networks or an adjacent topic that you think would deserve to be read?

Perhaps some of the really old texts by Kahonen, Carver Mead, etc.?

For more modern material, there are a few new good books on Transformers. Transformers are interesting because they were designed for efficiency: layers the same size, encoding both data and time sequencing information in each sample (so recurrent NNs aren’t required), etc.

Mind sharing some titles?

I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code.

For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4].

[1] https://www.oreilly.com/library/view/natural-language-proces...

[2] https://github.com/karpathy/minGPT

[3] https://jalammar.github.io/illustrated-transformer/

[4] https://blog.nelhage.com/post/transformers-for-software-engi...

Not OP, but I think this is a great (and obscure?) book on neural networks: https://page.mi.fu-berlin.de/rojas/neural/

Published in 1996, but the content is all foundational.

Read up on the calculus of variations if you want to be obscure with respect to neural networks. Genetic algorithms for hyper-parameter search is also interesting.

The "deep" part of DNNs has basically thrown mathematicians and statisticians into an infinite loop that they can't quite compute yet. It's a brand new world and we need them to participate.

Where are the uses of the calculus of variations with neural nets?

Make an effort man.. jesus christ.

Reply on news.ycombinator.com