Hacker News Reader

Comment by Zetice | Hacker News Reader

Does anyone have a good recommendation for a book that would cover the underlying ideas behind LLMs? Google ends up giving me a lot of ads, and ChatGPT is vague about specifics as per usual.

View on HN

The 2nd edition of Deep Learning with Python[1] (by François Chollet of Keras fame) has a pretty good chapter on the transformer architecture[2]. In fact, the explanations in the book are clear and lucid across the board.

[1]: https://www.amazon.com/Deep-Learning-Python-Francois-Chollet...

[2]: https://en.wikipedia.org/wiki/Transformer_(machine_learning_...

Not a book, but here's a really good explanation in blog post form from Stephen Wolfram:

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

I do not trust that man one iota.

The blog post is very good.

Why?

He's got a habit of self aggrandizing, antagonism, and deception in an effort to promote himself and his brand, I worry that his explanations are designed to maximally benefit him, rather than to maximally explain the topic.

He's a brilliant man, I just don't trust him.

Is that the case with this specific article?

That’s not how trust works.

I agree generally but read the post and it only mentions cellular automata briefly and promotes Wolfram Alpha once. Overall it's very good at moving from Markov chains to neural nets with decent examples and graphics.

ML these days have moved forward fast that books are outdated after couple years. Look at articles, podcasts, youtube videos, and research papers for the latest changes. ChatGPT utilizes existing algorithms. Its insight is scaling the weight parameters up to billions, and human feedback & guidance help a lot in finetuning the model to be user friendly to general users.

To understand LLM from ground up, the following topics would help.

  - Machine Learning basics. e.g. weight parameters being trained.  
  - Neural Net basics.  
  - Nature Language Processing basics.
  - Word vectorization, word embedding. e.g. Word2Vec.
  - Recurrent Neural Net basics.
  - LSTM model.
  - Attention and Transformer model.
  - Generative model like GAN.
  - Generative Pre-trained Transformer.

I might miss a few topics. Actually ask ChatGPT to explain each topic. See how far it goes.

Or you could save some time by clicking on:

"What Is ChatGPT Doing … and Why Does It Work?"

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

graciously provided above in this discussion by danenania.

As seizethecheese asserts, also above, "The blog post is very good."

Reply on news.ycombinator.com