MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video] | Hacker News Reader

MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

youtube.com

367 pointsmfiguiere53 comments

I wonder why that parameter is called "h"? Hmmm ...

Why not just say the word "hysteresis" and bring some magnets to class for show-and-tell to help your students develop an intuition for the "h" parameter in RNNs.

hysteresis is also important to understand for working with radio networking.

And for analog electronics in general!

And for non linear forcing of plasmas… but it’s been many years since my phd

Hidden state?

Like jimsimmons said below, I believe it traditionally refers to 'hidden'. which was in vogue at the time for both feedforward nets and RNNs as well as any other other neural networks in the 90's or so and on. This trend actually continued for a while and I learned it in one of Hintons' main online classes which was made somewhere between 2012-2015 or so IIRC (though I opted to switch to reading and trying to implement raw papers instead as my brain works intuitively strangely, on the whole).

You can think of it as everything the RNN knows about what you're doing and a thing that evolves from place to place as you go. Because it is iterated on itself as a map, it abides by some very interesting properties that let it represent some very difficult functions, though actually attaining a representation of those functions is rather difficult indeed in my experience from what I've seen.

There are one or two rather successful projects trying to keep RNNs both alive and competitive with transformers. I think they do very well on the whole, though the transformers seem to have slightly improved parameter efficiency, generally speaking.

I hope this helps you with your question, please do let me know if you have any other follow up questions on this topic/matter. (: (: :) :)

Hmm i read a tonne of RNN lit before 2020 and 'd never come across the term "hysteresis parameter" standing in for the hidden units. is it a recent trend? Google seem to suggest so

I didn't mention anything at all about a hysteresis parameter.

As defined by the lecturer herself, "h(t) = h(t-1)", the very definition of hysteresis.

My point is that the lecturer missed a golden opportunity to give her students a natural intuition of "h" that they can see, feel and touch and that will serve them well for their entire careers.

The only thing "hidden" about "h" is that hysteresis is hidden in plain site in her lecture - maybe the lecturer did not know herself.

Neural networks have an undeserved reputation for being mysterious, and maybe that is partly due to a lack of basic physics knowledge.

Back in the day, having taken some kind of statistical signal processing course would have been common before getting into neural networks. That would likely have covered a lot of intuitions.

> As defined by the lecturer herself, "h(t) = h(t-1)", the very definition of hysteresis.

How is that a definition of hysteresis?

Hysteresis is when state is a function of previous state, not identical to previous state.

It's just simplified pseudoscope using the lecturer's own notation from her slides to make my point.

The following is the lecturer's full TeX form if that helps:

h(t) = \tanh \left(h(t-1) W_{\text{hh}}^T+x(t) W_{\text{hx}}^T\right)

However, I don't want our readers to get distracted by line noise; h(t) = h(t - 1) makes my point.

Which is the best course or set of videos to learn the basics of neural networks and deep learning? Something that really gets the best explanation of things like backprop?

I’ve been watching https://karpathy.ai/zero-to-hero.html and they seem amazing so far, literally go to gpt from basic math and programming - he first codes the libs from scratch to show you how the internals work, and only then uses an off the shelf production lib.

Yes this is a great course, on every level. I recommend to do the exercises after each lecture to cement the concepts too. The exercises are good too. Stretching but achievable.

what do you mean by exercises are there any specific exercises or the ones who is solving while he's explaining the material?

under the first video there is a link to a google colab document with exercises - different but similar to what is in the video.

It's a great resource, but very focused on language models. I'd say if you are looking for a general overview of deep learning that fastai is fantastic.

http://cs231n.stanford.edu/

cs231n is still a really solid course, despite the more recent lecture videos not being publicly available.

Look for Karpathy’s CS231n lectures on YouTube. They are really good.

Karpathy's zero to hero series is excellent, and I really recommend it.

I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit.

https://github.com/tysam-code/hlb-CIFAR10 (convolutional networks and imaging stuff. it's fast. very fast.)

https://github.com/tysam-code/hlb-gpt (pruned-down base of nanoGPT with training-speed-focused changes built on top of it. Check out the 0.0.0 tag from the repo if you want the barest of bare-bones implementations. Thanks!)

Both of these implementations are pretty straightforward for what they do but CIFAR-10 has less dynamic scheduling and stuff so it might be easier to fit in your head. However, both are meant to be simple (and extremely hackable if you want to poke around and take apart some pieces/add different watchpoints to see how different pieces evolve, etc. I am partially inspired by, among many things, one of those see-through engine kits that I saw in a magazine growing up as a child that I thought was a very cool, dynamic, and hands-on way to just watch how the pieces moved in a difficult topic. Sometimes that is the best way that our brains can learn, though we are all different and learn best differently through different mediums in my experience).

Feel free to let me know if you have any specific questions and I'll endeavor to do my best to help you here. Welcome to an interest in the field!

I guess to briefly touch on one topic -- some people focus on the technical only first, like backprop, and though math is required heavily for more advanced research, I don't learn concepts very well through details only. Knowing that backprop is "Calculate the slope for the error in this high-dimensional space for how a neural network was wrong at a certain point, then take a tiny step towards minimizing the error. After N steps, we converge to a representation that is like a zip file of our input data within a mathematical function" is probably enough for 90-95% of the usecases you will do as a ML practitioner, if you do so. The math is cool but there are more important things to sweat over IMO, and I think messaging to the contrary raises the barrier to entry to the field and distracts from the important things, which we do not need as much. It's good to learn after you have space in your brain for it after you understand how the whole thing works together, though that is just my personal opinion after all.

Much love and care and all that and again feel free to let me know if you have any questions please. :) <3

Thanks, will check these out for sure. Already digging Karpathys code first approach in the first few hours!

Yeah, he is great. Hard to really estimate the impact he's had on the deep learning community on the whole/at large.

We could always use more people sharing and spreading knowledge around like him! I hope to find a similar brand to that someday for myself, though I'm constantly growing into those boots, I think. :') <3

If you have any questions on that too feel free to let me know here and I can try to answer them. It's certainly a very interesting field! <3 :DDDD :)))) <3

Since the topic of this thread is this MIT course, Is this video great for you? I liked it.

Not a course, but I would highly recommend Deep Learning with Python, by Francois Chollet, creator of Keras. Incredibly approachable book that covers everything from tensors and backprop, to mixed precision and multi gpu scaling, and includes time series, language, vision and audio in between

Andrew Ng's machine learning course on Coursera is still my favorite introduction to neural networks

Interesting that they're using TensorFlow. Is TensorFlow still common for new projects or is it because Google is sponsoring the class?

It is not common for new projects. The vast majority of new projects use PyTorch, with some using tensorflow and some using JAX.

Can PyTorch be used to implement LLMs / GPT?

Yes, see also the transformers library containing many transformer models in pytorch

PyTorch is the most used neural net framework in research. Almost everything appears in PyTorch first.

quite easily, yes

Are these the sorts of Transformers and "Attention" heads that play a prominent role in LLMs like GPT3 et al? Would this lecture help build a foundational understanding of those technologies?

Bit of a side tangent, but why does MIT upload (or allow the upload) of these videos under the staff member's youtube. Alexander Amini in this case.

It makes it hard to find and subscribe to. And also a bit weird from an ownership perspective.

For better or worse I think it's how Lex Fridman got his initial boost, I believe his personal youtube channel contained some popular MIT lectures of him at the start.

When I worked in higher ed, I learned that professors retain a lot more ownership of their content than I expected. I don’t know if this is how it works everywhere, but I wouldn’t be surprised if that was the case here.

In basically all cases course material is developed by professors and TAs, though the current lecturer might have adapted from several years of previous work. Who owns the IP exactly is tricky (likely the university has at least an equal stake?), but I expect that if lectures are recorded, but not open-access then it's the lecturer's decision. It might be copyright issues, re-use of someone else's slides, not wanting to let students see past years' work, etc.

For example, I work at ETH and we have a large internal video archive of lectures (as do many universities pre- and post-covid), but some lecturers choose to post material on YouTube too. It's not a blanket yes/no policy at the institutional level, as far as I'm aware.

I thought MIT had to remove a huge catalogue of institutional content (under the opencourseware label) because of an ADA lawsuit. The problem was they didn’t hire people to subtitle it, which was discrimination.

It was Berkeley, I don't think MIT was impacted.

Reply on news.ycombinator.com