As someone who missed the boat on this, is learning about this just for historical purposes now, or is there still relevance to future employment? I just imagine the OpenAI eats everyone's lunch in regards to anything AI related, am I way off base?
In terms of continued relevance - "deep learning", meaning, dense neural nets trained to optimize a particular function, haven't fundamentally changed in practice in ~15 years (and much longer than that in theory), and are still way more important and broadly used than the OpenAI stuff for most purposes. Anything that involves numerical estimation (e.g., ad optimization, financial modeling) is not going to use LLMs, it's going to use a purpose-built model as part of a larger system. The interface of "put numbers in, get number[s] out" is more explainable, easier to integrate with the rest of your software stack, and more measurable. It has error bars that are understandable and occasionally even consistent. It has a controllable interface that won't suddenly decide to blurt corporate secrets or forget how to serialize JSON. And it has much, much lower latency and cost - any time you're trying to render a web page in under 100ms or run an optimization over millions of options, generative AI just isn't a practical option (and is unlikely to become one, IMO).
I don't have a significant math or theoretical ML background, but I've spent most of the last 10 years working side by side with ML experts on infra, data pipelines, and monitoring. I'm not sure I could integrate the sigmoid off the top of my head, but that's not what's important - I've done it once, enough to have some idea how the function behaves, and I know how to reason about it as a black box component.
Which video is this?
His series on making a GPT from scratch is also great for building intuition specifically about text-based generative AI, with an audience of software developers.
Unlike other areas of ML, the nature of deep learning is such that its parts are interoperable. You could use a transformer with a CNN if you wish. Also, deep learning enables you to do machine learning on any type of data, text, images, video, audio. Finally, it can naturally scale computationally.
As someone pretty involved in the field, I lament that LLMs are turning people away from ML and deep learning, and following the misconceptions that there’s no reason to do it anymore. Large algorithms are expensive to run, have slow throughput and are still generally poorer performing than purpose built models. They’re not even that easy to use for a lot of tasks, in comparison to encoder networks.
I’m biased, but I think it’s one of the most fun things to learn in computing. And if you have a good idea, you can still build state of the art things with a regular gpu at your house. You just have to find a niche that isn’t getting the attention that LLMs are ;)
The whole thing is essentially curve fitting. The ML field is essentially an art more than a science and it's all about tricks and intuitions on different ways of getting that best fit curve.
From this angle the whole field got way less interesting. The field has nothing deeper or more insightful to offer beyond this concept of curve fitting.
I think there are some really cool problems, such as:
1. Is synthetic data viable for training?
2. How do you make deep learning agents that can do task planning and introspection in complex environments?
3. How do we efficiently build memory and data lookup into AI agents? And is this better/worse than making longer context windows?From a technical standpoint, it’s not correct analogy either, because it assumes you have a curve to fit. What curve is language? What’s curve is images? No answer, because there isn’t one. Deep learning is about modeling complex behaviors, not curve fitting. Images and language for instance are based in social and cultural patterns and not intrinsic curves to be fit.
At best, it’s an imprecise statement. But I’d disagree entirely.
IOW: to me, fitting a generalized linear model is very different than fitting a convolutional network.
OpenAI is for example not interested in developing small embedded neural networks that run on a sensor chip that real-time detects specific molecules in air.
After a glance, looks like too much for one book. Probably it was compressed with the assumption that reader already knows quite a lot. In other words it's not an easy reading.
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat...
So yes if you are prompt engineering, and wondering why X works and sometimes it doesn't, and why any of this works at all, it is good to study a bit.