Edit: Everything tagged with AI/ML
:-)
So on average probably 50 pages an hour.
With practice, that's entirely reasonable.
Without practice it's a lot.
Reading technical literature is a skill that develops over time. At first it goes slow. Then it gets normal. At first it might take an hour to read a 12 page paper. After a few months, fifteen minutes becomes enough to read a paper and make a cup of coffee.
BeIT-3 for example.
My big unsolved problem is Twitter — how do I avoid going on twitter more than a half hour a day, by using some type of twitter based filter/aggregator? Labml daily is a relatively good trend aggregator informed by Twitter. But I still keep discovering interesting things on Twitter not covered by any of the above. And BTW I bookmark twitter threads to Readwise/reader as well.
The people posting on Twitter do it to be heard by others. As the audience decreases, the significance of the platform decreases. Thus, people stop posting on said platform and use other avenues to get their voices heard.
As users drop off, advertisers leave, removing a large part of their revenue.
Your claim is so completely out of touch with reality...
He has recently started posting incredibly detailed weekly AI roundups. Here's one from yesterday:
https://thezvi.wordpress.com/2023/04/06/ai-6-agents-of-chang...
I find that I always come away learning a thing or two from his updates -- and feel as though I'm keeping up with at least those developments which relate to commercially-available AI.
His blog is not a repository of scientific work like aRxiv, but more like a curated summary of AI news. It is, after all, a blog.
edit: the readers page is not up to date the last books are available.
https://www.youtube.com/playlist?list=PLTPQEx-31JXgtDaC6-3Hx...
I just started going though it today, so I can't attest to the usefulness.
Then watch Neural Networks: Zero to Hero[1] to learn how autograd works, learn a few advance topics, and most importantly, to see what the workflow of an expert looks like.
Then read PyTorch tutorials and the PyTorch docs while doing some real projects.
If PyTorch starts to feel too rigid, then look into Jax or Julia's Flux library.
[0]: https://mitpress.mit.edu/9780262546379/the-little-learner/ [1]: https://karpathy.ai/zero-to-hero.html
The real challenge is finding the time to read them all.
[1] - https://www.reddit.com/r/ChatGPT/comments/12diapw/gpt4_week_...
For example:
> “babyagi” is a program that given a task, creates a task list and executes the tasks over and over again. It’s now been open sourced and is the top trending repos on Github atm [Link]. Helpful tip on running it locally [Link]
The babyagi project is is an extremely simple 180 line python script. The tips for running it is just rephrasing the readme to set some environment variables.
Is this what everyone is getting so hyped about?
I am planning on adding more feeds very soon to increase the amount of content
I also have an aggregator for Cybersecurity: https://cyberfeed.io
You'll find both curated news, stories, tutorials, tools, and in-depth content.
Disclaimer: I'm the curator of this newsletter.
- Andrej Karpathy: https://twitter.com/karpathy
- David Ha: https://twitter.com/hardmaru
- Yann LeCun: https://twitter.com/ylecun
- Jeremy Howard: https://twitter.com/jeremyphoward
- Riley Goodside: https://twitter.com/goodside
HN is okay for things you don't watch so you don't miss out on anything cool. It used to be 1-2 years behind the curve in the GPT-3 era, but now that things are moving faster, it's only around 3 months or so behind.
It's a newsletter/podcast.
Sure, I can provide a brief overview of the key breakthroughs and advancements that have contributed to the current state of AI, particularly in the domain of deep learning.
1. Availability of data: The explosion of digital data, especially from the internet, has provided a massive amount of training data for AI models. This has allowed AI systems to learn patterns, features, and representations from various data sources more effectively than before.
2. Hardware improvements: The introduction of GPUs (Graphics Processing Units) and specialized hardware, like TPUs (Tensor Processing Units), has significantly accelerated the training of large neural networks. These advancements enable researchers to experiment with larger and more complex models, leading to improved performance.
3. Algorithmic innovations: Key algorithmic advancements have been made to train deep neural networks more efficiently. Some notable examples include: a. Backpropagation: This algorithm is used to train neural networks by minimizing the loss function through gradient descent. Although it was introduced in the 1980s, it became more widely used and optimized in recent years. b. Activation functions: Non-linear activation functions like ReLU (Rectified Linear Unit) have been crucial in addressing the vanishing gradient problem and improving training efficiency in deep networks. c. Dropout: This regularization technique helps prevent overfitting by randomly dropping out neurons during training, encouraging the network to learn more robust features.
4. Architectural advancements: The development of various neural network architectures has led to improved performance in specific tasks. Some prominent architectures include: a. Convolutional Neural Networks (CNNs): These networks are especially effective at image recognition tasks due to their ability to capture spatial patterns and hierarchical features. b. Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM): These architectures excel at handling sequence data, such as time series or natural language processing tasks. c. Transformers: Introduced in 2017, the transformer architecture has become a key component in state-of-the-art natural language processing models like BERT and GPT, due to its self-attention mechanism and ability to handle long-range dependencies.
5. Transfer learning and pre-training: Instead of training models from scratch, researchers have found it effective to pre-train models on large datasets, followed by fine-tuning them on specific tasks. This approach reduces training time, requires less labeled data, and often leads to better performance.
These breakthroughs and advancements, combined with a growing research community and increased investment in AI, have led to the current state of AI, where deep learning models can achieve human-level or near-human-level performance on a variety of tasks.
I'd say the big change was first deep nets, mostly CNNs. 20 years ago, there was a lot of talk about fully connected hidden layers. And how they are equivalent to an ANN with one hidden layer, if you allow exponential scaling of the number of perceptrons. They took forever to converge. CNNs reduce the number of parameters, and hence the training by quite a lot. They also make it obvious that even if multi-hidden layer nets are equivalent theoretically, in practice, you can cheat a lot using convolutions, thereby making this theory pointless.
Then came embeddings that segmented training into an embedding stage and a prediction stage, further reducing training time.
RNNs never really took off as a name, and seems to have been replaced with LSTM, being a subset of the vague RNN term. Having a well-understood way of introducing memory is very useful for sequences, and makes scaling a model to sequences much cheaper. So yet another input size reducing thing. (I guess we're basically building Markov processes on top of ANNs now? When I was in school, it was an either-or choice.)
On a meta-level, there are ensemble models, which is just taking many models, running them in parallel and combining the outputs. Simple idea, but apparently effective enough that people keep doing it. :)
Then there was attention being added, which adds a stage where you find similarities between entities. This means the ANN doesn't have to figure clustering, which again speeds up convergence. (I think it also makes the models more modular, but I'm not sure.)
Reinforcement learning (Q-learning from the late 1980s) arguably benefits tremendously with all the improvements to the ANN models, so I think time will tell how awesome this will be. For now, it seems we're all amazed by how much just the ANN (Q-function) can do on its own.
All of this is enabled by vastly improved matrix arithmetic hardware, software libraries and RAM prices. (Aside from each of these techniques just reducing training time/size.)
And then there's the inevitable hype bubble that just turns it all up to 11 for a while, because people with money want to invest and people without money want to pretend to know things. I wouldn't discount the availability of money as a "news" here, though.