For the non-DL crowd, Transformers are a tsunami in deep learning for the past few years. They are topping benchmarks in many subfields. I do research professionally and this work is amazingly useful for people like me.
For the non-DL crowd, Transformers are a tsunami in deep learning for the past few years. They are topping benchmarks in many subfields. I do research professionally and this work is amazingly useful for people like me.
You may have seen the recent splash of MLP-Mixer for image classification using perceptrons (https://arxiv.org/abs/2105.01601#google) but FC-heavy or FC-only nets have popped up here and there, never quite going away, and slowly getting better initializations & residual layers making them ever more trainable. I have a bibliography of some links I've noted over the years about them: https://www.gwern.net/notes/FC
I've wondered if MLPs will be another example of the Bitter Lesson - the additional flexibility and power was their undoing until data/compute caught up and 'grad student descent' figured out how to use them right... If in the next 5 years everyone gets on the "MLP Is All You Need" train, I will be only mildly surprised.