Transformers Are All You Need | Hacker News Reader

Transformers Are All You Need

pinecone.io

13 pointstakiwatanga5 comments

The performance advantages of transformer architectures have little to do with "transformers" themselves, but rather other architecture changes that were made along the way. They can be matched or beat in performance by tweaked conventional CNNs: https://arxiv.org/abs/2201.03545

Yeah LSTMs can outperform transformers on a lot of small tasks. What transformers do exceptionally well is scale.

Part of the scaling benefit comes from the efficiency of training. LSTM and other recurrent architectures take more bits per parameter and the attention mechanism doesn't optimize for context - each lstm neuron uses gating individually, where transformers attention mechanisms can use the context of an entire layer.

LSTMs with attention are super interesting, and there are some neat self-growing hopfield networks with attention that could push into transformer style efficacy and efficiency gains.

> ViT models are outperforming CNNs in terms of computational efficiency and accuracy, achieving highly competitive performance in tasks like image classification, object detection, and semantic image segmentation.

Since then, this has been show to be untrue. Using more modern training techniques along with depthwise convolutions (https://arxiv.org/abs/2201.03545) results in equal if not better performance on vision tasks. Improved training methodologies have also been shown to boost the accuracy of ResNet50 - an 6-year-old pure convolutional architecture - on ImageNet-1k by over 5% (https://arxiv.org/abs/2110.00476).

Pure ViTs are also more difficult to train when compared with traditional convnets, although this has since then been somewhat remedied by Swin (https://arxiv.org/abs/2103.14030).

> Unlike BERT, GPT models are unidirectional, and their main advantage is the magnitude of data they were pretrained on: GPT-3, the third-generation GPT model, was trained on 175 billion parameters, about 10 times the size of previous models.

Although the number of parameters of a Transformer correlates with the amount of training data, this statement is misleasing. Precisely, the model wasn't trained on parameters but its parameters were trained on data.

Reply on news.ycombinator.com