Comment by famouswaffles | Hacker News Reader

Comment by famouswaffles | Hacker News Reader

Parent Full thread

Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing.

Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings.

as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision.

and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929

It basically just comes down to lots of layers and training

The original commenter is trying to build intuition so my statement was simply to help the commenter understand that Transformers can operate on patches.

As for > after all, the original authors didn't anticipate the use case in Vision.

To quote the original "Attention is all you need" paper: "We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video"

To say that the original authors did not anticipate the Transformers use in Vision is false.

I'm talking about using transnformers on their own, beating SOTA CNNs. People thought you needed CNNs even with transformers...until that was shown wrong. The point is that there isn't any special intuition that makes this fit vision.

Why would you feed pixels into transformers? Surely you could translate the overall image into frequencies using Fourier transforms and then into embeddings which feed the LLM?

It's essentially a process if denoising a spectrogram, and as such the data is processed that way.

Reply on news.ycombinator.com