Vision Transformers (ViT) Explained
pinecone.io
pinecone.io
Wouldn’t it make sense to have a process that selects regions of the image to focus in on, perhaps itself guided by some sort of attention model? So for a face it might produce tiles covering the whole face, the eyes, the mouth, whatever other details it finds helpful… all obviously accompanied by a vector representing the tile’s spatial relationship to the whole image.
Or is the assumption that this sort of thing is effectively done within the model’s layers as it combines the input from multiple tiles?
Don’t they? Most use BPE, which yields arbitrary but common sequence of characters that often are just syllables. Additionally, many use SentencePiece, which randomizes the subdivisions, which sometimes produces single characters instead.
(That said, what you suggest would be an interesting research avenue.)
And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.)
Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…
In my case, the entropy-optimized tokenization also matched conserved features of the German language.
If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well.
Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not.
But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.
In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).
Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?
That’s significantly more data either way though, which is probably the issue.
[0] https://paperswithcode.com/paper/voxel-transformer-for-3d-ob...
> We enable order with positional embeddings. For ViT, these positional embeddings are learned vectors with the same dimensionality as our patch embeddings.
> After creating the patch embeddings and prepending the “class” embedding, we sum them all with positional embeddings.
> These positional embeddings are learned during pretraining and (sometimes) during fine-tuning. During training, these embeddings converge into vector spaces where they show high similarity to their neighboring position embeddings — particularly those sharing the same column and row
Made no sense to me.
Like, okay, the positional embedding represents the position of a "patch" within the input image? What else? How are they used, what are they compared against, how are they fed into the neural network?
Without this, you might as well be watching "Introduction to X, for experts in X". Or "Introduction to X, by someone who is trying to figure out X".
What?
This is because graph networks are permutation invariant, but vision tasks are not. Nodes do not necessarily know their ordering with respect to the input. We use positional embeddings to inform the nodes where they are with respect to the input, because it matters.
To see how, or why, take any image, flip it upside down, cut it into patches, and then flip the patches.
The patches know what their content is, but not where. The edges in the graph merely point to the neighbours for further computation.
Positional embeddings are just vectors that you index or compute at runtime. These vectors then are concatenated or added onto the vectors of the patches or tokens.
We use positional embeddings for all sorts of tasks, such as text (position of token), vision tasks (position of patch), or even sound tasks (e.g. point in time).
They act as a "position signal" that modifies the patch embedding. The learned signals are similar to other neighbouring position signals, and the later layers of the model will use the "similarity" between signals to identify the proximity/order of different patches.
There is no explicit mechanism that tells the network to make neighboring position embeddings similar, it's just a result of the training that fortunately works and seems logical.
I can definitely try to explain further if needed
It's the most confusing part of transformers for me. How do we train the module that creates these embeddings?
Like, say the embedding of a patch was just a vector (a, b, c, d). To have the attention layer "understand" position, you could just concatenate the patch's position to get (a, b, c, d, x, y). I understand that.
What I don't understand is:
- Why on earth are cosines involved?
- What does that mean:
> During training, these embeddings converge into vector spaces where they show high similarity to their neighboring position embeddings — particularly those sharing the same column and row
Does it just mean that the network learns that "x = 1" is similar to "x = 2"? Because presumably, that's not very valuable by itself, is it? It's something you could easily hardcode.
Presumably there's a step where the network goes "if there is pattern A and pattern B and both patterns have a short distance between them in the positional embedding then we have C" where A B and C are neural network magic... but your article is explaining that step, I don't understand the explanation.
This is one of the issues with a lot of machine learning articles out there (not to nitpick on you, sorry); there are almost always easy and illustrative examples that you can use to break this down into a simpler explanation.