Hacker News Reader

Comment by jameshart | Hacker News Reader

Surprised (as a non-ML-practitioner) that a uniform tiling is used to ‘tokenize’ the image in this approach. After all, language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components)

Wouldn’t it make sense to have a process that selects regions of the image to focus in on, perhaps itself guided by some sort of attention model? So for a face it might produce tiles covering the whole face, the eyes, the mouth, whatever other details it finds helpful… all obviously accompanied by a vector representing the tile’s spatial relationship to the whole image.

Or is the assumption that this sort of thing is effectively done within the model’s layers as it combines the input from multiple tiles?

View on HN

> language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components)

Don’t they? Most use BPE, which yields arbitrary but common sequence of characters that often are just syllables. Additionally, many use SentencePiece, which randomizes the subdivisions, which sometimes produces single characters instead.

(That said, what you suggest would be an interesting research avenue.)

In this paper https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech).

In my case, the entropy-optimized tokenization also matched conserved features of the German language.

BPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it uses an existing semantic model for ‘what normal text looks like’ to choose potentially useful tokens (even for nonsensical text, which is why it works on stuff like code)

And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.)

Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…

We tried doing just that and the short answer is that we couldn't fit the gradients for arbitratily-placed tiles into a DGX with 320 GB of GPU RAM. For 32x32 pixel tiles, free location placement will 1024x your memory usage.

Perhaps hard attention could be a thing here. Possibly too tricky to train though.

Byte pair encoding (used in GPT2 models) is not semantic, it's purely based on statistics of occurrences of characters groups. It's not supposed to be semantic, it's just supposed to enable a complete coverage of the whole text. Similarly, the image patches have to cover the whole image and then the transformer can learn to extract the important parts.

Image patch extraction could be statistical rather than semantic. Like if the top part of the image is just all blue sky with no high frequency detail, you don’t necessarily need to feed the model dozens of blue tiles when you could just pass one or two big blue tiles.

If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well.

Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not.

But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.

I think that is not done because it's hard/unclear how to handle patches that are not the same size.

In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).

> I think that is not done because it's hard/unclear how to handle patches that are not the same size.

Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?

I think that is one of the benefits of having an easily parallelizable model.

The tiles would all be the same size - they’d just be sampled from the image at different resolutions.

Then you introduce a resizing component, which resizes differently for different parts of the image. And you still have to decide how to select the tile sizes: if it's heuristics, well, the field of ML has been moving away from that. If it's with another learnable component, maybe it's worth a shot, at the cost of a more complicated model, which probably has two stages, so the inference time is worse (and maybe even train time)

Reminds me of using quadtrees for image compression. The lower detail parts would be covered by a large single color-value block while more detailed parts would use smaller and smaller blocks.

https://github.com/alpharaoh/Quadtree-Compression

It seems to work okay without this, but an approach with less than totally free positioning would be to send in one more set of overlapping tiles shifted by half a tile size. Or better yet, image pyramids, which would help you handle different image sizes.

That’s significantly more data either way though, which is probably the issue.

Reply on news.ycombinator.com