This seems like a pricing hack that burns resources, that when the loophole gets closed the price of OCR will have to rise?
Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years. That trick being to rasterize 8bit characters into 8x8 pixels in a big image. 60% in a market worth trillions of dollars.
or
Anthropic's marketing team arbitrarily prices tokens to drive growth, according to vibes and feelings, and didn't think they needed to price images on par with text in their rush to burn cash & drive growth. Some folks take advantage of the trick during the first few days of the model's availability before Anthopic corrects their pricing, to align more proportionally with actual compute costs.
It's not a 60% percent reduction in cost for 100% of the same output. If you have a model and input text A, and you fix the seed etc. and run Text A through the model as text tokens and as compressed image tokens, you will not get identical outputs. You're specifically reducing the number of tensors needed to represent your input, which saves you on raw compute, but also by definition gives you less room to represent the information in your input. It's lossy, in other words.
Put another way, if you're using a model like Fable because you need the absolute frontier of capability and cheaper models cannot solve your tasks, then there is a very real chance that a compression strategy like this drops Fable's accuracy such that it's no longer suitable for your task. Which defeats the point of you paying for the most expensive model in the first place.
So, it's cool research. Might be useful for some people. Probably isn't something that has incredible utility in real use cases.
To me compression implies smaller size? However new line chars seems to be removed in the pic so I guess it could be expressed in fewer bytes than the original text with further compression ...
You could also imagine models where text tokens cover many characters and image tokens just a few pixels, which would invert the relationship, but this is typically suboptimal for the applications people have in mind when they train a model.
What makes the DeepSeek-OCR and related results exciting to some researchers is less about the fact that you could devise a tokenization scheme that has fewer tokens, and more about how well it works.
DeepSeek published a pretty well circulated paper on exactly this many months ago. It just hasn’t been attempted and shared publicly, asa retrofit, AFAIK.
Also, it’s no free lunch, the readme indicates that this “use images” hack is lossy and reduces success rates alongside the reduced cost. Most labs would focus on success increases regardless of price.
This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse
Sending an image of text instead of text reduces the number of input tokens, but they're still being processed by the model at the same precision. This probably also hurts performance in some way – the question is by how much.
The image trick reduces context because it’s lossy. The README says you can’t use it for anything needing exact recall. It produces a gist of the input.
You could achieve something similar by using a small, cheap model to pre-summarize information for the expensive LLM. This is what many people do already and it’s a much better way to do it for most situations.
Also I don't think you realize how much dumb stuff is still left on the table. That the market is worth trillions is quite irrelevant here given the dynamism of the field.
A text encoding uses 8bits per character on average, tokenization further compresses that
An image font would be 25 bits if 5x5, and most fonts are 12 pixels high
Of course it isn't efficient, this is a pricing inefficiency and a hack to exploit it (even the author describes it as an exploit)
Text tokens are high-dimensional vectors, not 8 bits per character. Every token has a deep embedding, e.g. 1024 float values per text token.
DeepSeek-OCR proved 10x+ compression from visual embedding of text, which was a groundbreaking result. [1]
Very cool to see OP's project hacking on this principle. It's still not lossless, as noted in the github, but is a promising research direction.
[1] https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
And we're talking about images of texts, not images that represent complex imagery such as a very detailed scene or what have you.
The paper notes two things:
1) While the compression ratio for visual text is better than it is for regular text, but the absolute space required is still higher for the images. OPs were talking about the space required, not the ratio.
2) The results of the OCR must still be fed into a text-based LLM for linguistic processing. Otherwise, all you have achieved is turning an image into a bunch of text.
You are conflating tokens with embeddings.
Tokens fit in a single word, modern gpt uses a vocabulary with 200k possible values, which would fit into 18 bits.
Have a good one
The top line can be the OCR-able instruction on how to decode the rest of the image, and the rest of the image would be random-looking colourful palette. It might not even need to use 8 bits per character, since ANSI is 7 bits/character.
You can achieve this by changing the extension of an image file from .bmp to .txt
Guys, not to be mean, but maybe chill with the state of the art research and go back to studying fundamentals.
Download the .bmp file and open it with 'edit' on windows or 'nano' on linux. You should see text. The bytes that precede the text are padding and a header that roughly 'tells how to interpret the data', in the sense that it describes the dimensions of the 'image'.
There's python code that shows how the file was generated, note that it just writes a string to the file and then wraps it with a .bmp header so that it can be interpreted by image viewing programs.
If you open that image with an image viewer you will see an image with colorful pixels. Whether you send it as an image or as text, the only difference lies in what you declare the file to be. For example if you change the extension, operating systems will "open" it with the appropriate program by default, but the data is the same, there's no difference between ASCII text and the encoding you describe where each subpixel describes one character, it's the same thing.
Your idea about using 7 bits instead of 8bits is more space efficient, but you'd need to decompress it so that the text is byte aligned. In terms of costs, you are not charged by the bandwidth consumed in transit, but by the computational cost, which wouldn't change as the text would need to be expanded to 8bits per char, (and then converted to the same token anyways)
Again don't want to be mean but these are 1st year comp sci topics, not at all related to LLMs. Hence why I recommend studying fundamentals more than papers that look like bleeding edge LLM research.
Educate me: what is an "optical token" when dealing with LLMs?
It kinda makes sense too. Because while people do read code word by word, we often "glance over" it and do roughly pattern recognition on it to know what it does. Only homing in on something when we need to answer a specific question. I think humans kinda naturally do this exploit anyway