Silly Image Compression Idea
snufk.in
snufk.in
From the abstract:
Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations...
Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than 1000× on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.
Pretty cool stuff!
[.] https://openaccess.thecvf.com/content_cvpr_2017/papers/Hu_Fi... Figure 9
Like regular JPG encoding, this can be set to an arbitrary level of "good enough."
https://en.wikipedia.org/wiki/D%C3%BCrer%27s_Rhinoceros
> Dürer's Rhinoceros is the name commonly given to a woodcut executed by German painter and printmaker Albrecht Dürer in 1515.[1] The image is based on a written description and brief sketch by an unknown artist of an Indian rhinoceros that had arrived in Lisbon in 1515.[2] Dürer never saw the actual rhinoceros, which was the first living example seen in Europe since Roman times.
And why stop when the image is unrecognisable? Why not translate the image of (say) a cat into an image of the word 'cat'? And if you've done that compression, take it a step further - translate the image of the cat into the text string 'cat'. Now you're getting seriously good compression ratios :)
You could fine tune some existing text to speech model to sound like you. Once you transfer that to your friends/frequent contacts you could have voice conversations over very low bandwidth channels.
There are obviously plenty of caveats to this, like communicating sounds that aren’t speech. It could be a useful fallback e.g. if your network quality is seriously degraded then your client switches to TTS mode.
I don't think I've ever had a functional connection slower than that. You need packets to be making it through in a timely manner if you want a voice conversation, after all.
I _may_ still have a 300 baud modem in a box in the garage. Possibly a 1200/75 one too. (1985 sometimes doesn't seem _that_ long ago...)
https://blogs.nvidia.com/blog/2020/10/05/gan-video-conferenc...
This idea - trying to investigate behavior at different scale parameters - is also related to renormalization. But I think you can go further than this - currently you are just preserving that which remains consistent. You should also somehow store the complement of that information - the difference between the original and what you get after downscaling then upscaling the downscaled version again.
I imagine it's going to be mostly the noisy data local to certain regions of the image, some kind of high frequency details. But if you keep track of both, you can make each rescaling bijective, making your 'compression' invertible, hence lossless, in some sense. Whether you get any size savings doing that is another question altogether :^)
For computers, the question is meaningless because they only interact with our outside reality in the form of a "Chinese room" translation task. There's nothing inherently salient about image data to a computer, because the "ground floor" reality of computers is instructions for moving electrical charges around on some chips. But, given computers with built in cameras and having some use for discerning its reality of objects in 3d space, subject to Newtonian physics, and projected down to a 2d CCD array, the ideal compression would be subject to the same pressures as influenced the evolution of the human eyes and brain. Thus it should be similar.
Isn't it funny how similar compression artifacts can be to the visual distortion of psychedelics.
You have a good point! Though I wouldn't say the 'same' pressures. We can actually identify differences in those pressure, and use those to speculate.
Eg human computational capacity does well in terms of total operations performed, but individual units are really, really slow compared to electronics. That means human hardware can't run anything that requires a lot of 'serial depth', everything has to be almost embarrassingly parallel to work, or involve a lot of lookup from caches.
Also the design of humans is constrained by what evolution can do. Eg we still have blind spots in our eyes, mostly because evolution can only take 'small steps' (small in the space of genes), and inverting the eye so that the light receptive cells are in front instead of behind the nerves and blood vessels is not a small step in that space.
The power constraints on humans and computer are likely to be different. Humans are also not networked with sub-second latencies to the rest of world.
(Just to be clear, the examples I gave are second-order corrections to your much more important first-order idea!)
Though: I can believe that our arrangement is a local optimum, sure. (Local in some genetic sense.)
That still doesn't make me believe that are anywhere close to a global optimum. Behold the octopus:
> Do these design problems [of optical blind spots] exist because it is impossible to construct an eye that is wired properly, so that the light-sensitive cells face the incoming image? Not at all. Many organisms have eyes in which the neural wiring is neatly tucked away behind the photoreceptor layer. The squid and the octopus, for example, have a lens-and-retina eye quite similar to our own, but their eyes are wired right-side out, with no light-scattering nerve cells or blood vessels in front of the photoreceptors, and no blind spot.
From https://www.pbs.org/wgbh/evolution/change/grand/page05.html
When I asked how the compression part would be implemented, the author called me stupid for asking this question and told me the project was inspired by https://en.wikipedia.org/wiki/Sloot_Digital_Coding_System
To decompress you then just had to find the string of bytes that generated a matching CRC code.
It seemed so brilliant, but something told me it was too good to be true, and surely someone would have come up with this before me... So I tried figuring out why it wouldn't work. Still, took me a day or so before I figured out how long "just" was and that collisions would render it moot anyway.
Learned a lot from that though, as an aspiring programmer.
Needless to say, I never could solve that problem.
Have a look at compressed sensing!
You can’t restore a jpeg to its original image.
This isn't really true. JPEG is fourier or wavelet based. It will compress any image by discarding fourier coefficients, regardless if the input image makes sense to humans.
If the other creature's vision is in a different wavelength range, then RGB images are not applicable to it at all, in the first place.
What if there was a format that doesn't encode pixels but generates a "description" of the images using AI, of simple shapes to higher-level objects, and saves that in its own format, with a "decoder" that takes this input description format and constructs an image similar to the original, at least to some extent.
Of course this would have many drawbacks like encoding micro details and expressions of people, but still, I'd love to see how far the idea can go.
>Fleet Central refused the full video link coming from the Out of Band … Kjet had to settle for a combat link: The screen showed a color image with high resolution. Looking at it carefully, one realized the thing was a poor evocation…. Kjet recognized Owner Limmende and Jan Skrits, her chief of staff, but they looked several years out of style: old video matched with the transmitted animation cues. The actual communication channel was less than four thousand bits per second
>The picture was crisp and clear, but when the figures moved it was with cartoonlike awkwardness. And some of the faces belonged to people Kjet knew had been transferred before the fall of Sjandra Kei. The processors here on the Ølvira were taking the narrowband signal from Fleet Central, fleshing it out with detailed (and out of date) background and evoking the image shown.