Badly-Compressed Images Affect CLIP's Performance, New Research Contends

Kind of an odd blog post -- is this AI generated? Or maybe written by someone who isn't a native English speaker? The writing feels off.

In any case, maybe worth linking directly to the paper, it's an easy read and more informative: https://arxiv.org/abs/2311.14029

The tl;dr is they tested CLIP variants for zero-shot classification on CIFAR-10 and STL-10 at different levels of image compression and observed how performance dropped with more compression. They claim the purpose is more about demonstrating performance change attribution using this Integrated Gradients method but imo it doesn't appear more informative than just the general, intuitive observation that decreased image quality = decreased CLIP performance.

> CIFAR-10 and STL-10

> resized to 224x224

> susceptible to DCT compression

cifar-10 and stl-10 images are 32x32 and 96x96.. isn’t this actually a demonstration of locating where the textual encoding decision boundary is most fragile when the test image distribution is much lower resolution? or am i misunderstanding what the integrated gradients method uncovers? there’s a lot less ‘global’ semantic information to uncover and do inference upon when there’s a lot less ‘globe’ (visual information) to begin with, right?