Wit: Wikipedia-Based Image Text Dataset

github.com

84 points·jonbaer··8 comments

Please stay tuned and we will share the details about how to download WIT dataset.

But.. why post it at all if you're not going to give a download link? Arg, such a tease. (This is also technically an "announcement of an announcement" since there's no substance here, aside from the paper at https://arxiv.org/abs/2103.01913.)

Kudos to the effort. But where could this dataset be used specifically?

You would use this to train your own CLIP/ALIGN model, most obviously, to give you image classifiers on steroids. DALL-E/M6 would also be trainable with this. (All at a smaller scale, but this is a good starting point.)

11.6M images with 37.6M captions isn't exactly small scale. Isn't it within an order of magnitude of DALL-E? I seem to recall something like ~47M images, but my memory is often off.

I think the DALL-E dataset is 250M images. I think a possibly bigger issue is that the captions in this dataset are spread across many languages, which isn't ideal for doing something like DALL-E.

DALL-E is 250m: https://arxiv.org/pdf/2102.12092.pdf#page=14 (so 20x larger). CLIP (https://cdn.openai.com/papers/Learning_Transferable_Visual_M...) is 400m (34x larger). And ALIGN (https://arxiv.org/abs/2102.05918#google) is 1800m (155x).

(For 47m maybe you're thinking of a cleaned subset from YFCC100M or something that one of them used, but they all use total images an order or two larger than Wit.)

I'm not familiar with ALIGN, what is this?

that is a pretty smart data augmentation idea

Reply on news.ycombinator.com