A list of the biggest datasets for machine learning | Hacker News Reader

A list of the biggest datasets for machine learning

datasetlist.com

195 pointsnickplesha30 comments

Open articleView on HN

Missing Danbooru2018, released Jan 2019:

> Danbooru2018 is a large-scale anime image database with 3.33m+ images annotated with 99.7m+ tags

https://www.gwern.net/Danbooru2018

92.7m+ tags!! How did they annotate the images? Was it done manually by humans or a neural network?

All humans, believe it or not. Never underestimate the power of procrastinating grad students or anime weebs.

At present, the most advanced tagger for Danbooru, DeepDanbooru https://www.reddit.com/r/MachineLearning/comments/akbc11/p_t... , still isn't good enough to do annotation by itself but I think that's mostly because no one has really tried.

From the website

> The image boorus are longstanding web databases which host large numbers of images which can be ‘tagged’ or labeled with an arbitrary number of textual descriptions; they were developed for and are most popular among fans of anime, who provide detailed annotations.

i.e. it's pulling from a public website which crowdsources the tagging

The Reddit archive is a glaring omission given its recent use in OpenAI's GPT-2 model. https://www.reddit.com/r/datasets/comments/65o7py/updated_re...

Thanks for the feedback! I'll be adding it to the list.

Neat.

Would have been nice to add some sort of discriptor indicating what type of dataset it is. For example, I personally have no clue off the top of my head what the “MURA” dataset is.

Edit: I now see there is a little icon on the left of all lines. Bit ambiguous, but it sorta gets the point across.

Great job! A categorization system (especially if it can be crowdsourced/edited via github) would be great, since these datasets are likely to be useful across multiple domains - and not just "CV" or "NLP" - I'm thinking "Stocks", "Finance", etc.

Also https://registry.opendata.aws/ from AWS has a lot of datasets that could either be included en masse, or even linked to for the page to be more comprehensive. I like their categorization system (tags/labels) as well. They also have usage examples which is excellent to get a sense of what the data is/useful for.

Thanks! Those are good ideas!

Is there anything that stops you from adding a link to the dataset/torrent for the ones that are freely distributable?

Good idea, a link to academictorrents seems like a good candidate for adding to the table. I'll think about how to fit it in there without overloading the page with information. Thanks!

I really wish there was a large-scale corpus of completely benign email traffic.

Enron? https://www.cs.cmu.edu/~./enron/

500K messages from 150 users.

Now I’m just imagining a bunch of NLP models subtly learning large scale fraud.

AGI needs to be adept at fraud too, you know ;-)

Isn't it beautiful that there is a meaning of "benign" that does actually apply to these emails?

Amazon hosts a dump from the Apache email archives as of July 2011. IIRC, it contains about 7 million emails.

https://aws.amazon.com/datasets/apache-software-foundation-p...

/r/datasets on Reddit also has some surprisingly interesting datasets.

/r/opendata on Reddit is another one worth checking.

Stanford Cars dataset: http://ai.stanford.edu/~jkrause/cars/car_dataset.html

Let me be the first to say thanks for taking the time for compiling and sharing this list! Bookmarked it for future reference / use :)

I would like to see size of dataset, a form to submit datasets, categories (such as person, face, animals or plants etc.)

Thanks for the feedback, all seem like good ideas.

If the word "biggest" is being used, then some indication of the size of the dataset would be useful.

Thanks for the feedback, perhaps the stats and numbers are too well hidden in the description field. I'll see how to make it easier to gauge the size of datasets at a glance.

You should add this movie dialog collection: https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_C...

Alot of projects use it!

Thanks!

Does the licensing of a dataset also apply to the trained neural network?

Thanks a lot for sharing this :)

misses a ton of important datasets and corpora but anyways, interesting how so many of them are non-commercial only :(

Especially annoying are european universities and other government funded institutions

Reply on news.ycombinator.com