A collection of free datasets from Microsoft Research | Hacker News Reader

A collection of free datasets from Microsoft Research

msropendata.com

196 pointsactivatedgeek40 comments

I ask this not with snark but with fond(ish) memories, but does Microsoft providing open data like this make anyone else think of that Northwinds sample database that used to come with SQL Server?

Not sure if it still does or not, as I'm in a different DB engine entirely these days, but I kind of grinned with a wee bit of nostalgia checking out this site thinking of that. I cut my teeth on that data set.

Edit: Son of a gun they still offer it https://docs.microsoft.com/en-us/dotnet/framework/data/adone...

SQL Server ships something called Adventure Works since over a decade - same idea as Northwind but larger to also demo and play around with Analytivs and BI.

Adventure Works, pffft... I’ll take Scott any day of the week.

The Northwinds sample database predates SQL Server. I remember using it with Microsoft Access 1.0 back in 1993...

Heh, you probably read my post and thought "darn kids" to yourself, eh?

Also available in other flavours

https://github.com/pthom/northwind_psql

This page has problems loading the datasets. Is it overloaded or just broken?

Does anyone happen to know what licenses the various datasets are under?

Can confirm that I'm having the same issues, could be overloaded indeed.

Turn off your ad blocker. Fixed it for me.

How did you find that fix??! Just Curious..

Ctrl+shift+i in Chrome, Console, you may see something like "blocked by client".

Thanks!

Well if you are looking for a huge dataset, I would suggest stack overflow.

https://archive.org/details/stackexchange

I remember I wrote a node script to download all question and dump into my Postgres instance. It was fun, Postgres with an index could able to fetch results super fast

How many answers for this question?

How many questions are unanswered?

With gin index, I could do a free text search as well.

Wow, 230GB celebrity photo dataset caught me off guard.

Is it the one used in this paper?

https://youtu.be/XOxxPcy5Gr4

I thought for SURE this would be a rickroll video. Pleasantly surprised.

Dude watching this video is how I imagine a bad acid trip to be

It's worse. I think I damaged something in my brain by watching that...

This took me down memory lane to what could be done in the 1990's with a regular PC/Amiga with one of those special framebuffer cards that could render a frame at a time to a Digibeta D1 video recorder for your typical acid house pop video:

https://www.youtube.com/watch?v=TPFAYIr8z2I

Oh great!

On a similar note, does anyone know of a similar type of dataset/source (I'm expecting smaller) that's available for commercial use?

There's also a 2.3 GB dataset of celebrity photos from the web.

They don't make it easy to find out what the Microsoft Research License Agreement is, when they put it behind a login wall on a site that doesn't work with ad-block. But this appears to be a copy of it: http://www.cs.toronto.edu/~aditya/langid/msrla.txt

These license terms are a trap. Nobody should ever accept them.

You can't use the data for commercial purposes, which is harsh enough on its own -- it means the data should only be interesting to people who plan to be academics permanently, or who don't plan to succeed at anything they do with it.

You can't even open-source anything you make out of it, because open source software does not restrict the purpose for which it's used. You can't combine it with data under any Creative Commons license except CC-By or CC-0, because its restrictions are definitely not Creative Commons compatible.

The truly insulting part is that you have to give an unrestricted license back to Microsoft for everything you make out of this data. You can't benefit from what you make out of the data, but Microsoft can, with no limitations.

Whatever you make from this data, you can't benefit from it. Most other people can't benefit from it. Microsoft can benefit from it, and if they want to, they can just take it without crediting you.

This is like an artist being asked to do something "for the exposure", by someone who will just take it and give you no exposure.

"We're all in on open source." [1] Thanks for taking the effort to check. :) I believe this kind of attitude from MS is pretty problematic for Github.

[1] https://www.cnbc.com/2018/06/04/microsoft-ceo-satya-nadella-...

While I find it awful that this particular department of Microsoft is offering "free data" with poisonous terms, I see no reason to believe it has anything to do with GitHub. Microsoft is a big place.

I thought you are overreacting until I reached this part : " 3. That Microsoft is granted back, without any restrictions or limitations, a non-exclusive, perpetual, irrevocable, royalty-free, assignable and sub-licensable license, to reproduce, publicly perform or display, install, use, modify, post, distribute, make and have made, sell and transfer your modifications to and/or derivative works of the Data, including but not limited to the results and analysis of any tests that are run by or conducted by you with or for the Data, for any purpose."

"Not limited" - so basically any software processing it.

It's a different license. Here's a direct link to the Microsoft Research Data License Agreement (PDF): https://msrodr-api.azurewebsites.net//licenses/2f933be3-284d...

Disclaimer: I don't know if this affects your concerns, but it's at least the right license. Disclaimer 2: I work for Microsoft, but not for MSR, and don't know anything specific about this.

Thanks for the link. This version of the license has the same terms and then several more, including:

- If we accidentally gave you people's personal identifying information, it's your job to destroy it

- Your license to us is perpetual but our license to you terminates after 2 years (?!)

- The data is confidential and you may not talk about it for 5 years (OMG WTF)

"You can't use the data for commercial purposes, which is harsh enough on its own -- it means the data should only be interesting to people who plan to be academics permanently, or who don't plan to succeed at anything they do with it."

It'd also be useful for someone farting around with machine learning who isn't anywhere close to being at a point of shipping anything to anyone else.

What you can benefit from is what you learn while using the data. Then you start a new project using what you've learned, and using different data.

Why would you plan to fail? You could just use different data from the start that doesn't require you to sign an NDA.

It's not like MSR is providing a rare commodity.

"Why would you plan to fail?"

Because it isn't failure if your only goal is to learn.

It'd also be useful for someone farting around with machine learning who isn't anywhere close to being at a point of shipping anything to anyone else.

This is the only use I can see for this data under those license terms. And even then, you are still technically granting Microsoft a super-license to do whatever they want with whatever you come up with, and probably also not allowed to upload it to Github.

Absolutely nothing on this website works for me. Search gives an infinite spinner. Categories gives a blank page. What gives?

For me it was uBlocker Origin.

Confirmed uBlock causing it here too

Ditto

Are you sure that the culprit here it's uBlock Origin itself and not your configuration/filter lists?

I'm asking this because i've experienced similar problems with some pages and in most if not all cases it was the latter.

Do we really need more sample data sets

Yes.

I can't really fathom the snark here. Is providing this a bad thing?

I think the commenter above you missed that it's not "sample data", but curated datasets for machine learning tasks, which we most certainly need more of.

Well. In this case it happens to be a bad thing, because you have to agree to a poisonous NDA to download this data.

Providing actual free data would have been great though.

Reply on news.ycombinator.com