Hacker News Reader

Comment by mfrye0 | Hacker News Reader

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all.

We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information.

Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.

View on HN

Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?

Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.

They (Microsoft) have APIs for app/website integration, but that's all I know about.

https://developer.linkedin.com/

Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.

So this blew up today. Reviewing all the comments now.

1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.

> Yes, this is free to use ...

Including for commercial purposes?

Yes.

Seems at odds with your tos. https://bigpicture.io/terms

What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!

The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms

Reply on news.ycombinator.com