Hacker News Reader

Comment by akudha | Hacker News Reader

Do you know if scraping this data violates their ToS?

Even if it is, it doesn't matter. The US Court of Appeals created precedent that allows for the scraping and aggregation of publicly published information with that LinkedIn case.

Got a link for this? Could open up some possibilities.

Here's the actual case:

https://law.justia.com/cases/federal/appellate-courts/ca9/17...

There is an attached PDF with the full decision, Pages 27, 28 on reference some related cases that informed the decision.

> publicly published information

Is Apple’s iOS App Store “public” though? Ostensibly it’s only open to Apple customers with an iOS device: accessing App Store pages in a web-browser just redirects you to the App Store.

> accessing App Store pages in a web-browser just redirects you to the App Store.

Not if you are on an non-Apple device. Or access an iOS App Store page from a Mac. The pages show just fine.

Google's entire business is scraping websites

Maybe that’s a reason to not feel bad if it violates the TOS but there are real consequences to doing so so perhaps your response isn’t that helpful.

Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.

Websites can choose whether to be scraped or not, and google respects that choice

It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.

You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?

No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.

To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.

By creating an account and using their webmaster tools giving up even more of your data?

No you just have to put a noindex header on requests to your website

So they still can use up my resources?

How can they know without looking? Again, you could also use the webmasters tools to block it...

Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.

How dare they!

> robots.txt can accomplish the same thing

Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.

>but there are real consequences to doing so

What consequences ? Breaking ToS can get your account suspended if you even need an account for doing this, not sure what else breaking the TOS really implies.

I don't know if Google can claim ownership of this data as it's customer generated and publically available.

If anyone really wanted to stop you, they would put up an anti scraping mechanism like Imperva and not even bother with legal action.

But mostly people are old or don’t care about solutions when they are too busy so it’s easier just to throw money at a lawyer and cause you trouble.

Sure, but they don't scrape anyone who opts out

In the US, there was a ruling a couple of years ago that gives some legal clarity to scraping data made publicly available (in favor of the scrapers) in the hiQ vs. LinkedIn case. You can read more about it here https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Dunno about Atlassian and Shopify, but I can’t imagine Google launching a case against someone for scraping them.

I guess robots.txt is all that really matters.

Here’s Play Store’s: https://play.google.com/robots.txt

agreed, provided the crawler complies with robots.txt

This data is quite valuable and services like this aren't competing with these app stores. If anything, they are probably inviting more competition to the app stores resulting in better apps.

There is no reason for these companies to go after some small guy scraping the app store data. But who knows - Craigslist went after one man shops. These companies might too.

hm not sure how/if they will react if this blows up but it wont stop me for now.

Reply on news.ycombinator.com