Hacker News Reader

Comment by jsnell | Hacker News Reader

> This isn’t an astronomical number, but it’s large enough that we (at least, as a bootstrapped company) have to care about cost efficiency.

... by externalizing the costs to a third party.

In general, I'm really surprised that they published this article. It's like they described exactly the data that somebody working on preventing scraping would need to block this traffic, in totally unnecessary level of detail. (E.g. telling exactly which ASN this traffic would be arriving from, describing the very specific timing of their traffic spikes, the kind of multi-city searches that probably see almost no organic traffic).

I just don't get it. It's like they're intentionally trying to get blocked so that they can write a follow-up "how Google blocked our bootstrapped business" blog post.

View on HN

Or they just don't understand that what they are doing is illegal.

I'm always surprised by the level of ignorance, but I've seen more than one startup burn because the founders didn't understand which taxes were due and, thus, failed to account for them in their pricing.

Unethical? Yes. Illegal? How?

It’s not even unethical. Google publishes the data to the public for all to see.

Unethical to build a business scraping data from a company that makes money scraping data?

The Google Flights data is not "scraped." They interface directly to the airline reservations systems.

> what they are doing is illegal

It's not illegal. Google can sue them and bury them in court fees and potentially win a civil suit, but it sure as hell isn't illegal.

I'm pretty sure this is legal wrt CFAA because they are not circumventing any access control mechanism.

However, 300k requests per day surely is enough that this could be considered a sort of denial of service attack and violate fair use.

If Google wanted to, they would scale down their servers for a day, wait for this traffic peak to hit, document how it made the service unavailable to others, and now you have a valid offense to sue for.

> now you have a valid offense to sue for.

Yes. That's also what I said above when I said "Google can sue them". It's not illegal is my entire point. You can sue anyone, for any reason, at any time, and even win. That doesn't make something illegal.

If Google sends them a C&D, and expressly forbids them from doing this activity, and implements technical measures to prevent them from doing so, and they continue doing so then they may start approaching the area of illegal (Craigslist v. 3Taps would agree, hiQ v. LinkedIn would disagree).

> what they are doing is illegal

illegal means violate criminal code.

> now you have a valid offense to sue for

That's civil, not criminal.

It's pedantic, but words matter.

They are using residential proxies to evade rate-limiting by google. I don’t know if it’s enough to trigger CFAA, but it shows at a minimum that they know that what they are doing is not what google considers fair use.

I am surprised people think this could affect the servers? If they scrape 300k prices from 25k pages a day, and the crawl runs every hour from 15-22 UTC, then that means the 25k is spread across 3.5k pages per crawl. Even if the crawl is aggressive and completes within a minute that is still only 58 QPS.

The court decision in the hiQ vs LinkedIn case limits the CFAA's reach and scraping of publicly accessible data "likely does not violate the CFAA".

https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

> denial of service attack

300k requests / day is a little over 3 per second. That's not much.

? https://news.ycombinator.com/item?id=22180559 i dont think it's unethical in the age of ad driven web

Scraping public data is not illegal in the US.

> I'm always surprised by the level of ignorance

Such as the ignorance displayed in your comment?

I agree with you in principle, but having worked on both sides of this, I think there's very little chance they get blocked at their current traffic levels.

I do think that if they ever get traction they'll have a lot of problems - there's a reason GDS access to flight availability is slow, expensive, and difficult to implement well. Scraping definitely won't scale.

> E.g. telling exactly which ASN this traffic would be arriving from

The article mentions that they are using rotating residential proxies.

Why would they need those proxies if what they are doing is fully legal?

You do know that "legal" means "against the law" right? And when you say something is illegal, then you need to produce the law it violates; there's no proof from a company trying to prevent you from doing it.

Ah, thanks! I missed that part in the writeup.

There's a very high chance this is monetization of botnets, which adds even more to the overall shadyness.

This is almost certainly guaranteed to be either monetization of botnets, or inadvertently installed adware.

Reply on news.ycombinator.com