Hacker News Reader

Comment by biosed | Hacker News Reader

I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did little as the scrapers just created accounts. We had a few approaches that minimised the scraping:

Rate Limiting by login,

Limiting data to know workflows ...

But our most fruitful effort was when we removed limits and started giving "bad" data. By bad I mean alter the price up or down by a small percentage. This hit them in the pocket but again, wasn't a golden bullet. If the customer made a transaction on the altered figure we we informed them and took it at the correct price.

It's a cool problem to tackle but it is just an arms race.

View on HN

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

The real Jedi move

Especially if you charge for it, which would save them money, because they wouldn't have to redo their code every time you changed your website.

Yes. That's exactly what everyone should do.

If data is your competitive advantage or product, then what? Accept that your market no longer exists and that there's no way to stop theft?

You're going to need to explain how scraping publicly available information on a website is theft.

If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website, and should instead stick it behind an API with pay tiers and a very clear license regarding what you may do with it as an end user.

Note, a simple sign up being required to view a website makes it not publicly available information any longer and you can cover usage, again, in a license.

Then you have a whole bunch of legal avenues you can use to protect your work. Assuming you can afford it that is.

It is copyright information, no? So technically it is intellectual property theft if the scraping use is for commercial purposes.

No? If you place information publicly on a website it's pretty much free game, no copyright violation, especially regarding user generated information. That's my take, but legally it's a gray area and it's still going back and forth in the courts (at least in the US) but for a while before a decision was vacated by the supreme court scraping publicly available information on a site was legally protected and seemingly inline with my thoughts on it.

If we are to live in a mutually prosperous society, how is the labor and therefore well-being of the content creator improved by a web scraper? Does this precedent not injury future opportunities for exercising one’s life to making website data available for others to scrape?

By my understanding any website with a copyright disclaimer warrants their data as exclusively their own and are granting permission for other web users to generate it, ie people are not entitled to share their web data with anyone. So if they are, and we agree that it’s good that they do, and continue to create information for others to know, how do we avoid the implicit harm in extracting data without nothing being given in return but possibly harming the internet’s experience for everyone accessing the same information?

I'm actually cool assuming there is implicit harm and no benefit, but by that logic we need to tear google down too. I'm cool making that trade but it has to be done equally.

If you can't make that trade then you've weighed the value provided by an organization like google to be more valuable than the copyright of these content creators and I want other players who may want to be able to challenge google to have the same protections and access google does to have a chance at providing the same value.

Well actually, isn’t Google improving the value of the content ergo property itself by improving its accessibility? I was inferring a one-way street with the accumulated data that can lead to server crashes - which I don’t believe Google’s web crawl does at all (in fact that would be counter-productive).

I'd say most crawlers are looking to provide enriched value for content at their end use. Google is just an aggregator (the biggest by far) but other aggregators are looking to provide similar value.

So then an aggregator is different than a scraping service with respect to the value given to the rest of humanity? In that, in principle one adds value to the content creation and the other deducts through potential harmful interference with its reciprocity?

I'd say many aggregators do offer value to the rest of humanity but I imagine there are probably some exceptions and also not all scraper services offer no value it's just different value to different people.

Some scraping services make their money by offering scraping services to companies for specific information and you could argue they provide value to other businesses that way, but not to the broader "rest of humanity".

So I'm not sure it's as simple as just "aggregator" good "scraping service" bad as value provided takes on many different forms, and that's what makes this difficult.

I guess it may come down to your take on what you think of middlemen, because they are all effectively middlemen in the data economy.

Edit: I was rereading your comment, in respect directly to the value added to the content, then yes maybe it is more clear that aggregators are in principle different because they do add that value where scraping services that sell the data do not offer any enrichment to the content creator. I personally think protecting content aggregators that republish the data to create visibility or other value for the content creator to the extent that they're not worried about being sued for that is probably a worthwhile thing to happen because of the net benefit to our ability to find information/content.

Not all information falls under copyright.

If you build a database of touristic places and display in your website, the information is not protected by copyright.

In Europe they have laws covering _sui generis database rights_, but they are from another era and unenforceable nowadays.

> You're going to need to explain how scraping publicly available information on a website is theft.

Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them.

At some point people are gonna have to accept this.

I was responding to a question of whether it was theft, not whether such theft is morally grey or unstoppable in practice.

Yours is a somewhat orthogonal point and one I don’t entirely disagree with.

Well if you want to stick to the hard facts then it's even simpler: copyright infringement is not theft - those things are covered by entirely separate laws.

>"Do I need to explain that copyright is practically unenforceable in the 21st century?"

This sentence added nothing substantive to your comment, and made it rude; could you please be a bit more polite in the future?

https://news.ycombinator.com/newsguidelines.html

I was replying to a similar sentence, but it is true that in the end it did nothing but escalate the situation. I apologize and yes, I will try to be more polite.

For what it's worth, I didn't think it was rude. And I do think it was a substantive aspect of his response. It's a valid perspective even if I personally think it's somewhat orthogonal and incomplete.

What are copyright strikes, then?

Not even a thing in my country. Likely easy to avoid in others.

OP was talking about price lists. IANAL but AFAIK you can't copyright a list of prices.

That may be true. A list of prices might not be copyrightable in your particular jurisdiction. However I was only responding to the raw assertion made without any such qualification.

On the opposite end of the spectrum might be a photographer's website containing a gallery of their sample work. The fact that the gallery is openly published doesn't represent a relinquishing of copyright over those images.

No but those are substantively different situations such that this exact thing is being argued in the highest courts of the US. It's not quite the cut and clear case you seem to believe it to be.

I'm not American. I'm interested to read about these cases. Can you point me towards some relevant material, or at least cite the case name(s)?

Ah yup, certainly. So this is the one I was mainly thinking of -- https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Interesting. From what I can gather, the material in HiQ Labs v. LinkedIn was not claimed to be under copyright and that the argument being fought over in court was with respect to mechanically subverting access to a competitor.

You appear to have claimed that rights over material is broadly relinquished if it's published in public:

> If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website

And your distinction was further clarified when you argued that placing barriers to access fundamentally changes the equation:

> Note, a simple sign up being required to view a website makes it not publicly available information any longer and you can cover usage, again, in a license.

Perhaps you meant to speak only of material which is not subject to copyright? In which case I think your argument does track.

Mmm I’ll go with that, to be honest I saw that LinkedIn had initially made a complaint under DMCA (which HiQ then got an injunction for) as well and given how the case played out I was uncertain to the extent the case was signaling that you may be waving certain rights by making some content publicly available with no gating like a sign up.

No, but there is a legit philosophical argument about theft when it comes to copyright. There are two ways to look at theft: acquiring something you didn't earn vs. someone losing something they did earn. Generally, we tend to focus on the latter. From that perspective, "copying" is really not "theft", and arguably "copyright" does more net societal harm than any benefit it provides.

How practical is this really though? Like, imagine you're a newspaper. Unless you're the FT or Wall Street Journal or something like that, nobody is making an account to read an article. They'll just go somewhere else.

Well, not EXACTLY. The exactly should be to just do WebSub/PuSH. No need to invent your own thing and hope that bots learn how to use it properly.

Agreed. What I mean is people need to stop fighting these pointless battles.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers.

Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top.

This almost seems like a problem for AI to solve.

Even if your DOM is obfuscated, the rendered page remains vulnerable to OCR. Obfuscate the rendered pixels and you’ll annoy your humans and eventually find that the scrapers’ OCR is superhuman.

Still, maybe AI comes into it. Maybe poisoning the data is the right way to do it conditioned on ML-juiced anomaly detection.

pdf and print newspaper is still a massive pain in the ass to OCR accurately

This would have huge accessibility issues, breaking screen readers and the like.

To some extent those already exist and I get annoyed by them when they cause 1Password to be useless on their login page. But it probably would help with algorithmic scraping.

This is already common. It's mildly annoying for scrapers but generally a waste of time since you can usually still orient yourself based on the content of the nodes.

We already have react-native-web (<3), so we have that covered.

> It's a cool problem to tackle but it is just an arms race.

Plus, it's one you're going to lose. I was once asked at an All-Hands why we don't defend ourselves against bots even more vigorously.

My answer was: "Because I don't know how to build a publically available website that I could not scrape myself if I really wanted to."

> But our most fruitful effort was when we removed limits and started giving "bad" data. By bad I mean alter the price up or down by a small percentage. ... If the customer made a transaction on the altered figure we we informed them and took it at the correct price.

Is that legal? It would be a big blow to trust if I was the customer, but that's without knowing what you were selling and in what market.

It’s legal if it’s in the contract. Standard for contracts to allow for mistakes and confirmations of prices

It's not mistake if you do it deliberately!

Yes (not saying it's a mistake) but putting confirmation can be in the contract, no law says you only get 1 chance to display price.

I love the honey pot approach. Put tons of valued hrefs on the page that are invisible (css) that the scrapper would find. Then just rate limit that ip address and randomize the data coming back. Profit.

I think this falls into the "arms race" trap, though. If you can make an href invisible via CSS, then the scraper can certainly be written to understand CSS, and thus filter out the invisible hrefs..

Reply on news.ycombinator.com