Hacker News Reader

> A company has to pay to have this page, so this certainly does not include all companies.

I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.

Yeah apparently it's free, my bad, I remembered it incorrectly

> In any case, web scraping is a sort of gray area of the law.

I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

What rulings are you referring to? Been reaffirmed that scraping LinkedIn is legal multiple times, even by Supreme Court. https://techcrunch.com/2022/04/18/web-scraping-legal-court/

I believe that decision was that web scraping doesn't violate the CFAA unless there are access controls (such as the need to log in) to get to it, but they specifically said that the ruling doesn't comment about other possible claims against it.

That's why I think it's still a gray area. But I could be wrong -- I haven't been following this stuff all that closely.

So it’s not gray, it’s melange.

Multiple court reaffirms in favor of scraping, including by the SC, seems more like verdant to me..

What rulings have been against it?

You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed.

This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right?

And all the other search engines spidering the web. Do you think they are risking their business being in the “grey?”

There does seem to be some distinction if you log into and scrape:

"Only three claims remained for the final order - the violation of the CAN-SPAM Act, violation of the CFAA and California Penal Code.

The district court then granted summary judgment to Facebook on all three of the remaining Facebook claims. The district court awarded statutory damages of $3,031,350"

https://en.wikipedia.org/wiki/Facebook,_Inc._v._Power_Ventur....

That’s completely different though as they are scraping private info that’s proprietary to Facebook.

I think there’s no grey area for whether scraping publicly available data is legal as it is legal.

http get requests are legal.

It reminds me of a similar conflict in public records.

Many municipalities charge for copies of public records. One can go to viewing rooms and examine the records at no charge. We all own those records as members of the public that funded them, and the place where they are kept.

Many municipalities want to prohibit photography because people taking their own picture of a public record does not involve the copy fee.

The municipality confuses access to records as a part of their fee, and that is where attempts to limit photography come from.

The fee actually funds the work necessary for an official copy to be made, optionally stamped to be admissible in court or accepted as an "original copy."

A photocopy of a death certificate is no different than the one the city clerk made, except for the stamp and clerk being able to testify about making the record copy.

Some companies and government agencies will accept a death cert no matter what. Others want an official copy.

Getting back to scraping:

Clearly people have access and can make their own data copies.

Maybe the answer is for companies to make official data products available, or something along those lines.

LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

According to that wiki page, the courts eventually did rule that HiQ violated LinkedIn's terms of service, and a settlement agreement was reached.

Linkedin lost on the CFAA claims, there were others that they did not lose.

> LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law

That would appear to be false, since after appeals, there was a settlement wich would seem to make it pretty clear that hiQ lost:

https://www.natlawreview.com/article/hiq-and-linkedin-reach-...

> In any case, web scraping is a sort of gray area of the law.

I feel like it's also only a matter of time before we have decentralized web scrapers that can consume data and assemble datasets and store the results in a decentralized fashion in a way that is completely non-enforceable. LinkedIn data would be aggregated in all sorts of interesting ways, and there would be literally no name you could sue, no address you could serve papers to, it would be a hundred thousand bots in 100 countries that collectively assembled the dataset and put it on some IPFS or torrent.

If data is visible with human eyes, it's visible by decentralized bots, the only missing piece is the technical and financial complexity of the current state of decentralized compute.

> If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.

Well. This is certainly not true.

Even with Premium subscription you can hit a hard limit of personal profile views while just actively going through profiles without any automation. And Linkedin will temporary restrict and threaten to ban your account.

I have lots of LinkedIn pages for various companies (alive and dead). I’ve never had to pay: while there’s limits on page creation for free accounts (which appear to be prevent abuse) there’s no direct payment required to create a company page.

I’m not sure if you’re accidentally leaking a new strategy that LinkedIn are about to launch or if you’re mistaken but given you’re claiming to work for LinkedIn, probably worth correcting your comment either way.

Definitely not the latter. I am referring to the former point.

The limit only applies if you’ve created… I can’t remember the number exactly and it’s not listed anywhere but I recall hitting a limit around 5 pages.

You can see that it’s free to make a page here — the only requirement is a free account: https://www.linkedin.com/help/linkedin/answer/a543852/create...

TIL

> and is not being monetized

It seems to have a paid version with more data: https://docs.bigpicture.io/docs/free-datasets/companies/#fie...

(just the fact, I don't have an opinion)

I’m on the same page. It also requires users to create an account in order to get the free version, and I would argue that in most cases that’s part of a monetization funnel.

:O