Open Source LinikedIn Scraper | Hacker News Reader

Open Source LinikedIn Scraper

13 pointsdevildani21 comments

Hi All,

I am planning to build a LinkedIn scraper, which would scrape LinkedIn profiles, company pages, groups, searches (sales nav and normal searches), likes, comments, etc, for free. I already a set piece of codebase that I have built out. I wanted to know if anyone would be interested in using an open-source LinkedIn scraper? Will this be good option?

View on HN

I would be interested. I think LinkedIn aggressively tries to shut down things like you are trying to build though

Good to know. Can you share your email id? I will ping you as soon as I have the first version of application ready.

As if there was not enough recruiter spam mail.

This is why I private my social media accounts. There is already enough spam and I don't feel it needs to be any easier.

There are services that do that, might not be open source but with free tiers like common room.

LinkedIn might not like you scraping the site as well, esp if you use sale navigator to get access to deeper info. Keep that in mind

Yes, I am aware about this and I won't be using my account for extraction. Users will put their session cookies for data extraction.

Great idea. I love a tool that unexpectedly gets my account banned.

lol

Facts, they will ban your LinkedIn account for scraping. Good luck!

I work with someone in Spain and asked her to log into my Recruiter Lite account to review candidates. LinkedIn gave a pretty stern warning about sharing the account with anyone and that I’d be banned if it happened again.

I’m guessing that the same thing would happen if you started letting a scraper act on your behalf.

From https://www.linkedin.com/robots.txt it says to contact whitelist-crawl@linkedin.com

That's just for search engine crawlers, not random people who want to scrape all their data

Ironically, sometimes I wish that LinkedIn vs HiQ verdict went the other way and made scraping LinkedIn illegal

The judge ruled towards LinkedIn in the latest ruling of that case, which I suppose most don't know about due to it being an appeal.

I did read about this. Not most know about this.

I’m not a lawyer but does that mean scraping LinkedIn is indeed illegal? Or is it still yet to be determined?

TLDR: The short answer is: it is legal to scrape LinkedIn properly. However, scraping LinkedIn a certain way could be illegal.

Scraping data from public LinkedIn profiles? Ok. Scraping data from LinkedIn profiles with fake logged-in accounts? Not ok.

https://www.linkedin.com/feed/update/urn:li:share:7006059900...

I would be interested. How do you plan on dealing with IP address bans?

Use an LLM to do it

I am trying this out, I haven't really written any code to build the application. Generating codes via LLMs.

Reply on news.ycombinator.com