Hacker News Reader

Comment by gerpsh | Hacker News Reader

Dude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.

View on HN

I thought these multi-region searches were against Craigslist's terms of service.

Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.

They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.

I'm guessing that's because of AirBnB and how they got their start by essentially stealing posts from Craigslist.

I thought Padmapper got their start that way, by essentially providing a better apartment hunt interface than CL had at the time.

And they got the banhammer, too.

Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...

Which was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.

CraigsList might make them think twice. I don't think they want to overly exhaust their own resources, what they have works for them.

> I found that their anti-scraping defenses were really difficult to get around

What sort of defenses are they using?

I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.

They had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.

All scrapey automated stuff is. I doubt this will be alive next week, or tomorrow.

already appears broken.

What a shame to have invested so much in an app that will immediately be shut down by CL.

i'm not seeing this, has it already been cleaned up, or am i missing it?

I see it.

I clicked on the “search” button.

Then it sent 503 XHR requests:

https://i.imgur.com/R9D23iU.png

I know. It’s just a poc

Even a PoC needs basic security before being released publicly. It wouldn't even delay the release by a single day.

Proof of Concept is a vague term and doesn't give you any sort of protection or defense. If it's against Craigslist's terms you're taking on some risk by making and public and promoting it.

Reply on news.ycombinator.com