Search all Craigslist cities for a remote job | Hacker News Reader

Search all Craigslist cities for a remote job

remotejobs.today

89 pointschovy55 comments

Dude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.

I thought these multi-region searches were against Craigslist's terms of service.

Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.

They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.

I'm guessing that's because of AirBnB and how they got their start by essentially stealing posts from Craigslist.

I thought Padmapper got their start that way, by essentially providing a better apartment hunt interface than CL had at the time.

And they got the banhammer, too.

Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...

Which was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.

CraigsList might make them think twice. I don't think they want to overly exhaust their own resources, what they have works for them.

> I found that their anti-scraping defenses were really difficult to get around

What sort of defenses are they using?

I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.

They had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.

All scrapey automated stuff is. I doubt this will be alive next week, or tomorrow.

already appears broken.

i'm not seeing this, has it already been cleaned up, or am i missing it?

I see it.

I clicked on the “search” button.

Then it sent 503 XHR requests:

https://i.imgur.com/R9D23iU.png

I know. It’s just a poc

Even a PoC needs basic security before being released publicly. It wouldn't even delay the release by a single day.

Proof of Concept is a vague term and doesn't give you any sort of protection or defense. If it's against Craigslist's terms you're taking on some risk by making and public and promoting it.

What a shame to have invested so much in an app that will immediately be shut down by CL.

Craigslist will send you a take down notice in a week, this is explicitly against their terms. I'd advise taking it down before a lawsuit.

That's basically what happened to Padmapper, sadly.

With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.

I feel like anybody who quibbles about scraping has their head up there. It's like, are you on the internet or not? Yes? Then you are on the internet.

Maybe timeliness is the important thing to Craigslist.

If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.

I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.

Cool idea, but the implementation is kinda crazy. Good enough to just test the idea at home on your own machine, but pretty risky to expose it to the world and let a ton of other people go nuts making bazzilions of requests.

Also, exposed API key in the request.

I'm kinda curious how telecommuting to this job actually works. https://detroit.craigslist.org/okl/lab/d/asphalt-laborers/68...

Robots probably.

Looks like we're performing a DDoS against https://api.scraperapi.com (the underlying data source) at this point.

You should really consider caching results and only fetching them again after a certain amount of time has passed since the last time you scraped them. This way you're not blowing through your ScraperAPI limit.

Play with dog remotely? https://i.imgur.com/OebiBWr.png

It is a very popular request.

Can you stack up on those and make 5 grand a month?

Might want to take this out:

console.log(items);

I usually set those to 'debug' instead, so I can filter them back into my JS console:

console.debug(items);

The API it's calling seems to be overwhelmed at the time of this comment. The console is full of 429 (too many request) errors

should be good now. i upgraded my plan.

It'd be nice to have a loading indicator.

Thank you for this comment. I considered the webpage dead.

Wow, I stopped it at 400 requests. I'm not sure how the site hasn't been shut down at this point.

This is an amazing project! Probably one of the most useful ways to actually find quality job listings. If only Craigslist would offer site-wide search natively.

Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.

Its cool but the main problem I am facing with searching for remote jobs. Is that most remote US jobs are not technically remote. They are for US citizens only, so I can't work on it from Europe.

Hmmm.

"You're sending requests a bit too fast! Please slow down your requests, or contact support@scraperapi.com with any questions."

what I see the freelance job is playing with dog..... should there be a playing with cat job lol

thosse freelance dog walkers are huge spammers. I've already removed duplicate titles across cities and you still see a ton of them.

Can you just search Google for “<term> remote site:craigslist.org” for similar results?

Just enter your search phrase like "node.js" or "javascript"

All you critics are missing the obvious blue ocean market in remote dog entertainment services that clearly is begging to be served.

nice it doesn't work.

I, too, entered a search query, pressed go, and witnessed absolutely nothing happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real time from the client, so you have to wait a bit for results to pop up. It'd be nice if they had a loading/progress indicator.

I'm working on an indicator next.

Fill us in on your thoughts for not believing you will get sued by craigslist after multiple comments in the thread.

would this not fall into a similar category of happened to linkedin? The data is publicly facing, and not behind a paywall or a password, therefore, basically fair-use of internet?

ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.

hmm...doesn't work in chrome

Spamming 429s in console, oops!

i need a job

Reply on news.ycombinator.com