Hacker News Reader

Comment by gscott | Hacker News Reader

Craigslist doesn't have an open system. Craigslist blocks ip's of those accessing there system through a web application and they send out lots of cease and desist notices. It just is not worth the time to develop something that might even benefit Craigslist.

Piggybacking on bigger sites is a good idea, just find one that is interested. Paypal did for eBay and it worked out for them, There is a lot of room to develop things using ebay. For example, maybe a craigslist format but for ebay auctions.

View on HN

That's a pretty serious claim; any pointers? If you've got some I'll be sure to nag Craig about it next time I see him.

listpic?

http://mashable.com/2007/06/08/listpic-craigslist/

Ha, with EC2 you can get around that now. Separate your website IP from your crawling IP, and every time CL blocks an IP, switch to a new one. Eventually they'll have to block the entire AWS range, but that's okay, you can crawl over cablemodem/DSL connections that use DHCP. What are they going to do, block Verizon, Comcast and Time Warner?

Then you can get around referer [sic] blocks for your links to CL that the user clicks on by using a redirect, I think.

You can slurp the entirety of CL daily without causing them traffic problems. I mean it's equivalent to each page getting one page view per day, which is nothing. Just keep track of URLs so you only slurp new content, and serve thumbnails off your own hosts (it's fair use).

Why would you go to all that effort if they don't want to play with you in the first place? If they really wanted to block you, they'll do it through their lawyers.

That stuff is really easy.

And you may convince them to join you after all, like YouTube and the media companies they parasite off of.

I didn't say it was hard.

It's also very easy to differentiate between a crawler and a user. But they don't even have to do that. Blocking the entire AWS ip range isn't hard either. I doubt any craigslist users browse the site through AWS :)

So the issue is: 1) they were putting ads on top of Craigslist, and 2) they were using up too much bandwidth. I expect if you actually wanted to build something useful you could just email Craig and he'd get you a dump.

There have been a few others, one fellow made a search system (using the rss feeds) where you can search across cities (by using checkboxes to select the different areas). Craiglist didn't like that.

I guess the bigger question is, what would Craigslist like? I am not sure, I just have not seen anything survive yet because Craigslist hasn't liked anything anyone has done so far.

One example, I used to want to search for craigslist items regardless of location. I found some sites that briefly offered this service, but they were requested to stop by CL.

"Craigslist doesn't have an open system. Craigslist blocks ip's of those accessing there system through a web application and they send out lots of cease and desist notices"

Screw it. Right now, a group here should build a newer, better Craigslist. Enough of you guys are in touch with the SV/Bay scene to bring the eyeballs required to kick it off, because as I see it, it's not a technical problem but rather a marketing one: what you need is a brief but intense "attention burst" to launch something like this.

The bleeding-edge users are all here, or only one or two degrees removed, and could make it work. [Don't look at me, I'm stuck on the East Coast.]

"Paypal did for eBay and it worked out for them", which is true but it should also be noted that it worked only after eBay launched a competing product that PayPal was able to beat despite the enormous distribution advantages that an eBay-run competitor had.

wonder why he does that..... cl doesnt make money off page impressions.... seems like closing the system is just hurting himself

Reply on news.ycombinator.com