Craigslist Casualty: The Story of ApartmentsNearMetro.com | Hacker News Reader

Craigslist Casualty: The Story of ApartmentsNearMetro.com

williamlindner.com

29 pointswlindner25 comments

It's the inevitable fate for anything that uses CL for data. CL is too afraid of competition, yet refuses to innovate. It's a shame.

In defense of CL (which I have never before offered), I noticed some UI improvements recently: pic view, grid view, and map view.

These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere?

Whether they 'hate' competition is debatable. Clearly they view their data as THEIR property. They occasionally let things like PadMapper go while putting the ban hammer down on other things, very arbitrarily.

The Wired story on them about 5-6 years back suggested they are just very skeptical of third parties and human nature in general. It left me with the impression that Craig is just deeply selfish and authoritarian.

I've seen the UI improvements, but haven't needed to use them so I don't know how good they are. If those work: Awesome! In a way, that's a victory. If CL incorporates the best things about 3rd party sites into their own site, I think that's a big win for consumers. If CL refuses to allow 3rd parties to exist because they are skeptical of human nature. Talk about the ultimate irony: a site that lives off of people's willingness to trust strangers being unwilling to trust strangers themselves.

Good point. Here's the quote from the end of the article:

"These all signal Newmark and Buckmaster's wariness about what humans, including themselves, might do if given the chance. There may be a peace sign on every page, but the implicit political philosophy of craigslist has a deeply conservative, even a tragic cast. Every day the choristers of the social web chirp their advice about openness and trust; craigslist follows none of it, and every day it grows."

HN discussion: http://news.ycombinator.com/item?id=784663

CraigsList doesn't care if other websites use the same data to power their own services, as long as those other websites obtain the data directly from the source (i.e., the user) and not by scraping Craiglist.

If the application is blocked, can the author please openSource the script? If everyone starts using the script, I believe it will be quite impossible for CL to block it.

Yeah, that's a good idea. Craigslist basically blocked any connections from Google App Engine to their domain. But, if you ran the site from your own computer, they wouldn't really have any way to block it.

Similar story as https://www.padmapper.com/. It's sad that craigslist isn't willing to either update their application or allow someone to make it more useful for their customers.

I had that exact same pain point in 2011. It's too bad the Craigslist refuses to innovate or let others innovate on their platform.

Yeah! It almost seems like a source of pride for them... What were you working on that got squashed by Craigslist?

I was simply trying to locate an apartment near Metro. I toyed with the idea of scraping their site but remembered this Wired story from 2009 [0] so I just gave up on using them for my search.

0: http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...

What does 'innovate' mean in this context? Ditto for 'platform'?

Both terms are terribly over-used and make discussion challenging, since both sides end up assuming what they words probably mean (without every discussing the meaning).

If you want to compete with craigslist, build a better product and convince people that there is value for them to enter their data. That's what craigslist did.

FYI: this is unreadable on my WP8 device. The screen width is set smaller than the text width so I have so scroll left and right for every line.

Thanks for letting me know, I implemented a quick fix for mobile devices. Unfortunately I don't have a WP8 to test it on, but it shouldn't require scrolling left and right anymore to read the text.

Thank for fixing!

I think the easy fix for this (for the benefit of the poster) is to use a meta tag to set the "viewport" attribute to 100%.

I don't get why people say things like, "It's too bad the Craigslist won't ... let other people innovate on their platform." It's just childish in my view. Here's another child's toys - and, granted, he has a LOT of toys[1] - and he's chosen to share them to every child in the world regardless of their background, yet you're upset because he's not sharing them in such a way that would only allow people to play with the toys on another playground. "He" doesn't want that - he wants to play, play, play with everyone. So he decides to say "No!" to anyone who wants to take his toys off of his playground.

If you don't like the way "he" plays with his toys, then tough cookies - "he" doesn't care and neither does anyone else except those who want to also complain about him not sharing in the way they want him to. You'll see this damn near every time someone posts a "Help! My API provider just changed the rules!" type of post. There are two types of responses: "I hate that - they should be more open", or "It's ridiculous to even think that you can build a business based off of leaching from someone else and that that other entity would be okay with that forever."

[1]Unlike children (who typically are given their toys), these toys are the result of "his" work.

Unlike children (who typically are given their toys), these toys are the result of "his" work.

Er, no. They are the work of everyone who posts on CL.

So there's Walk Score. Apartments near (Washington) metro example — http://walk.sc/YPUq1a

This doesn't really solve the problem of Craigslist hoarding all their data to themselves (as pointed out numerous times already), but there are enough other sources to make it usable.

`http://trycourier.com/?utm_source=anm-blog` linked in the post seems to 403.

Sorry about that, it's just http://trycourier.com, I fixed the link. Thanks!

> Had around 1,500 unique visitors with an average time spent on the site of about 7 and a half minutes.

With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.

That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably get more visitors to this single blog post. Forgoing everything about CL blocking me and such, I never did enough to promote the site. I started the site in 2011 and had little experience with promoting content. As a newb developer, I was leery of promoting something I created. What if it broke? What if people didn't like it? When in reality, I learned over time, that most people frankly don't care. And that's not a bad thing.

The past few years I have been learning over and over: ship it, get feedback, improve, repeat.

So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=34...).

You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests

r = requests.get('http://craigslist.com) print r.status_code

This should print 200 and you're good to go.

Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'})

and r.status_code will return 404

bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base.

I created this gist: https://gist.github.com/wlindner/4752122 since the python code formatting is wonky

craigslist is hiring engineers. Interested?

Reply on news.ycombinator.com