That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably get more visitors to this single blog post. Forgoing everything about CL blocking me and such, I never did enough to promote the site. I started the site in 2011 and had little experience with promoting content. As a newb developer, I was leery of promoting something I created. What if it broke? What if people didn't like it? When in reality, I learned over time, that most people frankly don't care. And that's not a bad thing.

The past few years I have been learning over and over: ship it, get feedback, improve, repeat.

So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=34...).

You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests

r = requests.get('http://craigslist.com) print r.status_code

This should print 200 and you're good to go.

Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'})

and r.status_code will return 404

bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base.

I created this gist: https://gist.github.com/wlindner/4752122 since the python code formatting is wonky