The Craigwatch Story | Hacker News Reader

The Craigwatch Story

craigwatch.com

146 pointsstickfigure48 comments

I used Craigwatch. It was a useful tool - when a search pops, I get an email. I used to use Google Reader for this but when Reader died so did my RSS reading habit, so now email is the only part of my daily workflow that I poll frequently.

The real mystery to me is why Craigslist shut the service down, because (protocols aside) it wasn't fundamentally different from any other way of consuming craigslist. Craigslist's terms of service is actually pretty confusing:

USE. You agree not to use or provide software (except for general purpose web browsers and email clients, or software expressly licensed by us) or services that interact or interoperate with CL, e.g. for downloading, uploading, posting, flagging, emailing, search, or mobile use. Robots, spiders, scripts, scrapers, crawlers, etc. are prohibited, as are misleading, unsolicited, unlawful, and/or spam postings/email. You agree not to collect users’ personal and/or contact information (“PI”).

...which if you think about it, bans the use of RSS readers too. There's a short blurb on the RSS page:

craigslist RSS feeds are for your personal use only, and are not available for commercial use without first obtaining a license from craigslist. Please consult our <Terms of Use> for more information on using craigslist RSS feeds.

The only relevant part of the Terms of Use is the section mentioned above, which doesn't really shed any light on the situation.

Honestly this is really confusing. Once again, Craigslist disappoints.

Craigslist has a really bad history of abusing its users. For instance, a few years back they changed the TOS so that CL becomes the copyright holder of all ads placed on the site.

So trying to fend off padmapper (whether one agrees with that or not) has suddenly become "a really bad history of abusing its users"? I guess scrapers are users.

I think the objection is that the initial knee-jerk reaction to try to fend off scrapers/competitors had a (potentially unintended) consequence of being pretty anti-user by claiming exclusive ownership over the user's content. Craigslist doesn't have to play nice with other services, or aid their competitors, but trying to fight competitors by overreaching and harming your customers isn't OK, which of course, is why they then backtracked on that bad policy after enough public anger.

You can safely disregard that part of the TOS. Because law:

(a) A transfer of copyright ownership, other than by operation of law, is not valid unless an instrument of conveyance, or a note or memorandum of the transfer, is in writing and signed by the owner of the rights conveyed or such owner's duly authorized agent.[1]

[1] http://www.copyright.gov/title17/92chap2.html

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as google.com or archive.org even if it respects robots.txt!)

I happened to review these terms because I too was sent a letter from Brian Hennessy representing craigslist last Friday regarding a little Chrome extension I wrote a couple years ago[2].

[1] https://web.archive.org/web/20131128233421/http://www.craigs...

[2] https://github.com/yonran/craigslist-shortcuts

How can Craigslist accuse you of violating their terms of use? The act of providing a chrome extension does not use their service. Maybe if this was some sort of copyright-circumvention software covered by the DMCA I could understand it. But this sounds like an obvious legal bluff, no?

IANAL, which is why I am asking these questions.

IANAL either, but I think it would be against craigslist terms for me to continue to use their website while “providing” my accessibility extension. So I could publish the extension, but I would have to stop using craigslist. Additionally, the Chrome Web Store developer agreement[1] prohibits publishing an extension that “knowingly violates a third party’s terms of service”, so my guess is that the lawyer would then ask Google to take it down since any user of the extension would be violating the craigslist terms and thus (after the C&D) I would be violating the Chrome Web Store terms.

It’s pretty airtight, so I took my extension down.

[1]: https://developer.chrome.com/webstore/terms

Except that your extension is running in Chrome which is a "general purpose Internet browser" and expressly allowed in CraigsList's TOS.

I'm curious as to whether you got a response to your refusal to "comply." Did he get back to you after that?

Not yet.

another service i used a lot was padmapper(?) (CL killed and then stolen the feature) and then another site that allowed me to search on all craigslist sites around the US. i used it to check the price of whatever i was buying locally. or to find things so rare around i am that i would be willing to risk having my money stolen and nothing delivered... i don't even remember the name of the site, but nothing have come even close to filing the gap. one would think ebay is the logical solution, but ebay is filled with crap that it is useless.

In the past, I have sometimes used Craigslist's RSS feature to watch for the appearance of items. That is to say, the ability of CL to export a keyword search as an RSS feed that you can save in your feed reader and watch for updates.

In what ways did Craigwatch improve over RSS? If it still operated today, why would I use that instead of scraping RSS items directly from Craigslist to my feed reader?

Did Craigwatch use RSS feeds, or was it scraping material from the rendered HTML?

I do that too, but it can be pretty late. At least hours after the item is posted. Which isn't bad for some things, but sometimes you _really_ want to know. :D

I've never actually heard of Craigswatch, but I think it's something most engineers have done at least a couple of times. I've got a script I use to watch the FredMiranda forums any time I need some new camera gear, for example.

Looks like it was using XPath to parse the HTML: https://github.com/beaulm/craigwatch/blob/master/post.php

I imagine the 'selling' point was users not even having to know what RSS even means, let alone how to script it.

Thanks for the link. Time to try out Google app engine's PHP preview.

Useful free stuff is typically gone in minutes. Only junk remains posted.

Its RSS feeds have a refresh period of 1 hour, which is simply too long to have the opportunity to grab hot items.

As Craigslist has very shallow category listings in its "free" category, where you can't separate furniture from clothes as an example, the site forces users to rely on manually refreshing pages, which change very often.

I have better user experience using Craigslist from my iPad. Craigslist appears to selectively make the service available to app makers but not to websites.

In the past 3 days of shopping for a specific car within about 15 Craigslist local communities, I managed to get automatically blocked by the service several times. What I am looking for is very specific and I don't mind traveling a bit to get what I want.

I ended up having to write code with sleep timers to reduce my number of web queries once I narrowed what I wanted.

An essential feature in the iPad app is that I can cross off listings that are no longer interesting to me and highlight favorites. There is no listing cross-off feature for the website.

I guess my next step is to add that feature. laugh

Wiseleo, is that really true about the refresh rate? RSS readers have a configurable rate. Often there is a one hour default in order to be reasonably nice to the server. Basically, a Craigslist RSS item is just a URL with the search parameters embedded. Maybe I'm wrong, but I suspect that whenever you fetch this URL, the server executes the search and produces the results as RSS XML items, so the fetch rate is controlled by you (your reader). Or are you saying there is some additional throttling on the server side, so that RSS-based searches do not see up-to-the-second updates that are visible through the Web interface? So that no matter how often you refresh the RSS feed, you don't see new items that are already visible via HTML?

Gotta love being downvoted for an informative post... :)

If you look at the XML returned for RSS feeds, you will notice:

<syn:updateBase>2014-07-16T14:25:18-07:00</syn:updateBase> <syn:updateFrequency>1</syn:updateFrequency> <syn:updatePeriod>hourly</syn:updatePeriod>

That is the configuration setting for RSS readers to not update more than once per hour.

See spec: http://web.resource.org/rss/1.0/modules/syndication/

Craigslist has aggressive blocking for excessive GET requests. The RSS feed contains only the first 250 characters of text description of the ad. Thus, you will see that something got added but you will not see its details. More importantly, attributes are not available as part of the RSS feed.

That means that you were interested in specific colors of a car, you would need to define a separate RSS feed for red, yellow, and so on.

It is hard to test whether the RSS search results are additionally throttled, but you will likely get blocked while testing. :)

While legitimately shopping, I got blocked multiple times for becoming more efficient.

RSS is not hard to read. Here are some red manual transmission cars in San Francisco Bay Area http://sfbay.craigslist.org/search/cto?auto_paint=7&auto_tra...

I used to give away good free stuff, but it's just not woth the hassle(going to the wrong people, people not showing up, the spam, over agressive Flaggers--you people need to get a life, etc.)

While I'm here. What's with the begging to lower an already low price on a service, or item. I used to price an item realistically, then knock off 40 percent. People still want to Haggle over the price. I now just double the price I think it's worth.

There's a huge need for another Craigslist. The site has gotten too Ghetto.

Yeah, I am going to work on it once I am done launching my main product. My main product is for in-person customer acquisition, so I should be able to seed this properly.

I am thinking:

* Schedule pickup (I made a scheduling product a few years ago with a nice algorithm)

* [Innovative revenue source that is the secret sauce, not involving ads or subscriptions]

* Request hold on the item (with ability to flag no shows)

* Hyper-detailed categories (I have a very interesting plan for that)

Open API, but anyone including us is free to clone what you build :)

all that would remove the one killer feature of CL... no logins.

what you describe is just a ebay clone with pickup instead of delivery.

So will IFTTT get a cease and desist sometime in the near future, or have they come to an agreement with Craigslist to provide their channel?

A while back, I hacked together notifications using RSS, but lately, I just use IFTTT...

https://ifttt.com/craigslist

I use IFTTT's craigslist channel all the time for long-term polling of craigslist. Not sure how instant it is, but notifications are pretty reliable.

Do they get CL's permission to run that? Because it looks like it violates the ToS as written.

One thing I like about this reflection is the highlighting of the weirdness of the connection between a service's creators and its users. I'm on both sides of the fence, and I find myself wrestling with lots of different feelings towards my users (mostly positive, but not all). I guess this dynamic has existed since the printing press, but it feels a bit more intimate now since you can see the emails and read the tweets.

This is a very good point, and something that was also touched on in http://www.saurik.com/id/20 recently.

I would think CL using a proxy that silently blocks client ips that interact with the service suspiciously may be cheaper than sending out lawyer letters and more immediately effective. Just prefer technical solutions to legal threatening I guess..

For fun and as a learning experience, I built a simple CL scrapping web site. I set it up on a small aws instance about 6 months ago and after playing around with and making modifications/updates to it over the course of a month or two, I largely forgot about it. After seeing this thread, it dawned on me that I had not received an e-mail update on any of my search alerts for a while. I ssh'ed into the server and got a 403 with 'wget craigslist.com', so my guess is that they are doing some sort of blocking. Looking at the logs, the block probably started about a month ago.

Modifying the user agent to mimic a normal browser might solve the problem.

I ran a system that was a back end for craigslist for many years, and I never stopped even after getting a cease and desist from them. What really got me was after they started charging people to post ads... that really killed my business model. Generally its not a great idea to build a business model around the availability of a third party... I think the whole concept of craigslist sending a 'cease and desist' over what is generally considered public_html is a laughable joke in my opinion, and good luck trying to restrict public html from crawlers and robots.

A little bit off topic but interesting nonetheless.

I've known Beau for many years now and his writing style has improved so much. He is a testament to the "practice makes perfect" (or better at least) mantra. It's really interesting to see the transition in his travel blog from when he started writing to now. If you're interested you can check out. Compare the first post to the most recent. http://dangertravels.com/

Thanks for sharing, his travel stories are inspiring

I'd be willing to be that Craigslist shut you down because they're building their own "watch" feature. The same thing happened with Padmapper - they sent them a cease and desist letter, and then came out with their own map feature (http://newyork.craigslist.org/aap/#map).

Does it mean that any browser extension which is modifying CL web page(s) violating their terms of use?

it's a great story.

BTW, can anyone give me perspective on what it would take to recreate this in Python? As a learning exercise

Is it possible to do? how long would it take for someone relatively new, etc.

With all the libraries available for parsing and scraping I suspect this could be recreated very quickly.

Did craigwatch.com agree to the terms on the site?

There's probably a provision in the TOS that says that you implicitly agree to the terms by virtue of using the service. I don't know for sure, just speculating.

mmm

Stand up to the MAN!!!

Everyone is too law abiding.. But really what laws is he breaking here?? Trademark? Copyright?

Spoke to my legal counsel and I don't see either..

Terms of service are a contractual agreement between a service and its users. There's no laws being broken, but it's breach of contract.

Your legal counsel would know that I suspect.

Sorry, I forgot that people still live in the US.

click through agreements and terms of service agreements (TOS) are not legally binding contracts. they are used by service providers to provide fear, uncertainty and doubt (FUD).

http://en.wikipedia.org/wiki/Trespass_to_chattels#Beyond_spa...

CFAA?

Reply on news.ycombinator.com