Ask HN: Review my web app, CraigUpdate.com | Hacker News Reader

Ask HN: Review my web app, CraigUpdate.com

craigupdate.com

22 pointssil3ntmac38 comments

This site will likely either get down down for TOS violations - or its spidering IP will get blocked by craigslist.

If anyone on hackernews wants a (more powerful) ruby program/lib to do this exact thing (with multi-city amd regex search support!), I recommend you check out craigwatch/libcraigscrape : http://www.derosetechnologies.com/community/libcraigscrape

I agree - I've been using craigwatch for a while - once you configure your report, I keep a cronjob running once a day to update me with all the new stuff in my area that I might be interested in. The results are way better then craigslist's or these other online systems. Once its setup, it just works automatically - Plus, I dont have to worry about all the spam coming in from a forked-over email address

For me, my needs were a little complex, so I didnt want to invest my time in a bunch of searches on a site that might go away in a week. With craigwatch its just a simple gem install, and you know its not going anywhere since you can run it on your own box. Its well written too - I made a couple improvements on the lib for my purposes and it was a painless process

I've noted, over the years, many sites that scrape craigslist data for one purpose or another. They run for a few months, get popular, get noticed by craigslist, get shut down.

Do you have ongoing communications with them to avoid this fate?

From the TOU: http://www.craigslist.org/about/terms.of.use

u) use automated means, including spiders, robots, crawlers, data mining tools, or the like to download data from the Service - unless expressly permitted by craigslist;

If you are serious about this site and you think that it could become a largish site you should definitely get permission.

Agreed. You really need to read the CL TOS and make sure you're working with CL to use their data.

Could you name an example of this? I scraped CL for a while and they didn't say a word, and I've never heard of them shutting someone down.

If you visit http://crgslst.com/ -- a multi-city search site -- it pops up a message about how they were blocked.

They also disabled Jeff Atwood's multi-city search: http://www.codinghorror.com/craigslist/ (he posted the letter he got from them)

Then again, http://www.housingmaps.com/ is still functional, and might be the most popular Craigslist mashup -- I'm assuming they have a deal in place with CL.

how popular is your site? I remember they shut down some real estate site that mashed up craiglist data with google maps that got shut down pretty quickly. Not sure what the name was.

I think the site is OK, but I get this same general functionality from doing a search on craigslist and then adding that search as an RSS feed. While I don't get INSTANT updates, I'm in my RSS reader often enough during the day that I usually see things quickly enough. Still, not a bad idea, and if you're really looking to be the first person to see a new post, I can see how this would come in really handy.

I used to do that, but I'm not always at my computer. This way I can have results emailed to my iPhone and instantly be notified. SMS isn't a bad idea either now that I think about it...

Hi I've done the same thing, http://www.craigdiddy.com and I also have a list of all the other sites that do this and why mine is the best. Besides email you can currently get free sms to find out even faster (via Twitter) and I plan on releasing the non-interface code on github.

Thank you for not using JavaScript unnecessarily.

In fact, thank you for not requiring me to create an account or register or asking if I wanted to receive valuable offers via e-mail.

I'll comment back if it finds anything, but as far as the paint on the outside goes, it looks nice and is something I would definitely use.

[edit] - technically you did create an account and I did have to click a link to activate. I still stand by the thank you (and that's just smart going anyway).

So I said I'd follow up and I will.

I like the e-mail note. It only found one item but it said "A few items have been founding matching <my criteria>", which I take to mean that it groups its findings together in one e-mail, rather than sending an e-mail every time something is found.

Just my own pet peeve and this is disgustingly nit-picky and petty, but: "1 File(s) copied" has always bothered me. A plural check would be an improvement. Of course, had there been a plural check, I wouldn't have been able to comment yet.

Not bad at all. I had the same idea and was about to implement it, but you beat me to it. The only suggestion (small one) that I have, is that once you register for the first time, and it suggests you register again for another keyword, the city drop-down menu changes back to default. Generally speaking (although not always the case), the requester would probably choose the same city for every keyword search.

Thanks for knocking something off my to-do list. However, what I really wanted (and I wrote an ugly version for myself) is an update for apartments. We were looking for 1-2 beds that allowed dogs/cats within a certain prices range, not in certain neighborhoods. Add that feature and you have an awesome product.

It's cool and useful--the thing is there are a ton of cool/useful apps that COULD be built on top of craigslist, but craigslist does everything in its power to prevent that from happening. Is there a reason you won't get shut down?

- allow a '+' in the email address

- make the data sticky, if you give me an error when I enter a '+' in the email address, I want the stuff I previously entered to be there

But it looks nice, I'm hoping to get a free bike

I've been a fan of http://www.craigslistwatch.com/ for a long time, and they'd never gotten banned but unfortunately currently undergoing some sort of restructuring.

No about, no contact, not even a privacy statement where you assure not to do bad things with all the mail address you collect.

The privacy statement is a good point, I will add one in later for sure.

Cool.

Might get spammy.

I don't see a revenue model anywhere. Nor a marketing model. Pull out the checkbook for adsense?

To be honest, it was just a little web app I wrote last night. No marketing or business model, although I can see how one could make money off of ads in the emails?

I'm not really looking to develop this into a business, just thought it would be useful and some people would get some use out of it. :)

A craig's site I used before, has changed their name, one of the reasons for change was:

   This is the big one. Craigslist doesn't like related 
   sites that use their 'Craig' name. And that's only fair. 
   As always, I try to do everything possible not to 
   irritate the big guy. The site hasn't drawn his wrath 
   yet, and this change is an important step to keep it  
   that way.

Great! I'd like it if it remembered my city so I can add a few search terms in a row.

Good point! I'll add that feature tonight. :)

won't this spam the crap out of people?

or are you doing the smart thing and just sending people a single email "We've found a new match" with a link to the craigslist search results, sorted for their search preferences?

It's hard to remember that everyone doesn't have 7+ Gb of email storage. If I wanted to use a service like this, I can't imagine an email with the fulltext of a listing being big enough to ever bother me.

Giving your users the option to only receive links can't hurt, though.

one email is not a problem, its getting 40 emails a day

The emails sent out are short (around a paragraph), and every email that is sent to you has an 'unregister' link at the bottom that you can click and you will stop receiving emails regarding that search result.

How about adding results to a special anonymized (not password protected, because they can't really handle it yet) RSS URL?

My thoughts exactly. How quickly will you end up on a spam blocker list?

i couldn't find San Francisco or Bay Area or SFBay in the list!

Whoops, my bad. Fixed :)

this is great, very simple and straightforward!

I was just speaking with someone this morning about setting up CG updates to look for freelance gigs. Great start. Web workers will need to track several, if not all, cities though.

You can sign up for as many cities as you want, I couldn't think of an easy way to combine multiple cities/search terms without creating an account, which defeats the simple/quick signup aspect I was aiming for.

separate search terms with a comma, just as in email. select STATE (for all cities). select COUNTRY.

As of now I must go to Google and run a term+date site:cl.org to get results for all states. That works, but it does not send me updates.

Reply on news.ycombinator.com