Craigslist blocks Yahoo Pipes
romy.posterous.com
romy.posterous.com
As others have pointed out, Craig's whole m.o. appears to be preventing people from building for-profit businesses on top of his site. Craig is anti-business. That's his prerogative. You obviously disagree with his philosophy, but a difference of opinion is not cause to go around calling somebody evil. He's done more good through his site and the $$ he generates off his site (which gets plowed into his foundation) than you, me and the next 100 people who read this combined.
He's done more good through his site and the $$
he generates off his site ... than you, me and the next 100
people who read this combined.
Fortunately for you, I contribute little so the next 99 have to play catch up.
Check out the tax returns for the foundation: http://craigslistfoundation.org/about.html
It's safe to assume the site makes a whole lot more than $650k/year, so not a whole lot of it goes to the foundation. Where do you think the rest of the money is going?
"Possibly the only CEO ever described as anti-establishment, a communist, and a socialistic anarchist..."
In the email exchange I've had with Craig, not all of which I've posted, I've asked him to clarify on this point:
"Oh, and about being noncommercial. A bunch of the RSS mashup sites I've seen, like Craiglook, throw up advertising presumably to cover server costs or make a few bucks. If I didn't want to go that route, would asking for donations be acceptable ?"
In his response, he said he can ask. I said please do, and that was the end of it. So for CL to break things when there is absolutely no sign of commercial purpose is a little unnecessary in my eyes.
Just don't post any advertising from your site. I think that worked fine for Craiglook.
Futhermore, Yahoo pipes seem to be working fine with craigslist. Are you sure it's not your site being blocked ?
I had hoped that the culture might become more developer friendly after Jeremy Zawodny (formerly at Yahoo) joined up. Zawodny even wrote an article about how all this terrific data was available via RSS, but there was no good way to piece it together: http://jeremy.zawodny.com/blog/archives/008513.html
Craig is a programmer and he comes across as a normal, humble sort of guy. We like him and we like his service. I don't understand his obsession with keeping craigslist a) locked in the dark ages for functionality and b) not letting anyone build something that adds useful features. It really starts to make you think. Is he just strange or (if you don't give him the benefit of the doubt like I do) evil?
I like to think that Craigslist isn't free, I just pay for it with my time and attention by flagging the junk that doesn't belong there.
Craig very famously types the command "sweep the leg" (a Karate Kid reference) to remove spam. Maybe it's the same command for small projects based on the service. :-)
You mean the mentality that has created one of the top five websites ever? I mean EVER? ... Oh, that poor Craig, silly idealistic fellow...
If craigslist were solely out to monetize this website, that'd be fine. But they don't. They keep some things free and Craig has made a big deal about making money in the categories that needed a charge to prevent spamming. So there's a disconnect between a guy who wants to fight the "man" and someone who clearly just went out of his way to squash a bunch of little ones.
The correct moves in this situation for a top five website are:
- build the feature yourself. It sucks for the little guy who made his website but your users get the benefits anyway.
- don't build the feature because this feature is trivial and your users don't want it anyway. So this external website languishes and provides no benefit and no harm.
- buy the feature. Buy the company for a little money and implement the feature. The top 5 websites do this all the time. As an added bonus, you can acquire this talent who can create lots of great features for your users.
- hire the guys who made the feature and ask them to build it for you. They'd probably leap at the chance.
- Give the external website your blessing. Invest a little money in them so that if they strike it big you'll benefit.
- Make them pay a data access fee. Charge them per query.
Any of these would have been better options. This would be how a top five company should behave.
1. Craig was having a discussion with Romy that was clearly misleading. The fact that there's been no further communication and pipes have been turned off demonstrates that.
2. All pipes were shut off. Even if you thought that Romy's pipe was commercial, some might not have been. Drowning everything might be easy, but it doesn't seem good.
3. He's on the board of the Wikimedia Foundation and the Sunlight Foundation. These are both great organizations that advocate for openness, transparency, and the sharing of data. These groups would despise an action like this.
Craig has a right to run his business the way he wants to. However, there's a odd level of hypocrisy here and it merits being pointed out.
37signals, another company that seems to have a decent handle on how to get users, also believes that not providing features is part of why their apps work for so many people.
I would guess they are operating under the principle of "If it ain't broke, don't fix it"
I agree with you that money is not the primary motivation, but I think the choice is more complex than simply opting in or out of potential revenue.
There are of course other possible explanations: 4) Craigslist RSS server runs on a Pentium w/ 256MB of memory and can't scale 5) Someone at Yahoo stole Craig's girlfriend so he hates them.
Any others?
I'm also pretty sure they only charge when the community begs for it (apartment listings in major cities, as a barrier to keep out spam) and are legally obligated (escort services, http://www.seattlepi.com/business/386655_craigslist07.html)
If you don't like that, then go build your own site that has the same level of success and define your own rules. And deal with all the people who tell you you are doing it wrong, that's a lot of fun.
For the record, I have no connection to craigslist other than admiring it. I'm flaming because I've got my own company and I deal with this crap as well and it's annoying. If you don't like the way someone else is doing it then either go do it better or shut the fuck up.
Its easy to come out and say "Oh, he can do anything he wants because its his company, and if you don't like it just leave". It takes a hell of a lot more effort to actually put the finger on why a move like this (blocking an entire site which hosts various mashups, over 2000 targeting just Craigslist, without notifying the users or having any real method of notifying the "developers" why it was done, or what they can do about it) and trying to point out to the people who actually could do something about it.
As a tip, if you're getting that much flak from your users that you have to start flaming users on the internet about it, you're probably doing something wrong.
And your claim that you or the other folks are a user is based on what? Craig takes care of the _actual_ users, the people who buy/sell stuff there, use it for job listings/postings, etc.
Your point about listening to your users would be valid if you actually were one. Or better yet represented a whole class of users.
The whining here comes from a set of people who are leaches. They are people who want to leverage craigslist.org. Whether it is for profit or for fun or for fame, it's leveraging the work of someone else. In no way are those people "users" of craigslist.
Everything you wrote could have been used as a defense of Windows 98, or IE 6. You may be defending a shitty user experience with a monopolist position, but we love you Craig because at least you hate capitalism.
But rather than asserting that it is a fine product (which it is, though I agree it could be made better), how about spelling out what it is that you think would make it better? I wouldn't be surprised if Jim and/or Craig reads this thread at some point.
This isn't simplicity. This isn't minimalism. This is Ludditism, backed by a monopoly position. How many millions of hours are wasted every day because of Newmark's anti-market fetish?
And Jim and Craig have made it abundantly clear in interviews that they don't care about user feedback, so I don't expect anything to change.
Well ... that's how we have Firefox now isn't it ? I mean people went ahead and built something better.
"Let's get rid of IE6" is a great way to rally people. "Let's get rid of that bastard Craig Newmark," not so much.
Craiglist is different, much less corporate in the traditional sense. From the very beginning they were trying to provide a service that is largely free (and hugely useful, I've bought and sold stuff there and never paid a tax like Ebay extracts). It's a pretty sweet setup, end users get a valuable service for free, craigslist gets money from entities that can afford to pay, why screw it up?
I tend to agree with some of the people here who want to make it better but if you really want to do that then go talk to them about a joint project or go work there. Personally, I'd love a way to search all of the craigslist sites at once (I shopped for months to get a 2006 Sprinter, wanted the smaller motor that gets better mpg, finally found one in portland. It was painful doing all those searches). So I agree that there are things that could be made better for the users of the site. I think the way you get those things done is to work with them from the beginning.
If your app depends on Yahoo! Pipes AND Craigslist, well ... that's probably extra dangerous.
I'm old, 47, been in the valley for 20+ years. Seen a lot of shit, seen a lot of people get screwed over in stock options, seen the good and seen the bad.
What I'm trying to do is establish credibility (prolly didn't do a good job). Whatever. You can listen or not.
Craig is good people. Really really good people. He cares, he does the right thing as much as he can do it. Craigslist could be at least 2 orders of magnitude more rich if they wanted to fuck you over. They haven't done that, they repeatedly try and take care of their users.
If you are unhappy with craig I pity you. Real life is going to fuck you.
It's hard to establish your credentials (account created 7 hours ago, just like your post).
Normally I would have looked through your previous comments/blog and tried to establish veracity of your claims. Freshness of account means thats not possible.
But in my mind there's two options
1) You're claims are not legit, you're a stooge
Shame on you.
2) You are legit, and you are mentioning your personal opinion
Thank you! It's nice to hear someone vouching for someone.
The HN startup community do it all the time. They just happen to be 20 somethings looking (rightly) to fit into the YC community.
I don't think you are appealing to authority as your comment will be judged on its own merits. Fuck those argument technicality Nazis anyway... you're not claiming to be a doctor, or the head of some company. Nor are you claiming to have special information that other people don't have... I'm sure lots of people have met this Craig (not me I'm in London) and could deny your view hiding behind screen_names.
I'll wait to see whether you continue to post to see if its 1) or 2). In case you are 2) please don't be discouraged by ignorant haters.
good point on the recent account creation, does make me look like a stooge.
It was/is my personal opinion and I'll try and stick around and establish a track record. I mostly don't do that sort of thing any more (used to be very active) because I got tired of all the armchair quarterbacks telling me I was doing it wrong. There are millions of them and only one of me :)
One slight quibble, I am the head of a (small) bay area software company. I'd kinda like to not be, I'm tired, but for now it is what it is. To all you folks who want to run your own thing my only advice is have fun along the way, spend time with your family, friends, whatever you like, but have fun. The time passes faster than you can imagine.
Anyway, thanks for the pleasant reply, I'll try and emulate your civil tone.
So, while I am sorry that a bunch of cool hacks got shut down, it might be a pretty reasonable decision on CL's part to cut off Yahoo Pipes. Plus, it's his site to do with it what he wants...
(There is some great interview with him in Wired that got linked here not too long ago.)
But the sentence is moot since you already stated the facts previously.
they're not morally obligated to provide unfettered access to their data to anyone who wants it- i fail to see how it is "broken"
Want a map view so you can see all the relevant listings in one view?
Want craigslist to deliberately filter the same text or picture used for different listed apartments or jobs so you don't waste your time or get scammed?
Want to search by squarefoot or by actual bedrooms or by picture?
Apple would say "There's an app for that." Craigslist says "screw you".
There's no way to (legitimately) outwardly index, categorize, or aggregate craigslist content.
(Related gripe: apartment searching in google is useless too: http://www.google.com/search?q=mountain+view+apartments It gets slightly better with searching on maps, but not by much.)
Craigslist gains value from it's users not they other way around.
As for determining motive -- profit or otherwise -- I suspect it's more about keeping others' greedy mitts off their community than cold cash. CL has been about the community of people using it since it was just a mailing list, and publicly Craig and Craigslist have been pretty clear about what they value. So, they probably see pipes as a threat to that community.
...nor will it be the last I suspect.
[Edit:] Thanks!
Now, I know I might be able to get some where with them if I tried emailing them, and asked for the data I wanted. Then again, it sounds like I probably wouldn't. Either way, while the data was surprising well suited for what I wanted to do, it isn't really available, at least not in any way I could actually use it.
It makes me kind of sad every time I see someone do something like this. You've created something super cool, something with tons and tons of data. For what I was using it for (localized text samples), I can't think of a better, more complete place to find data from. But when you see something cool, something you never thought of/never thought was important enough to implement, and go out of your way to squash it, it makes the part of me that loves data feel bad about it. Craiglist has, at any given time, literally gigs of written text with all sorts of metadata on it. In the short bit I processed, I took all of the data from just a handful of boards, and ended up with over 150mb of data. Considering that most of that was <2 weeks old, They likely process up to a terabyte of data every year, with fairly good metadata on it. Seeing all of that only being used at face value seems like such a waste.
(As a side note, anyone with a bunch of written text samples, with information as to where each of them was written, I would really like to talk to you. Actually, that goes for anyone with any sizeable amount of data. Just because you don't see a use for it doesn't meant that there isn't someone out there who would absolutely love to get their hands on it.)
What's predictable here ?
I'd ask Craig again on Twitter about the Pipes thing, he's better at responding to tweets than emails sometimes. http://twitter.com/craignewmark
Craigslist have no obligation to let you use their data however you like.
Maybe your correspondence coincided with a more general security tune-up they decided to roll out on Thanksgiving?
They'd never block Google Reader -- they'd sooner just disable RSS entirely.
That is until they turn that off too.
My point is that their closeness will be major factor in their failure.
EDIT: I don't buy Craigslist hype. To me, their service is poor. Outside the US, especially in the East Europe, there's a lot of better alternatives. There was a lot of discussions why they succeed. I point out how they're likely to fail.
So they've gotten huge (not in number of employees), but they're behaving just as arrogantly.