Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Back in 2004 I wanted to use Kleinberg's hub-and-authorities algorithm on Delicious, and I ran a crawler on it anyway, despite the robots.txt file. I got blocked, and when I complained, I got an email from the founder telling me to buzz off.

I've long seen the no-crawling policy of Delicious plus the Roach Motel API that was all about getting people to put their data in but not about letting people get it out as the dark side of "Web 2.0"; often we hear about an API as if it were a gift, but it's often a self-serving effort to take our data and give nothing back in return.

View on HN

The API let you get your own data out. Everyone else's data wasn't yours to have. And that sort of crawling tended to knock down the site.

Perhaps you're right, but in some sense that data belongs, collectively, to all of us.

In my mind, delicious represents everything that was great and everything awful about 'Web 2.0'. Yes, it did something that everybody assumed was impossible because bookmarking sites failed so consistently in 'Web 1.0'. Although delicious provided a useful current awareness service, it never did any of the interesting things with the data that would have made it possible to move onto 'Web 3.0'. And it probably never will -- but maybe that's fine because it opens up an opportunity for the rest of us.

I just want to make sure you know who you're replying to. Joshu is the guy that wrote/founded delicious.

Perhaps "in some sense" you feel that that data belongs to all of us, but it's not really your place to decide. Delicious had terms of use, and you violated those terms. Your argument sounds a bit like the "the internet is public domain" cookbook lady from last month. It was wrong then, and it's wrong now.

I know who he is. As I've said, I've dealt with him before.

He's free to put up whatever defenses he wants to put up. Other people are going to try to tear them down. That's the way of the world.

Probably a substantial fraction of the public delicious data was collected for this:

http://ilpubs.stanford.edu/858/1/2008-2.pdf

The primary issue is that it does not seem terribly clear what the legal status of redistributing such data would be, and/or whether this is changed in any way by Y! shutting down delicious.

Could you use of EC2, proxies and tor (horrid bandwidth of course) to get around some of the limiting?

You're missing the point

I've done that kind of stuff and, against an advanced opponent, you tend to lose. (Although you can roll the average webmaster)

Remember that IP addresses have a market price of about $3 /month, and that's what an honest proxy cost Honest proxy providers rent machines in data centers and have them bind to a wide range of addresses, all in the same netblock. If you're coming from 20 different addresses in a netblock (paying $60) a month, you still look suspicious. These guys might have machines in several data centers, but they can't put you into hundreds of different netblocks.

The economics might get better if you're sharing the proxies with other people, but those other people are up to the Devil's work, and are busting their asses 24-7 getting the IP addresses in everybody's block lists.

As for Tor, quite a few organizations block or limit Tor traffic... Databases of active Tor gateways are available, and sites like Wikipedia use them... Wikipedia won't let you make anonymous edits from Tor, because they don't like dealing with griefers who use Tor.

Now, some people will use hacked machines as proxy servers. A botnet can create a nearly indetectable cloud of IP addresses, but as far as I'm concerned, use of a botnet is an ethical line I won't cross.

Sound like some fun stuff to profile a site when you're doing a security screening of them. I'll have to keep this in mind.

Reply on news.ycombinator.com