Hey Yahoo, Can you please open source Delicious? | Hacker News Reader

Hey Yahoo, Can you please open source Delicious?

web.archive.org

413 pointscodybrown84 comments

joshu on Twitter: http://twitter.com/joshu/status/15492062459731968

"@cdixon open sourcing it would be an enormous pain in the ass. selling it not as bad but yahoo infrastructure and auth problematic."

Besides that, open sourcing delicious wouldn't solve the problem - someone still has to host it, and maintain it, and import several million people's bookmark collections in to it. That takes significant time and money. Having the code is just a small part of it.

A smarter thing to do would be to campaign for the (public) data to be released as a huge data dump, ready for people to run their own analysis on (an Amazon Public Data Set for example). This too has plenty of problems though - under what terms should that data be licensed? People's bookmarks belong to them - would they be happy with their contributions being released as part of a massive data set for anyone (including sploggers) to do anything they liked with it?

I've taken a quarter of a million of the most popular bookmarks from thousands of tags, and created a similarity search engine with it. Seems to be working pretty well. Thank you, delicious.

is it available for public?

Jaiku seems to be running strong on App Engine so it is possible to open source a large application and have it keep jugging on. I don't see Yahoo being interested in porting to GAE though.

Jaiku also underwent a rewrite before being open sourced, and I'm sure their install was given something akin to reserved instances of App Engine. So, yeah, while that's doable, it helps if the company behind it is Google.

You have an opportunity to pick a license when you set up a Delicious account. Mine is set to CC-Atttribution-NonCommerical-ShareAlike. I'm not sure what the default is.

Why bother with the code? The value is with the existing links database, and since it belongs to the users, Yahoo shouldn't have the right to sell or distribute it freely.

Since users can already import/export their bookmarks, the only support Yahoo needs to provide is keeping the import API open for a little longer after the site is shut down. Of course this assumes that users would want to migrate their data to a replacement service.

Of course this assumes that users would want to migrate their data to a replacement service.

Or, for that matter, that users would share their tags with researchers who build on social tagging data:

http://groups.google.com/group/get-theinfo/browse_thread/thr...

Here is a (scraped?) delicious data set, for those who are interested: http://arvindn.livejournal.com/116137.html

Why would one want 10-days of bookmarks from 1.5 years ago?

Because they wanted to research how people's bookmarking activity corresponded to new information emerging concerning the 2009 flu pandemic http://en.wikipedia.org/wiki/2009_flu_pandemic

Unless I am mistaken here (as far as copyright goes, if that's what you're talking about) the only data Delicious' users can claim any right to is the short descriptions they are able to add to each bookmark. The mere fact that they liked that page is data that can be freely distributed.

Nah, the tags themselves is what he was getting at. Not saying I agree or disagree with him, but that's the value.

I actually forgot that Delicious was all about tags. It's been a long time since I visited the site.

The individual links can't be copyrighted, but a person's collection of links probably can.

http://en.wikipedia.org/wiki/United_States_copyright_law#Com...

Google has open sourced: Jaiku, Etherpad, Wave among other things, before closing them down.

Open sourcing it wouldn't solve the discovery/hosting part, but I still think they should do it.

I'm sure it wouldn't be "easy," but it wouldn't be unprecedented.

Yahoo! folks also should take a look at dataliberation.org, for best practices on getting on data out (also a Google joint).

(Nope, I don't work for Google :D, just think they've done some good things in this space. )

Data liberation is different from open-sourcing. Delicious allows you to dump every bit of metadata about your bookmarks either through their Data Export menu or via their API. Yahoo! did fine with Delicious in terms of data liberation. I imported all my bookmarks into pinboard.in without any trouble at all.

Definitely agree that data liberation is different that open source, I meant to express them as two separate ideas. Maybe that didn't come across.

A bit of data I'd like to get out of delicious that i can't right now, is social stuff/ relationship stuff. It really helped w/discovery.

Though you're totally right: link stuff is thorough.

s/good/responsible/

Sadly corporate responsibility is not the norm, though it should be.

The actual code doesn't matter.

It's the tens of thousands of man hours that have been spent creating one of the best indexes with careful tags for millions of pages. The data's what makes Delicious matter.

And, if you open-source the data, I see people crying foul over privacy.

Not if the links are public, which I suspect most people's are.

I'm pessimistic: it takes vastly more work to keep it alive, via either OSS or selling, than to kill it. Nobody who Yahoo cares about will notice any news related to Delicious, for fair or foul.

Since Yahoo! probably wants to avoid the embarrassment of seeing Delicious flourish under different management, they're unlikely to change course here.

But having >1,000,000 people with freshly tagged and exported links in a standard format seems to provide an opportunity to those who think they can do better.

One request for whoever that is: can you add the 'sort my links by popularity' feature that Yahoo! never developed?

This is not a copy of delicious, but I wrote this yesterday in a hurry. http://selficious.appspot.com - It imported my bookmarks and I use it to manage them and my future ones (add, edit, delete). The code will of course be opensource. I just need to clean it a lil' bit :)

All right, the code is online. I'm not really proud of it, but it seems to work just fine. There are tons of features to add/fix and I hope that you'll give a hand: https://github.com/initpy/selficious

For ppl who haven't exported their bookmarks yet, I wrote a quick webapp to help (for the command line skittish)

http://mattcrampton.com/delicious

For non-command-line-skittish people (on Unix), you can say something like

    wget --quiet --http-user=`head -n 1 ~/.deliciousrc` --http-passwd=`tail -n 1 ~/.deliciousrc` -O ~/docs/delicious.xml http://del.icio.us/api/posts/all

where ~/.deliciousrc is a text file with two lines:

   yourusername
   yourpassword

(or something similar with curl)

I run this out of cron nightly and back up my bookmarks in a git repo, along with my other dotfiles. I really appreciate services like delicious, but I don't trust them.

I'm going to miss delicious's data set, though. Searching a really big link collection with tags was great - I found a really good Ethiopian cookbook, some good+obscure research papers, and lots of other stuff that way.

Using curl:

  curl -n -o ~/docs/delicious.xml https://api.del.icio.us/v1/posts/all

where `-n` flag forces curl to read ~/.netrc file:

  machine api.del.icio.us login yourusername password yourpassword

ok, so did anyone manage to get either the curl or the wget method working?

`curl https://user:passwd@api.del.icio.us/v1/posts/recent` works just fine.

`curl https://user:passwd@api.del.icio.us/v1/posts/all` does not.

Maybe a server-side problem?

That doesn't work for newer accounts linked to a Yahoo account. In fact it's a real pain getting your data out in xml format from such an account.

There is an open source clone of Delicious, Scuttle: http://sourceforge.net/projects/scuttle/

I feel sorry for all the people let go just before xmas.

Weren't most of them moved to other jobs at Yahoo?

not the (reported) 4% of all Yahoo! employees that lost their jobs.

I doubt we'll see it open sourced, or saved in any form by Yahoo. It's too much cost/effort for them, when that's the reason they're scrapping it in the first place.

What i'd like to see is people quickly rolling out tools that help us get our Delicious bookmarks into other sites easier. Eg into Google Bookmarks. That currently requires you to install the Google Toolbar and import them through that (ugh).

You want a tool to do this job so you can avoid installing google toolbar? How much do you reckon thats worth to you?

I'd say a lot, because to do the import you have to swamp your firefox bookmarks structure with your delicious one, and then import those using the toolbar. Of course this looses all of the tags, and comments, which are pretty much the whole point of delicious.

A system that actually gets all the data from delicious into google bookmarks seems to be worth while.

I hacked together something a long time ago to do this with a bookmarklet. It'll import delicious bookmarks into Google while preserving tags.

http://blog.lifeslip.com/articles/2006/10/25/bookmarklet-del...

That worked perfectly! I am hardly able to express how grateful I am :-)

It even worked more smoothly than the Google Toolbar's own importer, which complains about dupes (wrongly), and prefixes all the labels with "Tag:" for no good reason.

Will blog this ASAP.

We are also compiling stories about present use cases of Delicious. If you are a user, post them here and I'll update the question.

Delicious was one of the best resources for finding articles on niche interests, for surfacing articles that were especially good, and for finding others interested in the same topics as you.

As for two specific examples: Learning Ruby on Rails has been a long, slow, struggle for me. A number of reasons for that, but one of the main ones being that I don't have many other programmers in my area that I can talk about it with. Although there are scores of blogs and tutorials on Rails, it's hard to know which articles are good, and nobody has the time to read them all. So if I was looking for a post on, say, integrating jQuery with Rails, I could do a quick search at http://www.delicious.com/tag/rails+jquery, and I'd find a bunch of articles that had been selected by individuals, for their own use. It was great for finding high-quality content.

Another example deals with niche interests. Delicious made it easy to suss out who is talking about $randominterest. Just by going to http://delicious.com/tag/randominterest, you can see who else is bookmarking it, who's writing about it, and so on. Especially when dealing with, say, fringe programming languages, or uncommon design details, Delicious makes it easy to find people.

Delicious was one of the best resources for finding articles on niche interests, for surfacing articles that were especially good, and for finding others interested in the same topics as you.

I wonder if this could be implemented as an API of websites? (Or perhaps on behalf of websites?) Instead of one single site, you'd have a p2p network of sites using protocols similar to those used for digital cash schemes. Think: a distributed open marketplace of middlemen. A sort of Diaspora middle tier.

Just a side note, but if you're still struggling with Rails, both Zails for Zombies[1] and RailsTutorial[2] are really good. And integrating jquery with Rails is now really easy, since they've moved to unobtrusive Javascrtipt. [3]

1: http://railsforzombies.org/

2: http://railstutorial.org/

3: https://github.com/rails/jquery-ujs

I never really cared about the social features. For me Delicious (and now Pinboard) is like a personal knowledge base, supporting my unreliable memory. For example when I want to check my favourite Vim plugins I search for tags vim+plugins+favs.

I have more than 2k entries there collected over the course of the last five years. I only bookmark sites that are for some reason important to me, and have some lasting value (eg. I don't bookmark most HN submissions). I tag everything I save and some of my tags have special semantics that is probably only useful to me.

Ditto for me but not just over my own links. I don't bookmark everything (though I have 6000 bookmarks there) but if I read an interesting article about, say, bioinformatics and it was something to do with Japan, I could hit up Delicious for articles+bioinformatics+japan and 9 times out of 10, be reconnected to content that Google would struggle to find given the same terms.

Updated the question. Thanks.

I've been using it to find interesting links.

The difference between Delicious and news aggregators like Reddit / Digg / HN is that the rating of links is a side-effect, not a conscious action.

As a result ranking on Delicious does not prefer yellow journalism, obscure interests or vanity displays; much like Google's ranking but small enough to not be a target for promotion through SEO.

When I bookmark a niche or obscure page, I can see the handful of other users that have also saved it. Because I can click through to their bookmarks, I often find a wealth of highly pertinent and (otherwise) hard to find links on the topic in question.

In other words, the whole social aspect.

I haven't done much with it lately, but I still maintain a "daily" tag, containing things I want to read every day. I then subscribe to the RSS feed in Firefox, and which gives me a bookmark I can middle-click on to get my daily reading.

The main problem was that the RSS feed was limited to 15 entries.

I've been using Delicious for storage and management of news links on http://checkuitcheckin.nl/ (a view on public opinion on the Dutch public transport card OV-chipkaart).

I will miss Delicious, I would post a link then use Mechanical Turk to have about 80 to 200 other people also post the same link, then the link would be duplicated on other sites that show "popular" links, social aggregator sites would create a "conversation" about the link, etc. Pretty soon you have 100 sites that link back to yours. Now that is all going away.

I want Open Source Federated Delicious. Anyone wanna help make it?

Done and done: http://blog.fupps.com/2010/05/25/truly-scrumptious-bookmarks...

That looks pretty good, but it appears to have no concept of users. In which case when you start replicating with somebody you end up with the same editable list of bookmarks. That may be what you want, but I'd quite like to be able to separate mine and theirs.

I was actually thinking the same thing. Pity I don't have enough time. Though if someone makes a project I'd have to look into helping a little.

A possibility would be to build this on top of Noserub: http://noserub.com/

There are a few decentralized social networks around. I was considering the use of status.net as a base since it's got a load of the right features in there already (tagging, groups, federation, etc); though it may be best to just start from scratch using new and exciting technologies.

Can anyone recommend a replacement for del.icio.us, ideally one that can import my existing data?

I'm a big fan of http://pinboard.in

Check out http://favilous.com/

Yep, open sourcing wouldn't help anyone. By the way, I think it's written in PHP and Symfony framework.

This service just needs to stay alive.

Couldn't we do a service where delicious users can export all their bookmarks (bar private ones if they want) and then send them onto a aggregator so a new service can pick up where delicious left. I am assuming here that technically delicious is not (very) challenging, its the data that is precious.

Delicious already provides an export mechanism. It is up to other services to read that format.

http://www.delicious.com/settings/bookmarks/export

I know. What I meant was a service where you put your delicious username and it automatically fetches all your data and merges them into one big dataset. Something to keep the data aggregated and then re-launch.

Why not try the Xmarks method? Seeing as a fair number of people are paying for pinboard as a replacement, Yahoo could ask people to commit to paying $10 (outright or per year) to keep their delicious account.

Is there anything particularly special about the code which would make it interesting? The value seems to be in the concept, and then the content produced by the large user base.

There are many alternate services of the same type. Just export your bookmarks into the new service. I don't see what the fuss is about.

Because it's not just about my bookmarks -- it's about using the wisdom of Delicious' huge crowd to find quality links on any given topic.

Care to provide a few examples?

Diigo.

Notify the editors of the Oxford English Dictionary. The word 'many' has been redefined to mean 'one'.

'Few' can mean one or more so logically you got nothing. Although if you look up 'troll' you will find 'mfukar'.

You wrote 'many'. Provided 'one'. Now go away.

why not hand the data over to DMOZ to take care of

The most valueable assets are its data and brand. I would be suprised if there weren't already similar open source services.

There should a legislation. If company stops support or goes bust it should open source the software.

Open sourcing proprietary software isn't necessarily easy. I was at Netscape when we sanitized the codebase to release as Mozilla, and there was plenty of third-party code that had to be left out. (That's part of why Mozilla was so slow to get off the ground. That, and the fact that compiling Navigator was always a pain in the donkey, even when you had the whole thing.)

That doesn't really seem practical. What if it uses proprietary technology that the company still wants to retain?

Pfft. The code is pretty worthless without Yahoo's infrastructure.

More interestingly, I wonder how much Yahoo would sell it for?

yahoo doesn't actually have much infrastructure that delicious relies on. the main bit is the text indexer.

That makes the question of how much they'd sell it for even more interesting.

Hey Yahoo, or anyone else, please build an open source app so it loses all monetary value.

...but only the earlier perl code base :)

Donate it to the Mozilla Foundation.

very good suggestion ...

Reply on news.ycombinator.com