Distributed Social App with CouchDB. Brilliant | Hacker News Reader

Distributed Social App with CouchDB. Brilliant

victusspiritus.com

70 pointsmessel33 comments

There's a bit more info in the projects's README at http://github.com/maxogden/couchappspora

Unlike the link submitted it suggests it's intended to be an alternative implementation of Diaspora's protocols, which for my money makes it a bit more interesting than yet another distributed social network.

My bad, yeah I caught that he cloned the API and updated the post this AM.

couchappspora is shooting for ostatus/SWAT0 compatibility. I just started a separate project to work towards implementing ostatus in commonjs http://github.com/maxogden/ostatus-js

so what do you need: PuSH, Salmon, anything else? update learned to read :D OStatus consists of the following protocols: WebFinger PortableContacts Activity Streams PubSubHubBub Atom

There are little technical details. As far as i understand the thesis is: CouchDB provides all the groundwork for distributed state, so there is no need to build a lot sync/auth/notify code like Diaspora does. Really?

Will be interesting to see how well it works with 1 million nodes.

Isn't that the point with these projects? Managing a central service for 500 million users is hard. But with a decentralized setup, I won't need to connect to a million nodes. I don't know quite that many people.

Course, a friend request changes from a simple button click to getting my local couchdb instance to do a filtered sync with your couchdb instance.

You might have a point there. Although some people on Twitter have 100000 Followers and more.

I think if you have 1million active followers some kind of push publishing setup could help. But as far as I know there's nothing in the tech of couchdb that prevents replicating to a million other nodes.

One obstacle is that you cannot have one million open sockets (for long polling) at the same time on one node. I don't know how easy this is to mitigate.

Hmm. The changes are posted through comet in continuous replication? That would mean another distribution system would be better for heavily published nodes.

You can have a million open sockets in Erlang, although it requires serious RAM: http://www.metabrew.com/article/a-million-user-comet-applica...

I don't think it's that capable of doing all that. As far as I understand, CouchDB does a "dumb" version-control type synchronization. Server A can copy all the data that Server B has and do it in a nice clean manner over HTTP.

I can think of several disadvantages (though I may be wrong and naive):

1. If you're running a node, and I'm running a node, my data schematics might not be compatible with yours. If I write a uniform backward-compatible RPC, then it'll be a little better.

2. I can't selectively allow you to copy some data (photos) and not others. If you are a user on my database, you can access everything you have permissions to. It's hard to set fine-grained permissions.

3. Push notifications will be hard. I'd have to poll the database for changes.

For that matter, the same can be achieved by putting a giant XML file on Git, and having each person set up a remote, or using a complicated set of shell scripts and rsync. Couch's replication facility is really neat, especially if you want to scale horizontally or if you're adding redundancy, etc.

CouchDB is becoming the new hammer. It'll get tempting to write source control, file backup, etc, using this database because of the data sync, and decent query capabilities. But I feel that building a social network on top of CouchDB's replication might not be the best of ideas.

couchdb has filtered replication so you can selectively choose what gets replicated

and it has continous replication with a comet changes feed, so push notifications are also built in.

the data schematics are a bit of a non issue, at some point someone needs to agree with what the data format is.

basically couchdb already has a lot of this functionality baked in as first class citizens, I agree that there is a lot of situations where people will be hammering couch into a hole that doesnt fit (source control would be a good one), but I certainly dont think this is one of them.

Yeah I wanted to build a stand alone media sharing app (big data files, images, movies, etc), but now I may not need one. Continuous replication of parts of subscribed databases was the idea I was going to go with.

The data schematics will be implemented via the OStatus protocols. CouchDB's role is to comply with OStatus and push and pull data when necessary.

As Dale stated, this is a good fit for Couch's replication mechanisms. Protocols like PubSubHubbub are ripe for porting over to Couch.

CouchDB sync is more like rsync than DVCS sync. The idea is that it is very simple, yet powerful enough to build more sophisticated systems on top of. Building small and large P2P systems on top of CouchDB is very much a design goal. The fact that sync can be used to build scalable HA systems is orthogonal.

To your points:

1.) I think the idea of this project is to define a common ground to allow seamless sync.

2.) CouchDB supports filtered replication, so you can define what can be synced and what not.

3.) CouchDB has a push-notification changes feed that does not involve polling.

Is it actually the case that filtered replication can be used cleanly for access control now? My understanding from the various docs was that currently you can write a filter but not require that users replicate through it -- that is, if you have permission to replicate, you have permission to replicate and that's it. For real access control using this I think you'd need something like a permission to replicate distinct from permission to read and a permission to replicate only through a specified filter. I suppose I'm off to have a look through Max's source, but if I'm wrong about these things I'd appreciate the correction...

...ok, back. Right now there appears to be no access control at all. Granting that this is a relatively early-stage project so it doesn't necessarily matter too much, but my question still stands.

Your personal data db could be private and you could use filtered replication to create a local clone with only the data you want to allow replicating.

Do you know if this thing will replicate all data of every person in the network? Or does it just connect to your friends' nodes and pull only their own changes?

The particular implementation will pull all I think.

But it's not a big jump to setup a "follow" node that replicates only the last X messages from personal friend nodes. At least that was how I was planning to do it, but I hadn't intended to mimic Diaspora's protocol.

What I could do is work with Max's construct and set it up for personal publishing, and distributed following. One node for broadcasting out, and as many different nodes as I want for following updates from other nodes

A system to push/pull selected messages to distributed servers (securely) would kill, and would be the basis of a revolution around the internet. I think Diaspora botched that chance, even though I sent them an email about it. Maybe they didn't really, but I hear their code is bad.

Besides, we need a platform, something like CouchDB, Diaspora is more of an app on top of the platform (but maybe I'm wrong on this one as well).

In a nutshell, easy/secure push would be a killer platform to build the new Twitter/Facebook/FourSquare on.

Of course, this couldn't be monetized easily, so maybe I'm wrong.

Looking forward to restart of continuous replication after restart, according to @couchdb on twitter it's in the trunk :D

@messel: Interesting find :) But I get a bit suspicious when the title says "Brilliant". Do you agree that removing it would be in the spirit of the submission guidelines for HN? (Though, of course, the guidelines do not say to remove such words)

oh brilliant was in my post title.

when I came across Max's app that Guinness image was the first thing that popped up.

To clarify, I'm not suggesting that you remove the "Brilliant" from the linked post, only from the HN submission title.

The Guinness reference is completely lost on me when seeing the picture, and not much clearer when just seeing the title :) I'm sure I'm not the only one...

I don't mind moding the post here, but I can't seem to edit it.

Admit to the odd reference, it probably doesn't click with everyone. Maybe I watch too much BBC on netflix?

I love couchdb so much. It has made relocation and big data crunching so much easier.

This is a really good example of not reinventing the wheel.

Instead built-in functions of the wonderful CouchDB are used - indeed brilliant.

Nice work Max. You should do a show and tell at tonight's pdxruby brigade.

Man I wished in Portland sometimes, just to go to some of your guys hack meetups.

becareful with supplying your email address! it said i had to provide it for my gravatar, but it turns out it is _publically_ visible and _unchangable_ after you entered it..

it's a trap! :-P

:D I don't think Max expected the implementation would get so much attention.

Reply on news.ycombinator.com