DIY Meteor-like Realtime Functionality Using Socket.io and RethinkDB Changefeeds | Hacker News Reader

DIY Meteor-like Realtime Functionality Using Socket.io and RethinkDB Changefeeds

scotthasbrouck.com

71 pointsginkgotree13 comments

RethinkDB changefeeds are great for dealing with data streams which are common to all users, but changefeeds can get tricky when you need to deal with hundreds of thousands of data streams which are more or less unique to each user (or unique to many small groups of users) - This is particularly true when you need to scale your app beyond a single process/instance/machine.

The approach described in this article may not be suitable for a lot of use cases. It's fine if all users shared the same central collection of Fruits (E.g. a giant Fruit basket)... But what if users were divided into many small groups and each group had its own independent Fruit basket? It becomes extremely difficult to manage all these unique changefeeds on the backend (especially if you have to scale beyond a single machine).

The team behind RethinkDB is currently working on a project called Horizon (previously called Fusion) which should solve this problem.

What approach would you take to solve the problem you describe of individual fruit baskets that can scale?

The simplest way to implement this is using a client-side pub/sub engine and have a channel subscription for each resource on the frontend.

So for example, if you have a Fruit basket with id 123 and you want to get realtime updates when that basket changes (E.g. new fruits are added or whatever), you could subscribe to a channel 'fruitbasket/123'.

You would have a separate channel for every basket but because the backend treats all channels in a generic way, the complexity on the backend remains minimal (and is handled by your pub/sub engine anyway).

You need to make sure that your pub/sub engine is good at handling lots of distinct channels (and is designed to be user-facing - A lot of pubsub engines like Redis are only designed for backend use). There are user-facing pub/sub SaaS services which you can hook into your app but they are a bit expensive.

If you want a self-hosted user-facing pub/sub solution, there are only two popular options:

- SocketCluster https://github.com/socketcluster/socketcluster

- Faye https://github.com/faye/faye

Both Faye and SocketCluster are good at handling lots of unique channels - Based on benchmarks I ran, both can handle the same load of about 10k new subscriptions per second (per CPU core). SC makes scaling across multiple CPU cores trivial so if you have an 8-core machine, you can handle 80K new subscriptions per second with SC out of the box. You can also scale with Faye but it will require a lot more work to get working.

Disclosure: I'm the main author of SocketCluster.

Great write up! I wrote about this idea last month, although in a less detailed way. I've also included a repository in my article with a simple React and Redux front end that demonstrates some of these concepts.

http://www.andrewcoelho.com/essays/rethinkdb-realtime-apps/

Oh that's a really cool example! Great idea with the animated gifs, I think that really helps it sink in how powerful combining these two technologies really is.

Hi a small note, it's impossible to see links on your blog, make them easier to recognize.

Noted, underlined them. Thanks!

One thing that I've never understood is why websockets instead of server sent events?

WebSocket is more well known than SSE, and it's also what Socket.io uses, so I think this isn't always a conscious choice. Developers are still discovering that SSE is a thing.

Another factor is that most apps using a socket-ish connection tend to run almost all traffic over the socket as opposed to just "push" traffic. This can make sense if the exchanges aren't very resource-oriented compared to a REST API.

Possibly Internet Explorer support. Even Edge doesn't support them yet[1].

With deployments of HTTP/2 increasing, I think we'll see greater adoption of server sent events. At the time of writing, HTTP/2 and WebSockets are two incompatible standards. For non-latency sensitive applications (i.e. not games) it is very easy to implement a WebSocket like interface over HTTP/2 using SSE (server -> client) and POST requests (client -> server) with very little overhead (thanks to HTTP/2's multiplexing).

[1] http://caniuse.com/eventsource

I'd love to use a sane, simple SSE server impl in Java. Does one exist? I see there's very little about plain java event source servlet implementations. Looks like there's some promising things in Jersey and Atmosphere at least, but things tend to get complicated quick in the java world...

meatier is this but less DIY.

Perhaps downvoters thought you were making a joke? https://github.com/mattkrick/meatier is very appropriate to this discussion.

Reply on news.ycombinator.com