Show HN: Deepstream.io – A scalable server for realtime webapps | Hacker News Reader

Show HN: Deepstream.io – A scalable server for realtime webapps

deepstream.io

159 pointswolframhempel41 comments

I looked at the source code and it seems that it mostly glues socket.io with Redis/Memcached. What did I miss? How exactly is it more scalable than anything else?

The front-page looks really cool.

It goes a bit beyond that :-). Deepstream scales horizontally by allowing to connect multiple deepstream nodes to a message bus (e.g. AMQP, Redis, Kafka etc...). The nodes multiplex messages within each other and sync their client's state.

I'm very excited about this! Without reading too much into this, is it safe to say it's like a self-hosted firebase?

"Due to the way deepstream structures its data (JSON blobs, identified by a primary key) it is recommendable to use an Object oriented or schema-less database (e.g. MongoDB or RethinkDB), rather than a relational database like MySQL."

Is there a way to run complex queries against the data or it designed to abstract that all away from you to be as fast as possible?

Yup,the data is still queryable.here's an article on the subject http://deepstream.io/tutorials/searching-and-querying.html

One of the major limitations of firebase is that it's not possible to filter results by permissions. Is this something that deepstream can do?

"it's like a self-hosted firebase"

Basically, yes. see the subscribe() function.

Similarly (without reading too much about it) can I use InfluxDb instead of RethinkDb? (Disclaimer: I'm a RethinkDB fanboy, but I'd like to know my options.)

InfluxDB is a time series database (think reporting/analysis of collected system metrics). It's not intended as a general purpose database.

No, its definitely not. However, I can think of a few instances where I might use it along with Deepstream.

Why would you compare InfluxDB and RethinkDB? Those are two databases for very, very different problems. CouchDB and MongoDB are more comparable to RethinkDB if that helps you get a start!

Does anyone know with what these style of websites are made with? there must be hundreds of .io sites here on HN that look and feel the same

I am not a designer so do not know what this latest web design practice is called

I've used the Startup Framework Generator[1] for building this style of website. Highly recommended.

[1] http://designmodo.com/generator/

Oh wow, this is great. Thanks for sharing!

I would love to know this too. This one is exceptionally well designed, here is a site with some landing page templates that have some similar design themes: http://html5up.net/

I think it's more a design trend than a magical tool that creates sites like these. Look up flat, metro and material design if you are looking for a name.

It's to the point where I'm becoming embarrassed to release anything unless I spend a month doing graphic design on the launch page!

Most that you've seen are probably using bootstrap (http://getbootstrap.com). Though the only indication of bootstrap I'm seeing within this particular site is font-awesome loaded from the bootstrap cdn.

Very necessary. A few questions:

- How big can records be? Are they stored as a single object in the persistence layer or using something like materialized paths. The firebase data model is one huge JSON structure, so this looks different.

- How are conflicts handled? Last writer on server wins or last writer anywhere? Do you ensure eventual consistency of all the clients even when you apply optimistic writes on the client?

- Can data be larger than the cache size? How is cache eviction handled?

Thanks - Records have no hard size limit - but are limited to the maximum size of your cache entry or database document. The tcp connection can send a single record update over multiple packets.

- Records already have a version that's incremented with every change and also persisted as part of the record's meta-data. Currently deepstream uses a simple first-wins strategy and subsequent updates for the same version number get rejected. However configurable merge strategies are in planning as part of a "record-options" setting.

- Currently data has to be explicitly deleted from the cache.

This looks similar to SC http://socketcluster.io/ - But I guess SC doesn't take care of the storage layer - You can use your own database of choice.

It’s clear that you’ve designed something that gets people’s attention (your animation on the homepage might be one of the best examples of real time apps I’ve ever seen), but based on the questions in this thread…people are looking for differentiation.

If I were you, I’d start by answering everyone’s questions here. These are your potential users and leaving them hanging seems odd. The next step would be answering those questions on your site.

This is the kind of service that people are going to start using for a new project (rarely are people going to switch existing infrastructure to something brand new). But any new framework, language or platform requires investment and with developer related services and tools, people are going to research first. Help them do that research efficiently.

Good luck taking this to the next level!

Thank you for your feedback. You're absolutely right, but I guess we've missed the window for Hackernews to an extend now. We're currently putting together a "how is deepstream different from ..." page and a general "about" section for the webpage which will be released shortly.

Again, thanks for your advice, much appreciated.

Wow, much needed, I'm downloading the samples right now to take a look.

(fyi: bad link at the bottom of this page http://deepstream.io/tutorials/simple-app-using-react.html it links to the ko demo, not the react's)

thanks :-). We're working on fixing it.

Very nice self-explaining landing page! Although the bulleted list webfont renders rather poorly in Win7 Chrome: http://i.imgur.com/j0a0rAQ.png

oh dear, we'll definitely have to have a look into that. Thanks for posting the screenshot :-)

Same in Win7 FF38

This looks promising. Does this server architecture also has an embedded API framework which can utilized to create APIs by just specifying end-points? and is there any testing framework integrated for node.js applications?

I believe the permissions example with Lisa in the tutorial has it's logic backwards -- or am I missing something?

http://deepstream.io/tutorials/permissioning.html

Link to client api is broken - http://i.imgur.com/egcKVyv.jpg

(case insensitive file system on macos may be? :))

This looks great! I am thinking of writing a Firebase compatible wrapper for this when I have free time. The main differences are in permissions and auth I guess.

That'd be great :-)

Interesting. How does it differ from https://getstream.io ?

This is for realtime apps, and Stream is for scalable activity feeds. They're not really similar at all.

The GetStream server isn't open source, is it?

I believe they Open Sourced a significant part of it here - https://github.com/tschellenbach/Stream-Framework

Is there a documentation of transport protocol? i.e http sse, websocket? ...peeking the source code a little bit - tcp-socket?

Deepstream is using a TcpPort to allow for connections from other Node-processes (and hopefully, going forward Java, C++ etc..) and engine.io for browser communication. Engine.io initially connects using long-polling http and upgrades to Websocket if available.

A (rudimentary) overview of the message structure can be found here: http://deepstream.io/tutorials/message-structure.html

Why go through the trouble and reinvent the wheel and not use one of the available protocols and serialization methods?

Very nice. I'd be curious to hear a bit on how this is different than what Meteor provides when using MongoDB. Thanks!

What about offline first?

Reply on news.ycombinator.com