Deepstream.io – Open Realtime Server | Hacker News Reader

Deepstream.io – Open Realtime Server

deepstream.io

166 pointswolframhempel67 comments

I really like that the documentation is quite honest and unbiased. Like with the integrations: https://deepstream.io/tutorials/integrations it always mentions downsides next to the upsides.

This is a really good point actually. I particularly liked the overview of other realtime libraries[1] for the honesty factor, and it really helped me understand how it relates to more familiar frameworks too.

[1] https://deepstream.io/blog/realtime-framework-overview/

This is great! I wish somebody would write a Firebase API compatible wrapper around this so escaping Firebase lock-in would be possible without altering code.

I might start it myself when I find some time.

Or even just integration testing Firebase without hitting a remote service!

that might be overkill. Here is a better solution: https://github.com/urish/firebase-server

I looked into it. firebase-server doesn't work with recent firebase clients and it's not a priority for the author.

From a quick look at it it looks like a self-hosted Firebase.

Would that be a good approximation?

yup, that's not too far off. It also gives you Pub/Sub and Request/Response though. Also Firebase uses one huge data-structure for data-sync whereas deepstream breaks data down into smaller objects called "records" with individual lifecycles

Huh. I visited your site and reading the tag line "a fast, secure and scalable websocket & tcp server", I assumed it was just a socket server and had no idea it was actually a data-store.

Yeah I mostly understood that from the example.

You really need to make the explanation clearer ^^

Very much agree. Would you have any recommendation?

I would show what it does step by step: one box with data-sync, one with real-time data sharing, etc... with an animation for each.

Your animation "built for a new generation of apps" is awesome but it doesn't not show clearly what role deepstream plays in it.

I don't think it is possible to explain everything clearly in one sentence so I wouldn't start with a tagline like that.

+1 something like this is defintley needed

I know the question wasn't directed to me, but the animation on the 'Cluster Messaging'[1] page really helped me understand.

[1] https://deepstream.io/tutorials/core/cluster-messaging/

While it looks interesting, I honestly don't feel like building a system like this on top of node. Not because of the library itself, which seems properly implemented (hard to see libraries using proper formatting and JSDoc these days).

node is a simple, friendly technology, until you to profile it. Then it's not so friendly. With a realtime system you require a great deal of profiling and quite honestly the tooling is not there.

"Oh but what about Flamegraphs and DTrace?"... it is nothing compared to what you have in Java. And node tooling is better on obscure operating systems like IllumOS.

If I had to implement a system like this I would opt for Scala + Akka, or Elixir/Erlang. If it's for simplicity/budget reasons, try PubNub.

What tools do you use to profile a Scala + Akka service? IME they are quite opaque to traditional Java tools like VisualVM. What am I missing?

Scala can be profiled with VisualVM.

You can monitor Akka with kamon.io, or using the tool offered by the authors.

Genuine question, how does this different from https://github.com/centrifugal/centrifugo ?

I'm afraid I don't know too much about centrifugo, but it looks like a pub-sub server you can connect to Redis. deepstream on the other hand provides datasync (stateful, persistent JSON objects you can subscribe and listen to), request-response and pubsub in a scalable way with granular permissions and authentication strategies and an ecosystem of connectors that make it work with different databases, caches and message buses.

Thanks, any plan for the android client?

sorry, not quite following. There is an android client here:

https://deepstream.io/install/android/

Looks quite interesting, I was evaluating few real-time frameworks (esp socket.io) for the past couple of days . Could you pls shed some lights on how easy to use deepstream to setup a one-to-one in-app chat/messaging setup for android? I guess its the event's are the way to go.

For eg: client.event.subscribe('channel1', eventCallback) client.event.emit('channel1', 'hello how are you?')

How scalable is that? can it support thousands of channels without any hiccups on a single medium digital ocean instance?

It scales very far. We've conducted benchmarks lately that showed that a single deepstream on an AWS t2.medium instance can comfortably handle 160.000~200.000 messages a second and a cluster of six ~ 4 Billion an hour. Please find the detailed results here

https://deepstream.io/info/performance/four-billion-messages...

For your chat though you need to consider whether your messages will be persistent and mutable (changeable after they've been posted) or are just one-off messages. In the first case using a list (https://deepstream.io/tutorials/core/datasync-lists/) with references to records (https://deepstream.io/tutorials/core/datasync-records/) is the way to go, in the second case pub/sub (https://deepstream.io/tutorials/core/pubsub-events/) might be better

Do lists offer pagination for lazy-loading?

Ah sorry, didn't notice.

Do you provide a version of Firebase's onDisconnect? https://firebase.google.com/docs/reference/js/firebase.datab...

It's very hard to correctly manage distributed state without a guarantee similar to this.

Yes, there are granular connection states and associated events. Messages are queued during connection loss. Please find more information here.

https://deepstream.io/docs/client-js/client/#connectionstate...

https://deepstream.io/docs/common/constants/#connection-stat...

Sorry, I should have been clearer. I don't mean "does Deepstream tell the user when it's disconnected", I mean:

Does the Deepstream client have a facility (like Firebase) to instruct the remote server to perform an action in the event the client disconnects in the future and the client is unable to get any further message to the server?

ehm...this is not Horizon. But its a good question none the less. No, it does currently not have that facility, but it would certainly be a good addition

Hah, sorry, too many tabs open lately. I'll edit. Thanks for the answer.

Since this is not built-in, it may be possible to make with deepstream's onClientDisconnect.

https://github.com/deepstreamIO/deepstream.io/issues/191

Also they seem to be working on 'presence'. https://github.com/deepstreamIO/deepstream.io/issues/118

Could you please shed some light on concurrent connections benchmarks ? Im looking for a scalable solution - I am trying out socket.io and I read from others there are some performance issues as it scales.

edit: added socket.io part

How does this compare to something like ShareDB (https://github.com/share/sharedb)? It seems on the face of it that the data part of it is pretty similar (albeit Deepstream is more fully featured with auth etc.), however, there is no mention of how Deepstream handles conflict resolution if two clients are modifying the state.

Internally ShareDB uses Operational Transformation with special operations which are designed for use in JSON documents.

Does anyone know how this works in Deepstream?

There's a choice of merge algorithms as well as the ability to hook in your own.

https://deepstream.io/tutorials/core/handling-data-conflicts...

Thanks! I hadn't found that before.

Sounds like having those merge algorithms/hooks and splitting up the document into multiple "records" might actually be enough for my use case.

Wow, deepstream looks really good (and I'm a competitor, http://gun.js.org/ ). Your guy's product seems to have significantly matured over the last year. I'm very impressed! Up for chatting? mark AT gunDB DOT io (I still like your guys old website better though with the server/client graphic animation).

Ooh, thanks so much. Just heading out for the inevitable "we've made the frontpage drink", but let's get in touch tomorrow. The diagram is still there, but buried in https://deepstream.io/tutorials/core/cluster-messaging/

Yeah that one! I love that graphic! Your profile doesn't have your email address in it, so you'll have to ping me. Tomorrow sounds good. Congrats on the launch!

I love NodeJS. So many NPM's to be had. I want to open source our ANSI-C Realtime Server - maybe someday. Our https://www.pubnub.com/blocks-catalog/ BLOCKS engine actually starting in Java. That was a M-I-S-T-A-K-E-E-E-E. It's better now.

I like it but coming from socket.io I have a hard time with it. For instance, Where/how would I put my server logic? The "Node API" as its called isn't really helpful or not documented enough. But then again, I could have misunderstood what deepstream is all about or how you are supposed to be using it.

It's a different beast from socket.io :-) your server logic is just another client, written in java or node. There's a lot of functionality that let's you listen in on what clients request etc. Please find more details at https://deepstream.io/tutorials/core/active-data-providers/

Can you state your stance on AGPL[1] on the website like MongoDB and RethinkDB do to put people's minds at ease.

[1] https://github.com/deepstreamIO/deepstream.io/issues/298

Realtime? In nodejs?

It must be a webscale realtime, or at least reactive realtime..

If you're willing to put the work in to load balance then it can probably scale quite well.

I'm always baffled why people choose JS over Erlang/Elixir which is designed to do this and comes with the tools/libs required to scale.

Exactly word "probably" is a deal breaker in this case. I am also curious how someone was thinking JS will be a good idea for such project. But hey, there are people building cars out of plastic bottles...

We are working with Alex,creator of µWebSockets (https://github.com/uWebSockets/uWebSockets) on moving all connectivity and networking aspects to low level C

yes, that's one major aspect of it. engine.io adds is own protocol and isn't exactly the most performance tuned library

Is there some data available on which factor the perfomance benefitted from this change (like some benchmark results)?

Sorry, it doesn't let me reply for some reason :-( Development is in progress, there will be figures soon

It seems like you have to wait some seconds for the server to process the comment, if you refresh it should let you comment again. Anyway, looking forward to the data release, through a blog post I suppose?

That feature of HN is by design. It's there to force you to think about your replies instead of engage in knee-jerk flamewars. :)

Would this mean you are getting rid of engine.io as the base, or is this just meant to be a general addition?

Are there any plans for an iOS/Swift client? So far it seems only js and Java clients are available...

Yup, we're in the process of finalizing it. There's a related issue at https://github.com/deepstreamIO/deepstream.io/issues/68 with links to the beta artefacts.

Nice, really appreciate the progress here! However, is there any reason the artifacts aren't on github? It may be more convenient when it comes to issues/changelogs etc.

agreed, they will be soon. At the moment the core is cross-compiled from the Java-Client using j2obc, hence the underlying repo is the java one. We are in the process of evaluating how well this worked.

:D I am actually trying to do something similar with the js client. I recently tried to convert it to Dart using jsparser, since i am working with http://flutter.io, a hybrid app development project from Google. It worked out just fine for now, currently testing it to get the weak spots fixed.

How does this compare to RethinkDB / Horizon ?

Please find a detailed answer here:

https://deepstream.io/tutorials/integrations/db-rethinkdb/

oops, thanks so much, fixed it

Can't read the text, please review the contrast part of https://www.w3.org/TR/WCAG20/

This might help you out for now (if you are using Firefox): https://support.mozilla.org/en-US/kb/accessibility-features-...

I'm confused. 1.1 has been released 13 days ago https://github.com/deepstreamIO/deepstream.io/releases/tag/v...

yes, but we wanted to give it an initial test phase before announcing it to make sure we didn't miss anything major

Isn't that what a release candidate is for?

I'm still confused. You are announcing 1.0 but the actual version is 1.1

Page Not Found. When I clicked on Getting started

https://deepstream.io/core/getting-started-quickstart/

Edit: Clicked on tutorials and then Getting started.

Reply on news.ycombinator.com