Re-using Backbone.js Models on the server with Node.js and Socket.io | Hacker News Reader

Re-using Backbone.js Models on the server with Node.js and Socket.io

andyet.net

76 pointsericflo22 comments

I wanted to cross post this here as I already posted it in the comments.

-----

I am glad you wrote about this, but I feel you are missing the real 'magic' that can be done with node.js and Backbone, or any client code you choose. Socket.io is great, fantastic even, and I think it is something that really shows the power of node.js; there is something built on top of socket.io that is even more wonderful, and it is called dnode[0]. It is really simple to use, but has a lot of potential for server process to server process and server to client communication. It is in effect RPC, and I really can't do it justice.

For the example you give up here, you could easily overwrite the Backbone.sync function and have it instead send the appropriate objects you need to fetch models on the server, while having the server call back to the client with said models. Not only that but you could easily remotely invoke the model changed functions in Backbone directly from the server. There is so much more you can do, but I lack the knowledge at this point to put it in play just yet.

[0]: https://github.com/substack/dnode & http://substack.net/posts/85e1bd/DNode-Asynchronous-Remote-M...

Ohhh this is beautiful. I'm glad to see someone is finally taking node to a direction that could really unleash its potential, rather than re-implementing a rails-like framework in javascript for the nth time.

Granted this is only the first nugget, but the optimist in me wants to see this approach evolve into a framework that truly spans server and client, instead of covering only half of the story.

How's this different than JSON-RPC, for example?

You can pass functions around in your JSON.

That's pretty cool. Like an asynchronous functional version of Cocoa's Distributed Objects.

Are there ever problems with one side referencing objects that no longer exist on the other side?

I've run into these kinds of problems due to the lack of weak references in JavaScript. Hopefully a future version of JavaScript [1][2] will fix that.

[1] http://wiki.ecmascript.org/doku.php?id=strawman:weak_referen...

[2] http://wiki.ecmascript.org/doku.php?id=harmony:weak_maps

Referencing objects isn't a problem since dnode doesn't do replication, it just makes copies, but referencing functions when the client disconnects can occasionally be bothersome. You do get a hook when the client disconnects:

    conn.on('end', fn)

so that helps. At browserling.com all of our backend and frontend spanning lots of processes over several servers is tied together with dnode, and we haven't had any big problems tracking references that I can think of.

Symmetry between code on client and server has long been the promise of server-side JS. I'm working towards a similar goal of reusing Backbone's model/collection structure server-side. It does not attempt to pub/sub events to all clients with an instance of a model/collection, at least not yet.

My project simply proxies client fetch/save/destroy calls via Backbone.sync to the server. Through a RESTful URL convention and Express.js the model and collection of the client's request are recreated, any transformations applied and persisted, then sent back down the pipe. Server-side, models and collections extend their client-side bases and introduce additional functionality (i.e. per-user access control). Models and collections are not kept in memory server-side beyond the duration of a request.

Yeah, I've been using a similar approach. Not as fancy as using DNode or Hendrik Joreteg's approach, but it's closer to traditional web app development and thus easier to wrap one's head around. If real-time isn't a requirement, that little bit of extra simplicity can mean a lot when testing, debugging and scaling.

This is a very interesting look at things. I'm currently building a backbone-based frontend in a very similar way, only without the node.js counterpart - yet.

One problem I immediately see in this hybrid approach is the requirement to keep the state for all [active] users in memory on the server. This does not scale well - which is why conventional web development tries to be as stateless as possible.

It's not hard to remedy this partly by adding basic lazy-fetching and cursor capabilities to Collection. But you lose some of the most interesting features in the process (push-notify on changes), and preserving those at the same time becomes much more tricky then (bordering on classical ORM problems).

There's also still ground-level stuff missing in Backbone, i.e. the support for recursive updates of nested collections (e.g. from one big, nested json blob) is just not there yet. A few approaches have been discussed, but afaik not implemented yet. The models need to be beefed up with some sort of schema (e.g. json-schema) to truly solve this one.

However, none of that is insurmountable. All it takes is some more of the smartness that the Backbone developers have already demonstrated. And elbow grease.

For example I'd love to see backbone paired with 'dnode' or a similar RPC approach. That might open amazing possibilities.

Thanks for your feedback. I've just been playing with this stuff and wanted to put it out there despite its warts.

Regarding the scaling issue. Yes. This is definitely a weakness (I actually mention it as a weakness in the last paragraph or so). We're working on a solution for that using redis as a sort of shared memory. But it's just not ready for primetime and I just figured I'd write up what I had going so far.

Regarding recursive updates of nested collections, the "mport" function in the post works well for re-inflating the entire state. But not quite as well for updates, but for this type of thing, I prefer piecemeal updates anyway, so we don't have to send as much data each time.

I'm definitely gonna check out dnode some more as well. Cheers.

I've just been playing with this stuff and wanted to put it out there despite its warts.

And thanks for that!

In hindsight my first reply came out more nitpicky than intended; all my concerns are aimed at (still to be fixed) shortcomings in the backbone/node ecosystem, not at your particular implementation.

Major kudos for not only coming up with (at the very least) an intriguing Proof Of Concept but then also writing a great blog post to describe it. This will be learning material for myself and many others.

Thank you. No offense taken :)

All the hubbub about the hashbang URL system got me thinking about sharing client/server javascript just this morning, and then lo and behold, a post about it pops up a few hours later.

Instead, though, I was thinking that perhaps controllers and views could be re-used across the server and client. When a request comes in to the server, it renders a page just like a typical MVC framework would, but also directs the client to download the view/controller javascript. Any links the user follows could then be rendered client-side. When the framework is running server-side, the model layer talks directly to the database - when it's running client-side, it talks to the server through JSON.

It could give you the best of both worlds - an initial request returns a quick, fully-formed HTML page, so it's speedy on the first load and crawlers will know what to do with it. Later requests use AJAX, so things are responsive and the server isn't weighed down with HTML rendering.

Take a look at liftweb. http://liftweb.net/ Its Scala which I don't know, but I've got some very intelligent friends trying to get me to back it at our company. (Scala is a lisp that runs in the JVM. - Someone's going to hang me for confusing what lisp is.) Its a framework that can leverage websockets/comet/longpoll to extend your classes to work client side.

Think message passing between client objects (in js) and server objects (in scala). You can do jqueryish things server side to the client side pages. (That blew me away when I saw it in action.) Your submit even can pass params to the server object. It's really simple and strait forward. I'm looking for lift for NodeJS.

Check out the ubiquitous "chat server" example in the simply lift book. http://simply.liftweb.net/

Quora seems to be taking this approach. Except, after the initial request, mutations make the server send rendered HTML fragments instead of raw JSON back to the client. I believe this is due to the fact that their view templates are coded in Python so they can't be easily reused on the client side.

Is it me or is this proposing, for the sake of symmetry, that you expose server-side implementation details on the client? Unless you have client-only and server-only versions of the models, I would think that's what you have to do. It seems like that's exactly what's happening when you have, in client-side code, a boolean called "server" that indicates whether it's running on the server or not.

Now, off the top of my head I can't think of a particular way that this could screw you. But I can't say that it looks like the Right Thing to do either.

I don't know how far the Asana team got with Luna, but I think their approach for a unified client-server model is much closer to how I'd want things to work: http://asana.com/luna/

Thanks for that link. I hadn't seen that.

that you expose server-side implementation details on the client?

This particular impl does it with javascript conditionals, but there's no reason the client must see code that it never executes anyways. The equivalent of an #ifdef (ideally buried under a framework rug) goes a long way here.

Moreover a function does not even need to resolve to the same code on the client and the server. For example a model-validation function could evaluate to the actual validation logic on the server (incl. DB lookups etc.) and to a mere ajax RPC (calling that function on the server) on the client.

I could very well see a smart node serverside-library that let's you write one js file and then transparently mangles it as necessary before sending it to the client. The trick is to keep the magic down to a sane amount (slippery slope!) so that the end-result remains always predictable.

Luna

Yes, it seems luna is basically taking this approach.

What I don't like about luna is that it's not javascript and that it's, for my taste, taking the abstraction too far. It claims:

  the client-side JavaScript, the server-side SQL, and everything in between

That's too much. Sure, for the developers it's comfortable to sidestep a whole cascade of hard problems simply by assuming control over everything ("My way or highway"). But this approach has not proven very successful in the javascript-world. We love small and pluggable components like jQuery or Backbone. The monoliths like Dojo or ExtJS tend to be too limiting on the My part of the way.

I agree with your comments about Luna. About the #ifdef proposal, I think a better solution might be that your server-side model object has a method like getSlaveObject() that gives you a version of the model that is acceptable to be seen on the client side.

Yes, you are probably right, my examples were just meant to deliver the rough idea, not fully shaped solutions.

Lots of room for experimentation and exploration in this space - exciting times ahead.

I really could have used this tutorial a few months ago. I made a bit of a toy node/backbone app. My first iteration attempted to unify the models across frontend and backend, unit testing them, etc, but it ended up being so kludgy and hard to debug that I went with frontend only and just stored the JSON on the backend. The benefit of the OP's strategy however is that its much safer to have the parsing and validation on the backend so people can't put bad stuff in your database as easily.

In the end I ended up redoing the project as a CLI app in Haskell.

That's quite the course correction, going from backbone/node to CLI/Haskell.

Reply on news.ycombinator.com