Meteor 0.7.0: Scaling realtime queries using oplog instead of poll-and-diff | Hacker News Reader

Meteor 0.7.0: Scaling realtime queries using oplog instead of poll-and-diff

meteor.com

101 pointsdebergalis23 comments

It seems as if continuing to address these scalability issues is what will be one of the largest deciders in Meteor's future. It seems as if I near constantly am perusing posts on the framework here, and after going through the Discover Meteor book as well as several small projects that were largely simple prototypes, I've began to have a lot of faith in it for these reasons:

1. It is EASY. Like, really, really easy. I went through Discover Meteor in less than two days, and was easily imagining how to implement other applications after one run through.

2. It is accessible. The free deployment on the .meteor domain, the open source sensibilities, the constant growth. It's got a good snowballs path going on growth.

3. The developers really believe in this product. Everyone on HN has an opinion of the best way to develop software. That said, I feel there is a certain intrinsic respect we share for one another here(MOST of the time ;]) and to ignore developers who are as passionate as these guys building a functional product that clearly has room to grow, we'd be fools to ignore them if we didn't have the time and/or interest (and the interest is hard to not have with what it offers so easily).

This article seems to really further my faith in Meteor. Stopping the "pull and diff" method is a great first step towards making it truly scalable in the sense other frameworks have developed the capability.

I've been working with the framework almost daily and regarding your comments:

1.) Yep. I've gone from a jQuery-mostly front-end dev to being able to build full blown apps inside of 6 months. Not only that, the apps run really well.

2.) Incredibly. Even without the meteor.com service, it's super easy to deploy using things like Demeteorizer (https://github.com/onmodulus/demeteorizer) which strips down a Meteor app to a raw node.js app that can be deployed anywhere (that supports node, of course) in < 2 minutes.

3.) My favorite part about the framework is that the core devs have meetups/hackathons frequently to talk about progress on development, show off demos of pre-releases, and allow community devs to showcase their own work (if you're curious, I highly recommend checking out the Devshop videos: https://www.youtube.com/user/MeteorVideos?feature=watch).

Even if it doesn't sound like your bag of chips, at least do what the OP did and check out Discover Meteor: http://discovermeteor.com. It's worth the day or two it will take you to complete.

The core devs are great with how active they are. Some have even reached out to me on twitter when I've mentioned them. It's great to see how much passion they have. What sort of product are you working on with meteor?

Right now I have two available: https://properapp.com and http://whatwhenapp.com.

These look great! Need any help with development? I'm about junior-dev skill level but am actively looking to get into some side projects until I land a job in the industry now that I'm out of the programming bootcamp I was at. I'd ton a bunch of C/Python/Ruby before that, but it really solidified my core software dev skillset and is allowing me to learn at a rapid pace and I'm loving it :)

EASY is almost an understatement. The Meteor team nailed it on this point.

IMO, everyone should take a look at Meteor if for no reason other than to see what's possible.

In terms of value, they pretty much eliminated all of the low value bit moving that consumes a substantial portion of software development hours.

Was 0.6.7-0.6.9 skipped? There was a release candidate for version 0.6.7

The big change with this version is the oplog tailing instead of the poll and diff which has dramatic performance improvements over the previous version. More details at the wiki: https://github.com/meteor/meteor/wiki/Oplog-Observe-Driver

About the version number: https://github.com/meteor/meteor/commit/8bd400e56065b0f2d61e...

I would have expected Meteor UI to be in 0.7, but that not a big deal and that will probably be in the next version.

Yeah, we have several big projects in flight, and we're shipping them in the order that they're ready.

We push them out first as prereleases (which you can run by passing a --release flag to meteor, and it will automatically download and install everything) and announce them to meteor-core and see what people think. In this case both oplog tailing and Meteor UI were out as prereleases, but there was more feedback and more feature requests for Meteor UI, so oplog tailing ended up winning the race.

We are rationing version numbers before 1.0, since we have multiple major projects that're slated to land in between now and then, including Meteor UI and integrated package management. (Though as Node has shown, an 0.10 wouldn't be the end of the world.) But we decided that oplog tailing is a big enough deal that we should bump the middle number.

I'm working on a project currently that needs this but we're an express / mongo stack and not a meteor stack. Anyone know of anything similar I can add to the mongodb node driver? Or can I switch to meteor's node driver without the rest of meteor?

I am not well versed enough to directly answer, but it seems as if this might point you in the right direction. http://stackoverflow.com/questions/14400839/combine-meteor-a...

@krebby, you can create a DDP client that works outside of Meteor, which would allow you to run a Meteor server + Mongo with your own front end.

The other option is to use Racer from DerbyJS.

If you're interested in adding a real-time data layer to an existing Node app, then Derby is probably the easier path.

Trailing the oplog for events is a great strategy. 10gen use it to generate point in time snapshots for disaster recovery and various companies have described systems doing this to react to data changes in mongo. Web Scale database triggers. Ahrmm.

Side note, 10gen has been renamed to MongoDB Inc

Given the flak MongoDB has been getting lately, that might have been a bad decision.

Can't you scale realtime queries by just timestamping everything in the database so you know if something comes back in your query where timestamp > oldtime, then it's new?

Looks like Meteor is becoming more like the Q platform.

http://qbixstaging.com/QP

http://github.com/EGreg/Q

(Not yet released though :)

No.

Why

Pretty cool framework Greg. Disregard comment above, they lost every key on their keyboard except for two.

i dont see the connection.

could you explain that?

Yeah - most client side MVC frameworks wind up having to sync the client Model and the server Model, and are based around data (what something is). In Q we took the approach of streams and messages, being broadcast to all participants and offline notifications sent to subscribers. So the apps are encouraged to listen for messagea pushed from the server, which may be "data changed" but could also be a chess move, or a song added, etc.

Reply on news.ycombinator.com