Getting Meteor to 1.0
meteor.com
meteor.com
I really wanted to like it. I was predisposed to do so.
But from the very beginning, I was confronted with something I consider a deal breaker. While I am wary and generally disinclined toward the increasingly popular pattern of curl an installation script and pipe into sh, in this case it is not only skeptical, it is downright ludicrous. Why would I, nay anyone, want to install an npm package this way? Should not the installation instructions simply be `npm install -g meteor`? And what if I need to work on multiple meteor projects with different versions? Surely there is support for adding it to your package.json file, but why is this not the primary means of installation and well documented?
Maybe I am being overly sensitive to these issues, but it really baffled me that the very foundation upon which a meteor project is predicated would be contradictory to the typical node workflow. I will probably try meteor to spike out a project at some point in time, but I do not foresee myself using it extensively in the near future.
Then create a site with it and specify whatever local version of meteor you want.
The deets: https://github.com/oortcloud/meteorite/
$ curl https://install.meteor.com | /bin/sh; meteor create myapp; cd myapp; meteor
you're up and running with a complete stack including MongoDB, node, the dozen-ish core packages that make up the Meteor pubsub and realtime templating stack, and the 'meteor' build tool (which can do things like compile coffeescript and less, generate source maps, minify your production code, and provide a realtime development environment where whenever you save a file your app updates in your browser.)
As for versioning, it's actually got a great way of doing that. Each of your Meteor projects is locked to a specific Meteor release version (similar to an Ubuntu release, it's a release-engineered snapshot of the Meteor core packages), which you can set with the `meteor update` command. And this has automatic `npm shrinkwrap` integration so that if you use npm packages alongside Meteor packages, everyone on your team is always running exactly the same versions of all of the code in your app. The Meteor tools automatically handle all the work of downloading any needed Meteor versions and keeping them installed side-by-side on your laptop, using the correct one for a given project, and notifying you of available updates.
It sounds like the installation script, aside from handling the MongoDB and node setup, could be treated like a typical node package. Based on the fact that meteor has no package.json, seemingly none of those core packages are installed via npm either.
I may be alone in this, but I guess I would just prefer if my web framework expected me to have the interpreter and database installed, and used the package manager already provided for this environment.
It converts Meteor apps into more traditional node.js with a package.json
Meteor is built against specific node and npm versions and the install script installs the necessary node binaries.
Considering what Meteor brings a developer, getting hung up on the install method is missing the big picture.
It is about my desire to work on a node application in what I consider to be a standard way. I want to be able to treat my entire application as a node package, leveraging npm for dependency resolution.
Think of it like a rails application. For local development, you probably use rbenv (or rvm), bundler, and have the version of rails you need listed in your Gemfile. You allow the package manager provided with the language and maybe a layer to keep environments separated (already provided by npm) to take care of dependency resolution and updates.
I want a node framework where I can use the package.json file as one does the Gemfile for ruby applications.
With regard to why not "npm install -g meteor", I think the short answer is that the Meteor installation setups up/installs a lot more than just one node package.
Saying that Node has a package manager is a huge understatement. Node has one hell of a package manager, but you have to hack[1] around Meteor to use npm modules, even in the latest version.
My impression of Meteor, and the impression a number of node people seem to share, and which I've seen reinforced on HN, is:
Meteor is in an almost adversarial relationship with the node.js open-source community, because while they're good hackers building something cool, they took millions of dollars in funding[2] and want to keep their options open for monetizing that codebase.
This makes every decision they make differently than other open-source real-time frameworks (to build their own package manager, require a contributor agreement on an MIT licensed project, and use a nonstandard install process[3,4]) considerably more worrisome.
And it's frustrating, because there's nothing going on in Meteor that necessitates going outside of the "node.js ecosystem" -- aka NPM [4,5,6].
That impression might be unfair in some way, and obviously the meteor people spend a lot of time responding to just these sort of concerns from the node.js community.
But "Meteor" means "worry" to me.
---
[1] http://meteorhacks.com/complete-npm-integration-for-meteor.h... [2] http://venturebeat.com/2012/07/25/meteor-funding/ [3] https://atmosphere.meteor.com/ [4] https://github.com/meteor/meteor/pull/516#issuecomment-12919... [5] http://derbyjs.com/ [6] https://github.com/substack/node-browserify
The incredible amount of work that we've put into JavaScript build tools over the last two years has all been with the aim of creating a radically easier, faster developer experience, because we know that that's incredibly powerful marketing for the rest of our crazy ideas. In other words, we did it because we thought that the UX of the existing JavaScript tooling was just too janky. Seriously, try Meteor for yourself and see what you think -- maybe you think our work sucks and that we wasted our time, but if so, I wish you would just tell me that (I would genuinely love to know) instead of reading ulterior motives into what was a labor of love and a ton of hard work.
Of course there are many rough edges and it's not done (that's why it's not 1.0 yet) but from people that have actually used it for a while, we actually get the opposite feedback, which is that they want us to go much further down this path, and that's why we continue to slave away at what is by far the least fun part of building a framework.
But about me 'imputing bad motives', or whatever -- I said "worry", and "almost adversarial", and I stand by that.
When a company pops up and says "Hi, we're from Silicon Valley, and we have a plan to monetize a framework made from your language of choice; we have VC funding and we leap-frogged similar community frameworks -- now, if you want to write apps with the new hotness, please use our new package manager instead of the excellent community one, and don't install our framework with the community standard method either" ...
You worry. You just do. I think that meteor/derby/etc style app dev is the future, so I'm worried by anything that might threaten to lock it down.
I'm glad to see that upon investigation that worry is diminished, though. And obviously I applaud your efforts in advancing that future (I just wish that it were done in an 'npm install meteor' way).
Meteor is one of those platforms that you should try. I could give you a menu of features, but the productivity boost from Meteor is one of those things you really should experience. The productivity boost is very real and its quantifiable.
An experienced developer can get a very good feel for Meteor over a single weekend: 1. Buy the book: http://www.discovermeteor.com (the book is $39...I did not purchase the other packages) 2. Spend a weekend doing nothing but coding
Come Monday morning you'll know what Meteor is about.
PS: I am not part of MDG, and I have nothing to do with the book (other than having bought a copy).
My perspective as someone who has been using it since its release. It depends on what kind of app you are building.
Coming from RoR, Meteor actually is more of a pain given that it still has edge cases that can eat developer time, especially once you dive past the initial levels of tutorials and documentation. RoR also has well established best practices and norms, unlike Meteor and perhaps Node.js as a whole. This means you will be writing a lot more boilerplate code. Meteor has some memory issues as well at times. You have to be aware of memory leaks and will frequent the profiler often, especially with large client-side applications. I've had some projects that I've worked on in Meteor hit 1GB of RAM client-side. Recent releases have been working on this, though. Live reload and hot code pushes are kind of a moot point with something like the Live Reload plugin for Sublime Text or equivalent.
For realtime applications, Meteor is a whole different beast. Realtime applications are much easier to code. I'm comparing this to making a realtime Node.js application. Meteor takes care of the boilerplate for you, pushing changes automatically to the client. It's not difficult to do something like socket.io and node.js but it is nice to have that all handled for you.
It's great for getting an application off the floor - it has a lot of that magic RoR had when it was released. Once applications get more advanced, the structure and maturity of RoR make it easier to work with. These issues are things Meteor is working on, especially in terms of stabilizing the spec/API and working on fixing the edge cases. With 1.0, it seems they are bringing some more stability to it by making it production ready. Its strengths are much more apparent in realtime applications as compared to a traditional web framework.
Edit: As compared to other node.js frameworks like Express, Meteor is much easier to work with - I probably would not go back to raw Express after Meteor. It's been moving fast to 1.0 where many of the issues I've had with it should be resolved with stability and as edge cases are resolved.
# Labor and spend allocation
- 75% of engineers were on our server-side (Java) (1)
- 25% of engineers were on front-end (JavaScript/HTML/CSS)
# KLOC (2)
- 14% Java
- 54% JavaScript
- Everything else is in other (Scala, CSS, HTML, shell scripts, SQL, etc.)
From speaking with other companies, we found these ratios are within the normal range. We've seen 60% server-side to 80% server-side, but the directional finding appear to be actionable. As mentioned in Note 1 below, we excluded backend work from server-side work.
Also, and this is important, these metrics hold for our single page applications (SPA) projects. I do not have metrics for projects with a more traditional server-side only rendering...but I would guess that these finding do not hold true in such a situation.
If anyone else is willing to share their metrics I would be very interested. I'm always open to seeing the bigger picture in case there are some important facts that I have not factored into my decision making process.
# Notes
(1) This 75% is after segmenting server-side work (i.e. Java application server) from back-end processing (i.e. Kafka, Storm, Hadoop). Obviously, data processing will remain in Java, so we excluded it from our comparison.
(2) KLOC is a poor metric, but it was quick and dirty for us and provided good directional information. Also, all developers (front and back end) are good. So we did not need to adjust for quality of code (i.e. good developers have lower KLOC on average).
I have lots of Clojure experience but little Javascript experience. Still, the Meteor implementation took me half as long and had more features. That was a surprising result for me.
I have found these "breaks" are minor and usually require a few minutes to fix. That said, I also tend to lag behind by a few weeks as other in the community work through the details of the breaking changes.
Meteor Hacks (http://meteorhacks.com) and EventedMind (https://www.eventedmind.com) are two great resources.
Is there a good demo / tutorial that integrates these missing components, either from meteorite libs or well known projects like Angular or Backbone?
You mentioned that some people are already scaling Meteor across multiple servers. Looking at it, round robin load balancing won't work given requests to a server seem to be stateful. How is that being done as of now? Do you guys have any guidance on how to do that?
This will be in the official docs at some point, but for now, Arunoda's blog post about this is a great starting point:
DDP is pretty simple (just a few messages) and the spec for the 'pre1' version of the protocol is in Git: https://github.com/meteor/meteor/blob/devel/packages/livedat...
This is pretty close to version '1' of the protocol that will be locked down in Meteor 1.0: https://trello.com/c/kMkw800Z/53-official-ddp-specification
The idea behind DDP is that HTTP got us three huge benefits: (1) shared tooling (I can write a caching proxy, and anyone with a website can use it); (2) interchangeable parts (your client and your server can be built with totally different languages and frameworks); and (3) easy APIs (I can describe my site's REST-y API to you in a page or two). These days, a lot of sites are moving away from REST/HTTP and toward ad-hoc, custom publishing schemes that run over websockets or a HTTP long-polling transport that is emulating websockets. Since they use these ad-hoc protocols to move around data rather than something like REST, they lose (1), (2), and (3). But all of these ad-hoc protocols are basically isomorphic to each other. DDP is an attempt to nail this kernel of data publishing functionality down semantically to the point that you can get those three benefits back.
As an aside, what's the pressure that's pushing these apps away from REST? The two big ones are:
- They cache data locally and need to keep the cache fresh (say they are apps that run in your browser and never reload the page, or they are native mobile apps), and they don't want to poll for updates. And/or,
- They need to do joins and they care about latency. (When you are loading the news feed, you want to fetch the feed stories, the comments, and the userpic URLs in one round trip, not first request the stories, and then only once you have them request the comments.)
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL), and many modern apps need to perform some form of latency compensation (predicting the outcome of a RPC and simulating it locally to update the client's display while waiting for the server's answer, but then reverting to the authoritative outcome chosen by the server.)
DDP attempts to address these needs and run over a stream transport like websockets, while recovering some of the benefits of standardization that made REST so successful.
DDP seems like a good technology for webapps, but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making. And I believe the lack of understanding of what REST means is preventing people from seeing a bigger picture.
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL)
I've heard that a lot, and I don't doubt that it's true in many cases, but usually it seems more representative of a difficulty in the modeling process.
For example, a transferBalance call certainly doesn't map well to updating any object, but creating a new Transfer (and passing the URLs of both Accounts) would be perfectly RESTful, in my opinion.
Can you tell me more about your vision?
If you want, I can try to form a more cogent reply later, but in a short summary, it would be keeping the current trend of sending structured data, but restoring the principles that made the web great - decoupling code from data, connecting it across services and communities (turning Hypertext into Hyperdata) and aggressively adopting standard formats.
I see a web where I don't go to Hacker News; messages come to me from everywhere, to where I want them to. Where I don't open an issue on Github, but on my personal tracker, which federates with theirs. Where businesses work for consumers, responding to their demands, instead of pushing their crap on us.
This view is mostly based on a mix of the concepts such as the Semantic Web / Linked Data, VRM, the Intention Economy and other ideas.
In the future, I'm super excited about investigating ideas like DDP discovery (asking a DDP server what data and methods it has) and description (having a DDP server publish metadata in some standard format that explains the semantics of the other data that it's publishing in terms of microformats). I think the main place we differ is that I think that these formats should use something more like a relational model rather than a hypermedia model, or a formal knowledge representation model like RDF. Isomorphisms exist between these models, and used properly they are all equally powerful, so really this just a design decision about what will be easiest to implement and adopt. And of course I believe that it has to be server push (websockets) rather than poll (HTTP), and that it has to support server-side joins (doing 1 query rather than N+1 queries, and taking 1 roundtrip rather than 2, to retrieve your newsfeed stories plus the comments on them.) I can't tell from your comment whether you agree with that or not, but like I said, these are two of the main things that motivate DDP.
All of that said, I think that by far the hardest part of the puzzle is the product design and federation issues. In your federated Github and HN, the technical work is the first 10%, the next 90% is finding a user experience for it that makes sense to and is ergonomic for a large number of real users, and the final 90% (really should be more like 9000%) is convincing the industry (read, creating incentives to force the industry) to cooperate with the federation and to resolve all of the political squabbles about advertising, control of the data, licensing, and so on. I mean, Google pulled out of XMPP federation -- all practical evidence suggests that keeping these federation together is wicked hard.
So that's why we're just trying to build the first 10% of that first 10%, which has a very clear use case for building rich browser apps and mobile apps -- and hoping that we will be a stepstool that others such as yourself can stand on to build the full solution that everyone dreams of.
Anyway, if you are sympathetic to this line of argument, I hope you'll check out DDP and see what interesting discovery and self-description features could be built on top of it, or suggest how its semantics could be strengthened!
We do try to improve AV with every Devshop, and we have recently started to bring in AV professionals to help. But honestly, pre-1.0, I think it's way more appropriate for us to be spending our time and money on advancing the framework than on super slick video presentations. In many ways it's more like the Meteor community's time and money.