App Engine JavaScript SDK | Hacker News Reader

App Engine JavaScript SDK

appenginejs.org

74 pointsdanh42 comments

If you're thinking of using the App Engine read this first:

http://highscalability.com/blog/2010/5/26/end-to-end-perform...

But read it well. It seems AppEngine comes out very bad, but it was tested for db transaction speed in a way that every insert locks the entire db (one big entity group). Any AppEngine developer could tell you not to use it like this. More info: http://code.google.com/appengine/docs/python/datastore/keysa...

It would be very interesting to see what the results would be with some more realistic testing.

Well, maybe you're right, but I'll say the following:

1. Unless I'm missing something here the entity group thing is a red herring. Only if the authors of the paper explicitly setup all entities in the datastore as having the same root entity, then this would be an issue. However, since there isn't anything to suggest that they did this, all datastore entities are their own groups (and hence don't lock up the entire db,) so transactions on them would not have had any speed penalty. Please someone correct me if I'm wrong here.

2. Unfortunately their results are consistent with my experience over the last ~8 months running a startup on the App Engine (and now switching over to EC2 over the next few weeks.)

> Any AppEngine developer could tell you not to use it like this

I'd really wish for some AppEngine developer to tell me how's the proper way of designing entity groups. The only advice given is to not exceed a single user's data, otherwise it's too big.

Really, I want to see an example of a request handler that creates data in multiple entities and takes care to not create duplicates.

> It would be very interesting to see what the results would be with some more realistic testing.

IMHO, my experience with AppEngine's datastore has been pretty bad ... the performance varies greatly based on factors you can't control ... i.e. processing something that does well on slow Fridays, may throw the 30 secs timeout error on Mondays.

Also, I don't know how people deploy real-world apps on it ... a non-relational datastore requires custom indexes (i.e. precalculated queries done by building specialized entities out of your quasi-normalized data).

You can't do that easily because of the 30 secs limit that also applies to cron or queue jobs. I tried a workaround by having a queue task that processes 5 rows at a time and then reschedules itself. The stuff was so god-damn awful to debug, and the results so unpredictable (30 secs timeout for 5 rows / concurrency issues) that I felt like crying.

But yeah, it's fine for hosting a blog engine on top of it.

Another issue we're running into now (as a result of having a growing number of users) is that we're noticing that downtime on the App Engine is shockingly bad, and getting worse.

Over the last few weeks, we've seen periods of downtime pretty much daily, and serious downtime (i.e. more than a few minutes) monthly. It's grossly under-reported in the App Engine status console, which shows a euphemistic warning (which in practice might mean that all server requests will return errors.)

Our software is used in the classroom during lectures, and we've had several customers experience system failure due to App Engine downtime. On top of that we've had more than one customer demo go wrong for the same reason.

It's pretty infuriating actually, because 1. you can't do anything other than just sit there and wait it out, and 2. there is no one to talk to at Google. Finally, the problems are only getting worse.

We were going to do a slow migration out of the App Engine over the course of a few months (because we've got a ton of code to rewrite.) At this point we're doing an emergency 2 week migration with all devs pulled from their projects for the task.

I honestly have no idea why these issues aren't getting more mainstream attention, given how many people are using the App Engine. The only thing I can come up with is that there must be tons and tons of tiny non-critical projects, and very few serious startups betting their business on the App Engine.

I was about to start a big project on App Engine, so your comment is worrying.

Anybody else having reliability problems with GAE or is this just an isolated case?

If you're interested in more gory details feel free to email me at mike at tophatmonocle dot com.

For what it's worth, we're an angel funded startup with some extremely bright hackers and we were signing praises about the App Engine until about 3 months in, when we started hitting it's various brick walls.

Oh, and take a look at: http://code.google.com/status/appengine

Click through the various items with green checkmarks to see the actual data that GAE considers "no significant issues." Including fun stuff like 80% datastore error rates.

There might be some problems but it's great to know that Google system administrators (instead of me) are taking care of it. When I hosted my application on EC2 twice I almost lost all my files due to hardware problems. I had to spend all night preparing a new server, not funny. There are no servers to fail in App Engine, I love that!

>I'd really wish for some AppEngine developer to tell me how's the proper way of designing entity groups.

here are some helpful links:

http://code.google.com/appengine/articles/datastore/overview...

http://code.google.com/appengine/articles/scaling/contention...

And a useful video:

http://code.google.com/events/io/2009/sessions/OfflineProces...

You can split it up any way you want to, so long as you don't require more than 1-2 transaction per second in each entity group. On the other hand, lots of tiny entities doesn't perform well either. So it's a pretty severe design constraint.

To get decent performance for batch jobs, I use the remote API with bulk gets and bulk puts and no transactions. But that's useful only for inserts or when you've taken the application offline.

You can't just port your application to App Engine. You have to start from scratch. Google enforces all the right constrains to force you to write scalable apps. The killer feature is that there is no need for system administration or maintenance. No need to setup dns rules to make your email go to the recipients, no need to update apache or linux, no need to setup an xmpp serrver, etc. Plus, you can leverage Google's battle tested infrastructure. The icing of the cake is that you can now use the internet programming language, ie JavaScript, while still having access to the vast collection of Java libraries. A match made in heaven if you ask me!

I'm always amazed at how popular Rhino is vs. how little active development effort it receives. I'm as guilty as anyone of course.

Indeed, Rhino is such an important project but still it does not receive any attention.

It's unfortunate, but it's also the reason we wrote Narwhal, which now means we rarely need to use Rhino at all. A welcome change.

Erm, narwhal actually uses rhino as its default engine. Perhaps you meant to say that you can now target the narwhal API, instead of waiting for rhino to expand its API offerings?

No, I meant what I said. We use narwhal-jsc 95% of the time. Rhino is this unfortunate thing we keep around for compatibilities sake.

The flexibility of being able to switch engines as needed is also a major win in Narwhal. I suspect in the next year we'll end up moving everything to v8, which should prove to be extremely easy to do.

no prob, thanks to CommonJS it was extremely easy to switch from Narwhal to the RingoJS: www.ringojs.org

The killer feature of Rhino for me is the fact that it runs on the JVM, and customers are OK with me using JavaScript, whereas they're not OK with Clojure or so. Too bad the Java integration in Rhino isn't exactly great. (example: you can only use the 0-argument constructor when deriving from a class)

the thing about rhino is, that it already is a mature js implementation. it has the most modern js support of all the engines out there (the version shipping with java is sadly quiet old) and it's battle tested. we have it in production since 8+ years.

i can see how it might seem rhino has "little active development" because we are comparing it to js implementations that just started and of course have more "going on" then the old horse.

The general consensus seems to be that Rhino doesn't really exploit the performance potential of the JVM. Basically, it could probably be a lot faster.

I would like to see support for "use strict" and some more aggressive optimizations. And hopefully, Java7 with support for invokedynamic will be released soon.

some more thoughts on appengineJS here: http://www.gmosx.com/blog/agVnbW9zeHIPCxIHQXJ0aWNsZRipwwEM/o...

Great stuff!

Cool project. I'll never understand why anyone wants to program in JS though.

Besides being an incredibly powerful combination of a functional and an object-oriented language, I just adore the fast redeployment cycle during development. With appenginejs, I test every change in the backend code by simply refreshing my browser. Not jaw-droppping per se, but certainly a breath of fresh air for a long-time "enterprise" Java developer, like me.

Javascript's functional powers are equivalent to or worse than every other popular dynamic language in existence. Speaking of functions, I prefer not to do argument arity checking on every single function I write(I'm looking at you too, perl!), or accidentally assigning global variables through a simple misspelling. I also really hate it if I accidentally invoke a function on a primitive and it silenty fails by returning null. Use 'use strict', you say? How's the implementation of that feature coming these days? I know firefox still doesn't have it. Does v8 yet?

From what I've read, other Prototype-based languages treat Javascript as their ugly step cousin, and I would overall consider it a much inferior OO system to that found in Python, Ruby, CLOS, Perl's Moose, and PLT-Scheme, to name a few.

You can easily get rapid redeployment in any dynamic language. You can even get it on the JVM with Java, as the Play framework is showing us. See http://www.playframework.org/.

If Javascript is your first dynamic language after developing in Java for X years you might think it is neat, but it is really far behind just about every other language out there it directly competes with on the server-side level. It was never designed to be a scalable general purpose programming language and the latest incarnation is still far from reaching that lofty goal.

I used to feel this way. The two people that changed my mind were John Resig and Douglas Crockford. Resig's jQuery project has been, for me, a guide to how JS should be written. Crockford's JS the Good Parts and JSLint have helped as well by providing some solid reference on what to do and, more importantly, what not to do.

Lots of folks only see the DOM and the bad parts when working with JS - these two projects have shown me there's so much more.

Lots of folks only see the DOM and the bad parts when working with JS

Agreed that you can't blame Javascript for the DOM, but there are lots of bad parts aside from that. Default global scope and semioptional semicolons are gigantic WTFs, the with statement should be nuked from orbit, and there are generally lots of ways to write code that appears to be correct but is just waiting to fail in bizarre ways. It's just annoying that we're stuck with Javascript because it's so entrenched, while Python and Ruby offer all of its strengths in much saner packages.

I believe I did say "and the bad parts". I agree - there are lots of bad parts to JS - I often wish it could be remade without the bad parts. However, a lot of these bad parts can be mitigated with sane conventions - that's what Crockford's "Good Parts" is all about. If you haven't read it, I highly suggest it.

I can't speak for Ruby, but in many ways JS is more powerful than Python, especially on the functional side of the house.

jQuery is a great project, but it's definitely NOT how JS should be written. Its goal is to be fast, unobtrusive, and work in most browsers at the expense of comprehensibility.

I'd argue that jQuery embraces a lot of ideas that new JS programmers often overlook - functions as objects for example. jQuery also gives you primitives to perform functional/set based operations.

For me, approaching JS as a functional language has been the key to writing good clean JS. jQuery helped me with this by allowing me to ignore all the psuedo OO stuff in JS and concentrate on the functional features.

Let me clarify: don't think JS is bad or anything. I just personally don't like it as a language.

With so many projects coming out using JS as the primary scripting language, I am probably going to have to suck it up and learn to love it methinks. Just not yet. ;)

1. JS is a great language. 2. You can reuse code on the server and client side, no mental context-switch. 3. Javascript VM's are generally faster than Python VMS. 4. Rhino provides access to the vast collection of Java libraries 5. App Engine is the easiest way to go from a toy application to a industrial strength, production application

1. No it's not. It's mediocre at best. See my previous post.

2. No you can't, not anymore than you can reuse Java code between client and server with GWT. The difference between a browser and a client environment is just so great. NaCl will hopefully provide a way to bridge that gap, but that will work for every language, not just Javascript.

3. PyPy is starting to catch up to v8. This point is irrelevant for the AppEngine, by the way. I doubt Rhino is any faster than Jython.

4. Ever heard of Jython?

5. Ever heard of Jython?

Certain languages get way over-hyped on HN, and Javascript is one of them.

> 1. No it's not. It's mediocre at best. See my previous post.

of course it is

> No you can't, not anymore than you can reuse Java code between client and server with GWT. The difference between a browser and a client environment is just so great.

pure bs, obviously you don't know what you are talking about. I am reusing extensively between server and client, not only for simple stuff like validations, but also to render graphics (have a server side canvas implementation that renders to svg), to perform web crawling (I use a simple version of envjs and jQuery/sizzle on the server side for crawling), i use the same template engine on server side and client side, etc, etc...

> 4. Ever heard of Jython?

Why would I be interested in Jython??!

1. You either didn't see my previous post or have no useful response.

2. Sorry, let me rephrase my point here. You can reuse code, but not all code. Any code you could reuse could also be re-used in another languages using tools like GWT, py2js, j2js, etc, mitigating server-side javascript's advantage here.

As for points 4 and 5, I couldn't care less whether you're interested or not. My point is that javascript has no unique advantage here over most other popular scripting languages, and you made an earlier comparison to Python, hence the Python example.

> 1. You either didn't see my previous post or have no useful response.

Javascript is a great language with some mistakes. a lot of the mistakes (including some you mentioned) are fixed in Ecmascript 5. I programmed in Ruby from 2002 since 2009, and I am happy I switched to JavaScript: I prefer the syntax, I totally love prototipical inheritance, I love to object literal notation, and the use of closures everywhere.

> 2. Sorry, let me rephrase my point here. You can reuse code, but not all code. Any code you could reuse could also be re-used in another languages using tools like GWT, py2js, j2js, etc, mitigating server-side javascript's advantage here.

as I said, pure bs.

> 4.5.

JavaScript has some advantages and some disadvantages to Python. AppengineJS is for people who know JavaScript (just about every web developer) and do not want to learn Python. If you know Python, just stick with the Python SDK.

if you only dislike js syntax, check out coffee-script: http://jashkenas.github.com/coffee-script/

In fact, the syntax is one of the things I prefer in JavaScript (vs for example Python), along with Prototypical inheritance...

Prototypal

yeah, whatever ;-)

Reply on news.ycombinator.com