Why we switched from Google App Engine to EC2 | Hacker News Reader

Why we switched from Google App Engine to EC2

tophatmonocle.com

128 pointsaxiom47 comments

I think it was a mistake for the GAE team to port Django to App Engine. Django is designed for a SQL based worldview and even the crippled version included with the SDK is non-optimal for App Engine deployment.

Since the included version is so bad, people try to get django-app-engine-patch or django-norel working. Those also suck.

There's a mindset that if Django runs, I can keep doing things the Django way.

Nope. Abandon your old frameworks. If you're going to use App Engine, you have to embrace its distributed, scalable way of doing things. Use something made for App Engine like webapp or tipfy.

Yeah, there are problems with App Engine. It has downtime and things slow down. So does every other hosting platform.

App Engine is getting better all the time. I've been working with it for about 6 months, and nearly every month there is a new release with great new features that make developing apps for the platform better. The new 1.4 release has tons of great features that will make working with App Engine easier, like reserved instances and warming requests.

It's also a shame the GAE team didn't pay any attention to any established frameworks that could have worked perfectly fine within GAE's constraints, like Pylons. Shortly after App Engine came out I put significant effort into making Pylons and related libraries work with GAE, but when there were problems on their end it felt like they reluctantly doing me a favor to respond to the bug reports and make changes. It felt like they had an anti-library mindset, even though there was no technical reason a lot of libraries wouldn't work well on App Engine.

"If you're going to use App Engine, you have to embrace its distributed, scalable way of doing things. Use something made for App Engine like webapp or tipfy."

You make a good point. I am not a Python programmer but I have tried a few AppEngine tiny experiments with Django and webapp. webapp was clearly designed for the platform. Similar case for Java: JDO or JPA never seemed like a good fit to the data store, but Objectify seems crafted to add a nice but thin layer on top of the low level APIs.

What's wrong with the Webapp framework? I like it quite a bit -- it's light, fast, and simple, almost like Sinatra but with a bit less syntactic elegance.

I agree that people who approach App Engine for the first time get quite confused about how to make Django "work" on GAE. There are too many options. Even after evaluating, developers may not understand how round-peg/square-hole the problem is.

I like some of the basic infrastructure that Django gives me, so I make use of `use_library()` Django 1.1 on App Engine. It's not "full" Django but I find it more palatable than WebApp and the parts of Django that fit GAE fit it well. Models, etc should be done directly on top of the GAE APIs, in my opinion.

I think Guido's rietveld sample app is a good place to start if you're new to App Engine and want to use (the useful bits of) mainline Django.

Personally I very much like django-nonrel. It gives me pretty much the same constraints as webapp, while still maintaining enough of my old worldview that the culture shock isn't too big.

Yes, you still have to adapt to the AppEngine way of doing things. But the adaptation is much simpler ... and I completely dislike the way webapp does a lot of things judging from the examples Google provides. I like my world nice and full of free-floating functions. Wrapping things into nearly meaningless objects hurts my brain.

Could you please explain what you don't like about Django-nonrel? Why do you believe that webapp or tipfy are better?

IMHO, Django is too slow and bloated. I'm more interested in the frameworks which have a good cold start time.

Cold start times won't matter, anymore, when App Engine 1.4 gets into production - which is soon. Then you have warmup requests and reserved instances. At that point small frameworks will lose all their appeal for non-trivial projects.

Thanks for the nice reply. My first experience with GAE was not good, after that I have never looked at it. I took a look at webapp and tipfy, both look good. I will make some time to give it a try.

There's an interesting detail in the final paragraph: “We pay about 1000x more for EC2 than we did for App Engine”.

One of the rarely-mentioned virtues of App Engine is that it's really, astonishingly, cheap — as well as removing the system administration burden, which is also a cost in either time or money.

For a large company, web application hosting is an insignificant cost, but for an unfunded startup it can be pivotal.

(I first used App Engine in 2009, in answer to a requirement for a simple application that had to be both massively-scalable and run on a close-to-nonexistent budget. It was a perfect fit for that scenario.)

Fully agreed. App Engine is insanely cheap.

We loved this fact when we were first starting out, but after we realized how much time the various limitations were costing us I really really wished they would just let us pay them a few $k per month so the damn thing would just work.

Probably the most serious challenge for the App Engine team has been dealing with the 95%+ of their userbase that will never pay them a cent for their services, but will suck all the life out of their platform.

  * App Engine's TCO is high because of development time
  * EC2's TCO is high because of operations time

Hosting costs are usually not that important (except outliers like file sharing.)

The real question IMO is whether the money saved by App Engine is more than the development cost of the premature scaling you have to do.

I'm curious how this works out? When I look at the App Engine pricing it doesn't look cheap - see:

http://code.google.com/appengine/docs/billing.html

What I see is Storage 0.15/GB/Month, CPU 0.10 / hr, Transfer 0.10/0.12 / GB. This seems basically on a par with EC2 except with EC2 you can get steep discounts for reserved instances or even use micro instances for some things.

So how is it that App Engine comes out so much cheaper?

You only pay for CPU if you really use it and not for "standby".

So if you have 100k requests in one month and each request takes 0.1 seconds you only have 10.000 seconds(or 166.66 hours or 2.7 hours) so you only pay $0.27 for that month. EC2 would bill you for 720 hours(30 days *24 hours). At a rate of $0.10 you would pay $72 instead of the $0.27.

(For this example I'm not taking the free quota into account)

Thanks. If I understand you right, it comes down to that GAE is counting the CPU on a per-request basis while EC2 is charging you per-hour - so if you load is highly variable (as many web startups would be) you get a big bonus because you're not billed for all the time between requests. If your load was heavy and consistent then it would be much more even.

I agree that it would much more even if your load is consistent and heavy, but if it gets too heavy you run in a lot of problems with EC2.

If you max out one instance, you would need to add another instance and as a result of that you would need a third instance as a load balancer. So if we take the numbers from the example I gave above and assume your instance maxes out add 100k requests and you get 110k request your cost would go from $72 to $216 ($72 for the load balancer and 2* $72 for the web/application server). That is 3x more but only an increase in traffic by 10% Of course the same math applies to your database server.

GAE pricing scales linear no matter how big you are and you never have to worry about maxing it out. Of course you have some trade off's because of that, but that is what the article was about.

I guess it's a question how you define heavy traffic. I've read somewhere that Zynga runs 10,000 EC2, at that scale adding another instance wouldn't have a huge impact, but I think if you have less than 15-20 instances (which is still huge) you're cheaper off with GAE.

(Even though it sounds like I'm affiliated to GAE I'm not, I'm going to use EC2 because of the trade off's mention in the original article)

re: very low cost: how true!

The web app I run on AppEngine 24/7 is a low volume site so I never appreciated how much the daily free quota gets you until last summer when I took one of the Google Buzz firehose reading demos and modified it to extract what I wanted from the Buzz firehose (large volume of Buzz, Twitter, etc. social messages). I just used the daily free quota and my temporary app would run for 5 or 6 hours a day before hitting the quota. Lots of bandwidth and CPU use.

That said, I never mind my monthly Amazon AWS bill because its a good value (and, for a long time it was free because of two grants they gave me).

I also would like a plan from Google where I gave them some money every month; this may happen when version 1.4 is released because I am trying to decide between AWS and AppEngine for a new project. I am curious what having three instances always spun up will cost.

App Engine has a billing plan if you want to use more resources than the free quota. http://code.google.com/appengine/docs/billing.html It is a lot cheaper than AWS

The beauty of App Engine is that they bill based on CPU time. EC2 bills you based on wall clock time. Huge difference.

I love AppEngine. It takes you time to unwrap your head from SQL and wrap it around the datastore but that's the only thing you have to overcome, believe me.

I'll tell you my story, a desktop developer gone web, starting with php and cheap hosting from godaddy, spending hundreds a month for my pet projects.

Then I learned GAE was free, so why pay for monthly hosting when you can get it for free for your pet projects? I have hundreds of them so I took the plunge and started to learn python. Damn easy, not going back to php, ever.

Don't waste your time with java unless your corporate overlords dictate it. Python is ten times easier and you can go from zero to ninja in one day. Or less.

I started with the hello world example using webapp, no need for frameworks at all, even if you come from django, drop it. Webapp can do it all with a couple of utilities and the template engine (same as django) in a couple of lines.

You'll be amazed at how simple it is to build an app, of any size, once you master the basics about models, sessions, authentication, etc.

You don't need sql connections, chmod configs, apache htaccess, and stuff like that. Just get some data and render it with a template. Fucking easy!

Don't believe me? You lose, not me. Give it a try, you'll be surprised.

I am a happy GAE user and my next startup (which I hope will have a million visits a day) runs on AppEngine and no worries at all about handling them all. Google will be there for me.

GAE 1.4 will come with Channel API, an easy way to send messages back and forth, we just got multitenancy, longer task queues and cron jobs, xmpp, etc.

You get it all at no cost, you pay only a couple of dollars when you need it, when you get slashdotted, dugg or techcrunched. No sweat, no sysadmins running like headless chickens, no server rooms burning.

And that, my friend, that is peace of mind.

I really appreciate reading stories like this. For budding startups like mine, we need every available resource at our disposal on technology choices. I also think articles like this will encourage Google's engineers to keep making GAE a better platform. AFAIK, Google is also notorious for eating their own dog food.

I was a bit taken back by the criticism on the last app-engine article. Quite frankly, sometimes you don't realize you the limitations until you hit them despite what the manual says. ("Every one's job is easy but your own.") If you haven't used GAE, you don't realize how easy it is to just get up and get going. It's absurdly simple. However, when you hit limitations you might encounter the sunk-cost fallacy, and that might propel you to keep working around its limitations.

I think Google sort of deserves some negative press after the major datastore problems of late (now fixed), but it's not reasonable to criticize the app engine platform itself. The so-called limitations are just reasonable trade-offs, and frankly they are less annoying than many that teams end up self-imposing in order to work around unexpected bottlenecks in their app.

For what it is worth, we've had 99.998% uptime for static files on appengine - not using datastore or any other features (about 11 minutes of legitimate outages the past year).

During the same time period we've had 100% uptime for m1.small servers in EC2's US East and West regions, 99.989% uptime for EU West and 99.997% for APAC.

http://cloudharmony.com/status

Anyone have a happy story of GAE availability in particular?

I am not so bothered and actually appreciate the programming model. Actually reading the story, seems like some of their architectural patterns may have run contrary to asynchronous intent.

98% availability is a deal breaker for most betting thyeir business and image.

I'll pile on this.

I'm a huge EC2 zealot (and customer) - but I'm also intrigued by the concepts behind App Engine. However, there has been a spate of negative posts such as these around App Engine.

I also would like to hear from someone who is successfully using App Engine for a large scale project, and what their lessons learned, best practices are.

I'm one of those "right tool for the right job" type of person so I'd love to see where App engine might fit in some of my business' architecture.

I run two profitable businesses on app engine. The occasional downtime is a tolerable exchange for the numerous benefits. (I also use ec2 for computationally expensive jobs. The ec2 instances are started and stopped by gae.)

Bottom line for me is this: on gae one dude, sans team, can be dangerous.

Interesting perspective. I'm a "one dude" on EC2 and geeze I feel dangerous at times.... I haven't had any spectacular failures, but I've certainly had some close calls.

I'd love to hear more about how you've leveraged GAE.

Sure, my email is in my profile. The lack of sysadmin is key for me. I focus on the app with gae, not updating BLANK.

I love app engine. The datastore issues have been the only negative aspect of it. Yes, SSL support and naked domain support would be nice, but it's a very nice platform to develop on.

Sure, the first time I wrote a toy app on app engine I ran into the so-called "limitations" b/c I didn't understand the way the platform is designed. But once you get it, it's really just a series of quite reasonable tradeoffs that you make in order to get effortless scaling.

Yes, the platform could be a bit more reliable and in some areas a bit more mature, and yes, one of my projects almost failed b/c of the datastore issues over the last few months (angry users, etc.) But we made it through, and as of Nov 6, I get very few errors and things just work again.

Is App Engine ready for every prime time app you might want to build? No. But it's definitely a platform to know about and to watch carefully.

One other feature to note is the app versioning capabilities. You can simultaneously run lots of different versions of an app... useless at first, but extremely valuable later when you want to get aggressive and roll out updates gradually, etc.

And, there is ZERO sysadmin work and NONE of the usual web server setup yak shaving.

Walk Score makes heavy use of GAE across many of its products, and is very happy with it. Despite the occasional downtime, the price/performance is on par with needs.

Walk Score uses AWS where it needs heavy compute back-ends, but all front-ends (including API "front-ends") are GAE-based.

I've been developing with jruby on GAE for about a year. http://code.google.com/p/appengine-jruby/ There has been a good deal to learn, and some changes to adapt to along the way for the jruby library - which is to be expected. My sites are mostly noncommercial and don't get a ton of traffic, so their generous free/cheap plan was an incentive I couldn't pass up. Alternatively you can do very little for free on Heroku (the other 'free' hosting option for Ruby).

It's been a good experience overall, but this last month I have spent weeks pulling my hair out trying to do a join. I have a work-a-round in place now, so hopefully it will be smooth sailing from here on.

If I do run into performance issues I may have to switch, but then it won't be to hard to migrate and I've still had a free development environment which has pushed me to learn and cost me nothing.

The thing people "forget" is that the further up the stack you move, the more support is required because there's less and less the customer can do to diagnose a problem.

By "people", are you referring to Google? If so, I agree -- not having technical support available is a deal breaker for a system of that scale, especially if outages are as prevalent as the author claims.

I think he means both Google and their customers.

Google should get it, in that they need to offer support if they want to run this kind of business. But customers should also know that Google doesn't do support, and they will need support, so GAE is probably not a good fit.

If that's right, I agree as well.

Wow, these stories about the horrors of GAE are a little bit scary for me. I was starting to develop a project with a friend in our spare time, and we decided to go with GAE. We've only gotten as far as the end of the tutorial and are reading the documentation for GAE, but I have a question:

As a college student who is looking to build a project on the side as a means for experience and a way to handle an issue we have, should I look into something other than GAE? I really do not intend to make our idea a startup or anything. Any help would be appreciated!

Go with GAE. There's a lot less overhead with things you really don't want to be dealing with.

With GAE would you suggest I use a web framework like webapp or the built in Django?

Is downtime associated with how complex the GAE app is? I'm planning to host primarily static files and I'm wondering if downtime is an issue at all in this case.

Is it getting fashionable these days to blame Google App Engine? I am reading lots of horror stories about GAE!!

I'm a developer at Top Hat Monocle.

There have been a number of posts attacking App Engine recently, and a lot of good discussions have come out of them. The two things issues that seem to get brought up most are app design and reliability.

- Reliability: App Engine reliability used to be horrible, but it's getting better.

GAE was the first service I ever developed for where reliability was an issue. I've hosted apps on Linode, Slicehost, EC2, and my experience with those companies caused me to dismiss claims of downtime and the importance of an SLA.

Nothing sucks more then seeing your system go down for days at a time, and not being able to do anything about it. I'm glad that GAE is addressing this issue, but I'd be wary of running anything commercial on their services without buying a business plan.

- App Design: Pretty much everyone agrees that you have to write applications the way GAE wants you to, or you'll fail.

The built in GAE Python library is great for writing simple, highly scalable websites, but it's too limited to develop complex applications on. Our application has over 20KLOC and uses numerous third party libraries. In practice, we found that it was difficult and time-consuming to write our application without turning to a more established framework like Django. We were also concerned with the vendor lock-in that would be caused by building on library that could only be run on GAE. Projects like TyphoonAE (http://code.google.com/p/typhoonae/) are starting to make this less of an issue, but they aren't mature enough to provide an exit strategy.

We built our application on appenginepatch, a third party port of Django to GAE. Django was not designed to run on a non-relational DB, and GAE was not designed to handle a heavy framework like Django. Our decision to build things our application on Django ultimately lead to most of our problems, but eventually saved our skin when we decided to switch away from GAE.

I'm really curious about where Django helps with this? Is it really that hard to use 3rd party libs with webapp?

Well, my motivation in writing the blog was in response to the people who replied to the last HN story on App Engine saying the author was clueless and just didn't know what he was doing. For what it's worth, there are a lot of other very bright people that have had the same issues and came to the same conclusion.

I think it's just that developing for GAE is just so different from developing on a typical LAMP stack that there inevitably a fairly steep learning curve. That combined with the (well documented) limitations you just have to find new ways of solving problems. The fact that this author thinks that other article was a "good read" doesn't inspire a lot of confidence in what this guy has to say.

I don't know if developing on the App Engine is still as bad as it was before. Maybe it's changed in the last few months that we've been on EC2

From what I've heard, things have dramatically improved from how they were a few months ago.

It's not a new trend. App Engine has had issues for a long time. Here's an article I wrote early last year about some of its problems and proposed solutions that the GAE team could use to fix them:

* http://www.espians.com/why-app-engine-is-not-appropriate-for...

But, despite the various problems, I still use App Engine for various applications. There are few platforms that can even compare to its power and simplicity. And good luck finding a NoSQL datastore which offers the power and scalability of App Engine's one.

In fact, if there was a decent open source alternative to the datastore, I'd switch away almost immediately.

For a much better solution, give Djangy a try: http://www.djangy.com - it's heroku for django (and eventually other WSGI frameworks)

Reply on news.ycombinator.com