Google App Engine: Datastore Performance Growing Pains | Hacker News Reader

Google App Engine: Datastore Performance Growing Pains

googleappengine.blogspot.com

33 pointsadora10 comments

I just started writing a GAE open-source community tool for rare disease orgs two weeks ago. I thought it was slow, but wrote it off as a limitation inherent to the Datastore. I moved a lot of things to the Memcache level, mitigating (mostly) the slow queries. That being said, I am glad it was actually a real problem.

I'm also impressed that they are not charging anyone until the problem is solved.

GAE does have some performance surprises even during normal performance periods, so it's good that you're using Memcache etc now.

Couple other ideas:

Try to minimize read effort as much as possible, so e.g. pre-calculate as much as you can and rather optimize for performance than data cost. Data is cheap, so de-normalise where you need speed. Calc offline, so add a task to calculate something and return immediately to the user.

I like the way they're dealing with this. Acknowledge the problem (and actually call it a "problem" instead of handwaving aroung it), and stop charging for the service until they fix it. This is a good pattern to follow if you run a service that ever runs into issues that affect your users.

On the performance issue itself, I actually used AppEngine for a project when it first came out and was astounded at how poorly it performed in general. Any page requiring more than a single "select where id={}" query would regularly throw CPU warnings at you.

I couldn't imagine building a site even as straightforward as HN on top of it. As soon as you had a page needing a dozen queries to construct, you'd run into the wall.

If I'm reading the current status graphs correctly they're regularly seeing 1.5-2.5 second queries and 400ms direct fetches. That seems unbearably bad. Can anyone running mission critical apps on the platform comment?

Yeah, it's really, really slow -- and the ETA is 2 weeks! Although I have some faith it'll be fixed sooner.

We're thinking of porting over to EC2. If anyone knows of any reliable tool that supports this for production, please feel free to shout. Otherwise, we'll probably be building something ourselves...

There's always AppScale. It supports EC2 and I think it's been officially recognized by Google. http://code.google.com/p/appscale/

If you write your own, would you consider open sourcing it?

Thanks for the link! Appscale looks a bit limited for production use, but definitely will play around with it!

Re: open source. We might just brute force it for now. If not, we'll definitely pretty it up at some point and open source it. At the very least we can publish our learnings.

Yeah we've been freaking out about this. We have our app used in the classroom, and have had several sessions disrupted because of this. Needless to say our customers were pissed.

We're in the process of doing an emergency move over to EC2 because of these issues. That being said, I must admit that this is a really decent response on Google's part. Unfortunately the problems go deeper than just these latest problems with datastore.

What are your more permanent worries?

Just wondering, are you porting to mySQL, Cassandra, or HBase? We're currently moving off app engine also, and would love to hear more about your final setup.

Reply on news.ycombinator.com