Goodbye, Google App Engine | Hacker News Reader

Goodbye, Google App Engine

carlosble.com

362 pointsvrruiz95 comments

Carlos Ble says: "Choosing GAE as the platform four our project is a mistake which cost I estimate in about 15,000€. Considering it's been my money, it is a "bit" painful."

Open articleView on HN

Does no one read the documentation before deciding to use a platform anymore?

App Engine supports Python 2.5. The Python interpreter runs in a secured "sandbox" environment to isolate your application for service and security. The interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C code; it is a "pure" Python environment.

At the top of the FIRST page of documentation: http://code.google.com/appengine/docs/python/overview.html

Google Apps domains do not currently support HTTPS. HTTPS support is limited to apps accessed via .appspot.com domains. Accessing an HTTPS URL on a Google Apps domain will return a "host not found" error, and accessing a URL whose handler only accepts HTTPS (see below) using HTTP will return an HTTP 403 "Forbidden" error. You can link to an HTTPS URL with the .appspot.com domain for secure features, and use the Apps domain and HTTP for the rest of the site.

HIGHLIGHTED on http://code.google.com/appengine/docs/python/config/appconfi...

A request handler has a limited amount of time to generate and return a response to a request, typically around 30 seconds. Once the deadline has been reached, the request handler is interrupted.

and

While a request can take as long as 30 seconds to respond, App Engine is optimized for applications with short-lived requests, typically those that take a few hundred milliseconds. An efficient app responds quickly for the majority of requests. An app that doesn't will not scale well with App Engine's infrastructure.

http://code.google.com/appengine/docs/python/runtime.html#Th...

I could go on and on.. reading this I see "I wasted 15000€ by not reading the documentation"

I usually think the title technical architect is a bit stupid (my business card says I'm one, so I can say that) but this guy needs a good technical architect to make platform decisions prior to wasting that much money

Agreed, absolutely. Before we settled on GAE, I spent time playing with it and wrote a few nontrivial test applications to get a good feel for it. We haven't been surprised by anything on the platform, except possibly the spurious 500 errors that were happening way too often in September (and have been fixed by the latest rounds of maintenance)

We made the trade of zero platform maintenance and near-free scaling and accepted the limits placed on what we can do. Many of those limits are being lifted each month as well. I can say for sure that there are a number of items mentioned in this post that have fixes in the works right now.

I will say that choosing GAE let us ship a project much faster than we could have without it and with much less manpower. The time we spend working around limits is easily made up by not having to do IT work.

In the comments he says that the restrictions were not THE problem, THE problem was the instability. List of restrictions is kind of reminder of what he had to cope with, and wasted money on, until it turned out that GAE doesn't work properly.

In my experience "instability" with GAE means you are pushing it beyond where it wants to go. If you a have code that just completes within the timeout, then guess what? When the system is under load it isn't going to complete.

You can use all kinds of techniques to protect your application from things like variable datastore performance. For example, nowdays I usually use a pattern where user-facing servlets only ever read from the datastore - all updates go via a taskqueue, which means that even if the datastore is in maintenance mode the updates will eventually be applied.

GAE isn't for everyone, but the first step to getting the most out of it is accepting it isn't anything like a typical frontend plus database architecture.

Late September was bad for the datastore. We were seeing latency on simple queries vary from 50ms up to 5s. These were simple properly-value queries as well. This was fixed in early October I think, and a recent bit of maintenance brought latencies down even further. The datastore is now extremely fast and rarely errors out.

We (http://gri.pe) are still happy on AppEngine, even though September was rough.

Very true. The datastore unreliability was a real nightmare. I was getting periods of 60 seconds where it was unusable...

But it does appear to be fixed now.

The Datastore problems have indeed been fixed and things have been smooth since the fix.

" If you a have code that just completes within the timeout, then guess what? When the system is under load it isn't going to complete."

Unless the platform scales as it should, which is the whole point of accepting all the limitations. If there are tight and hard limits on request timeout, then there must be equally stringent guarantees provided by the underlying platform. "We're working on it" is just not enough under these conditions.

I wish I could upvote this more. Implementation was a 'Square peg, round hole' architecturally.

I'm glad you wrote this comment so that I wouldn't have to. Every comment he made about the system and it's limitations were documented (except the 500's).

I think the author is right. GAE is not suitable for doing serious business.

"developing on GAE introduced such a design complexity that working around it pushes us 5 months behind schedule"

This is the core of all their problems. It is a mindset incompatibility between these app designers and GAE.

The GAE APIs and rules are actually pretty simple and well defined. It works really well, but only if you work WITH those rules. You have to adopt the GAE application design philosophy.

If you don't, and if you work AGAINST the rules and best practices set by GAE then you are in trouble. Big trouble. This is what happened here.

I understand this is easy to say afterwards. And you can't really blame them for finding out the hard way.

Note that the same applies to for example all the great services that Amazon Web Services provides; they only work if you build your apps with the Amazon specific design approach in mind. Things like eventual consistency, expect things to fail, don't do large amounts of work in single jobs. Etc. Etc.

These appoaches suck more or less if you come from a 'total control over a bunch of machines' background. But they are so needed to scale.

Learning to do things the App Engine way frustrated me at first, too, until I realised in a very roundabout way it was - as you say - teaching me how to engineer for scale.

Exactly, the blog post's author doesn't seem to have realized that the limitations are simply design tradeoffs.

That is exactly our experience with the development of Metric Mail.

Yes, there are limitations. And most of them might get you upset first, but if you "work with the rules" it is a fantastic platform.

Metric Mail has been running absolutely fine since its launch in August, it has never failed us. Even under severe traffic it scaled without any problems.

We knew from the beginning what the limitations are and this way there were no big suprises during the development.

There is less difference in expectations from AWS though, probably because it's lower-level.

I just launched my app 4 days ago (threewiki.com) and its backend is using GAE (Python + Tornado). On the second day, it was featured on TheNextWeb front page and the server got TONS of traffic. It scared me a bit, and I quickly change the daily max quota to $10. Anyways, GAE handles everything very gracefully. Over 70% of our users use either Facebook or Twitter to login and it hasn't been a problem at all.

I always think if your site can't be host by GAE, then it's probably not very scalable at the first place. I agree it might be better to host the data-processing end on EC2 or elsewhere if it's intense. Else in terms of the "View" part of your project, I wouldn't give it another thought for using GAE again.

Congrats on the highlight by CreativeApps!

http://www.creativeapplications.net/games/3-degrees-of-wikip...

Thanks. Did you also posted on Reddit? Your name is so familiar.

threewiki.com is very tastefully done - good job!

"""I always think if your site can't be host by GAE, then it's probably not very scalable at the first place. I"""

Well, your site is basically one static page. Do you really feel qualified discussing GAE?

its backend is hosted by GAE

So, all of the dynamic backend code powering the app is running on Google App Engine.

Of course I do. The ENTIRE game logistics runs on GAE, the iPhone stores zero data except a user key.

It sounds like a lot of his problems stem from the use of Django. I've tried it before, and believe me, Django absolutely sucks on Appengine.

First off, you have a full featured framework which was designed for SQL relational databases. Many of Django's features either have to be given up, or are monkey-patched beyond belief to get partial functionality. Not to mention quite a few Django apps use database features which are simply not supported by BigTable.

Secondly, Django is not exactly the smallest framework, so loading time can be quite expensive and will be tacked on to every cold start.

All that being said, I've had good success with the tornado framework. It's fast, well written, and thoughtfully designed. Check out my profile if you want to see some examples of apps written with tornado + GAE.

It's very telling that the author says they ported the whole site to a LAMP-type stack in a week - it could only be possible to do that if the architecture were built around that.

I've been using Django on GAE for a while now and it's working pretty well.

There's a port called django-nonrel, which is specifically designed to work on nonrelational databases. So yes, you can't do fancy joins and stuff, but hey, most of the time you can work around that; but it's still great to be using the ORM just as you are used to it in 90% of the cases.

Although I do agree with the OP that all the rules GAE imposes can be a pain in the arse a lot of the time I really really love them. It pushes me to design my application well and to make it able to handle the load I'm expecting to be handling in a foreseeable future. If I didn't scaling even up to a few thousand users could get dicey.

There are a lot of apologists here, and they're all missing the point. Google App Engine has a large number of random issues and limitations which, while they each individually seem like they ought not to matter, add up to a substantial risk for a project running on it. I'm currently working on a project with a web component, and considered GAE. I chose not to use it, because some of the stuff I encountered in the documentation is absolutely terrifying. You're supposed to handle exceptions from the datastore? Really? The best you can possibly do is retry, but if that was going to work the library would be doing it automatically... so I guess it's saying that your app might just randomly fail sometimes. No https with a domain? I guess ever taking credit card numbers is off the table. Thirty second limit per request? Sounds reasonable, except for all the complaints about the app engine taking thirty seconds just to load the Python interpreter and compile their code. That sort of shit will sink a project; until App Engine has had a lot more time to sort out its issues and mature, I want no part of it.

I'm not an App Engine apologist, although I'm seriously evaluating it for my current project. That said, could you please clarify the following points:

1. "You're supposed to handle exceptions from the datastore? Really?" — Are you saying you don't handle exceptions when your regular relational database throws them? I've seen Sybase and MySQL blow up with depressing regularity, and have had to write exception-recovery code for them.

2. "No https with a domain?" — What stops you from making https://<your-app>.appspot.com the handler for forms which read sensitive information? You can submit the form with XHR, have the server send back a structured response (perhaps as JSON), and deal with it with JavaScript on the client. I admit that it's annoying (you need different form submit targets in development and production), but it shouldn't be a deal-breaker unless you need your entire site to use SSL.

(If you do need your entire site to use SSL, then the appspot domain limitation obviously sucks. If that's the case though, then you're probably doing something with finance, and the datastore's transaction semantics probably cannot work for you at all.)

Re: https with a different domain. XHR doesn't let you access a different domain, you'd have to use JSONP which won't let you trap errors easily. And if I weren't sitting at a page served by HTTPS, I would not enter credit card information, and most web users are trained to look for the padlock icon and all that.

Google Checkout Gadget aside, I find it interesting that this is the closest thing Google has to offer for an individual or business to build a hosted online e-commerce presence (https exceptions aside).

Compare that, in terms of functionality and revenue generation, to Yahoo! Store (Viaweb), and it seems like Google really has its priorities with App Engine askew.

Obviously, the scope of Viaweb's functionality is much, much narrower than that of GAE, but what is the purpose of GAE from Google's perspective?

Surely, it is not to create bitchfests for upset developers on tech blogs. It doesn't seem like revenue generation, nor developer satisfaction is a priority, either.

It's generally not that hard to make it work well... though the datastore issues over the past few months were a real nightmare and could have (and probably did) sink a few businesses.

That said, it's a beta product.

"""That said, it's a beta product."""

It always is with Google. When exactly do you expect the "rc"?

A useful article for people to read before using AppEngine. I only use AppEngine for my own projects, so far no jobs for customers.

I was aware of most of the limitations of AppEngine that the author of the article mentions after just a few hours of experimenting with AppEngine. Now, AppEngine now no longer gives me many problems.

I think the lesson is to do a lot of experiments before committing to technologies.

I don't use the Python SDK. Most of what I have done has been using Java (but with small Clojure and JRuby experiments). One thing that helped was to start using Objectify instead of JDO (as an example).

Can definitely advocate Objectify. It's much nicer to work with and if you're working with Google Web Toolkit it is also GWT-serializable, so you can use the same objects in your DAOs and GWT UI.

Agreed.

At this point, I think using JDO on App Engine only makes sense if you're porting an existing JDO application to App Engine.

If you're starting from scratch, Objectify is the way to go.

All the points he makes are valid - and stem from the fact that GAE is designed to be 'infinitely scalable'. Because of this, it forces you into a number of design patterns that facilitate distributed, scalable software.

This can be a good thing, if you know scalability is going to be a killer feature in the near future. It can also a real pain in the ass if it's more important to simply get something off the ground quickly and see if it has market traction, and you don't want abandon the convenient but difficult to scale practices like long running processes and JOINs. In my experience, most startups fall squarely into this latter camp. Scalability is a nice problem to have for most of us.

AppEngine for Business now has a hosted SQL mode, which presumably uses a less scalable but ACID compliant alternative to the standard GAE data store (disclaimer - I haven't used it). Since he's already throwing down some serious coin on his app on GAE it might be worth investigating that before abandoning the platform completely.

The article mostly describes obvious, well documented limitations of GAE. It's really the story of a team that needed a saw, picked a hammer, and spent months trying to get it to cut wood.

This is a pretty common pattern in software, so it might be more interesting to write an article about why they chose the wrong technology and how they stuck with it even with it was clear it wasn't built to do what they needed.

While I feel the OP's pain in terms of difficulty working with the limitations of AppEngine, what he's not mentioning is the collection of benefits you get from AppEngine over other services. We've found that once you work out a solution to the datastore and request timeout issues, you have a remarkably robust and scalable system for free (or at least out-of-the-box).

With AppEngine, I've never had to migrate a database schema, build a load-balancer, hire a fulltime sysadmin, or even pay for servers that arn't receiving traffic. I don't have to set up a large-scale deployment system, nor spin up a new database server when traffic gets too heavy. AppEngine so far has been remarkably cheap (we're starting to bring in more customers however, so we'll see how long this lasts).

Many of the challenges he mentions come down to thinking about writing a webapp with a longer-term vision in mind. Datastore limitations crop up when you outgrow your first datastore in a standard system; in AppEngine they're properly enumerated and dealt with from day 1. Likewise long-running connections become very tricky to deal with with lots of traffic... this point is a little harder to argue with the recent popularity of asynchronous-io servers, but I think Google is working hard on these limitations. SSL is just annoying; we've had to deal with this by adding an SSL proxy until Google adds SSL support -- but it sounds like Google is pretty close to solving this one (it's been promised by end of year).

Also, AppEngine is written in a very high-level way; should you reach a point where AppEngine no longer makes sense, it is amazingly easy to transition over to another system (as the OP apparently found out; I would give more credit to the design patterns inherent in the AppEngine APIs than 'TDD driven development'). Tornado, webpy, etc have virtually the same interface as AppEngine's webapp framework.

There are definitely tradeoffs when choosing AppEngine as a production backend right now, and its certainly not the right solution for every problem... but for many people, us included, its been a pretty large net benefit for our startup. Google is actively improving the system, and I expect many of these problems will go away in the next 6 months or so.

A few of these points are just ridiculous:

1) If you want "SQL and Joins", use SQL. This is like complaining that you can't play Halo on Linux.

1A) There isn't full text search. If you need full text search, use a system with full text search as a feature.

2) Some of the points are out of date (or will be out of date soon). The 30 second limit for cron jobs will be 10 minutes after the next release. As noted, the 1000 results per query limit is gone already.

3) Anything can fail. If you assume your own system won't fail, you're going to be in worse shape later.

4) What objects would you cache that are >1MB anyhow? In almost any case, you'd be better off caching it as multiple, smaller objects.

I've gotten full text search working using techniques similar to those described here:

http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-A...

This type of full-text search works on a small dataset or a dataset which doesn't have a wide variety of data. It uses the internal merge-join functionality of AppEngine, which as the name suggests takes two queries and joins them together. Problem is the the merge join has a very small time limit. I forget the exact timeout (it's undocumented), but it's something 500 milliseconds.

The problem we ran into with the merge join functionality was the following:

Let's say you're searching for "lcd monitor", your code could do a search for lcd and monitor then merge the result (select * from ngrams where ngram in ['lcd', 'monitor']). There are many lcd monitors so the merge join will find 1000 results very quickly.

Let's say you search for "dell monitor". Unlike the previous search, there aren't many dell monitors but there are lots of dell products and lots of monitors. Your merge join will timeout because there isn't enough time to perform a query for dell and another for monitor then merge the results because of the internal merge-join limitations.

Also, it was VERY expensive to index every document (our data is in a constant flux) so we decided to use a different solution.

Not sure if this will solve your problem, but have you looked at GAELucene? http://code.google.com/p/gaelucene/

I find it amusing that Google doesn't have a full-text search engine. I seem to remember them having code to do that...somewhere. Hmm.

We have a product built on app engine, sure we've had our share of challenges but the benefits far outweigh them. To name a few benefits: A developer staff of 1 can focus on the application code and not the infrastructure details. Fast image serving auto-scales all your images to any size, and serves them off the fast servers used by Picasa. Versions can be used for testing multiple branches of code on production servers seamlessly.

#1 Has never been an issue for us #3 Is incorrect with the new task queue upgrades #6 We have a full-text system working just fine #7 Is a benefit when working with a distributed datastore #8 DB performance after the recent updates has been stunning #10 So they badly designed their queries and blame app engine? #11 Is flat out incorrect #12 What database is immune to failure? Would love to know

App Engine doesn't do everything, and no one is claiming it does. We have a secondary VPS we offload certain image processing tasks for example. But what it does do is extremely powerful from a develop perspective, and the application-centric model, like heroku or engine yard, is where things are headed. I would much rather leave the server and scaling issues to the experts so I can spend time improving my application.

I feel your pain, brother. I went through all of this a few years back. AppEngine was still pretty new back then, so it all seemed like things that they would fix before long. But it sure didn't feel ready for production.

I'm actually quite surprised that all those same limitations are still in place after all this time. I guess if I took a minute I could come up with an issue or two I had back then that has been fixed since, but his list of show stoppers are all things that people were complaining about, and that Google gave the impression of being on top of.

I agree that GAE has its share of problems. However, many of the complaints listed are about limits and limitations, and can be found in the documentation; at least points #1, 3, 4, 5, 6, 7, 11 and 11.

I developed an app with GAE about 2 years ago, and ran into many of the same problems (although some of the limits were probably lower then). Fortunately I could work around them, and the app wasn't used by tons of users anyway. I can see how it would be a serious problem otherwise, though.

We love AppEngine for many of the reasons which he gives. Because of its limitations, the platform makes you think and write code with proper design patterns in mind. If you just start to write code without thinking about it beforehand, yes you won't like AppEngine and it's not for you. It'll cost you LOTS of money and won't perform very well. On the other hand, by writing code designed for AppEngine we've been able to reduce our costs by several thousand $$ a month. Also a side benefit is that we migrated one of our IT roles to a development role, which means that we're able to iterate faster.

Granted we don't have a need for SSL and not being able to use C libraries in python has caused us many hours "pain", but compared to the alternative for a small company like we have, it's well worth it.

My biggest issue with AppEngine is that there's no full-text index functionality, and there's no way to create your own. We've tried everything, and nothing works if you have millions of documents like we have. Our search is still external to AppEngine but we're hoping that Google will do something about it sooner or later.

"Our search is still external to AppEngine but we're hoping that Google will do something about it sooner or later."

Certainly ironic for a Google platform to be so bad at search.

The biggest problem for me with GAE is cost. Currently I'm paying around $150 per week for around 50k daily active users (~6 million requests per day).

I'm almost sure, that i can run the same amount of traffic from $100/month dedicated server.

This is what I've never understood about GAE.

If you're small, GAE is free, but then, you could host anywhere for peanuts - just buy a linode or a small EC2 instance, it doesn't really matter.

Once your site becomes big, cost is going to matter and GAE is as expensive as anyone else, last time I looked, quite a bit more expensive.

So there is really not much advantage, it is only free when it doesn't matter, and when it does you're going to start looking for the exit pretty quickly, not just because of cost but because GAE imposes all kinds of limits which may make dealing with your problems much much harder.

I'm curious how the billing breaks down for your app. Is it mostly CPU quota? Or bandwidth? etc.

For me it's CPU and bandwidth (it's lots of real-time image transformations). So if you look at GAE it's just about the same as EC2 (0.10 / hr for CPU, 0.10/0.12 GB for data, 0.15/GB for storage), but there is no "reserved instances" discount which is really significant (unless I'm unaware of it). On balance it's comparable to EC2 but you have way less flexibility.

1) It's hard to move without rewriting the code. 2) It's hard to get a consistent data snapshot without taking your service off for several hours.

I'm paying a similar amount for similar traffic, and I do recall it was a bit cheaper when I had my own server. However I also recall wasting a looot of time configuring and maintaining that server. I'm willing to spend some to save time.

Can you share general numbers on this? I'm interested in which quotas you're spending money on the most.

This is for yesterday.

Resource Used Free Billable Charge

CPU Time: $0.10/CPU hour 46.03 6.50 39.53 $3.96

Bandwidth Out: $0.12/GByte 2.54 1.00 1.54 $0.19

Bandwidth In: $0.10/GByte 0.76 1.00 0.00 $0.00

Stored Data: $0.005/GByte-day 750.78 1.00 749.78 $3.75

Recipients Emailed: $0.0001/Email 0.00 2,000.00 0.00 $0.00

Total: $7.90

Thank you.

This hasn't been my experience.

Yes, existing techniques for full text search works or things like geolocation queries won't work but there are other[1] techniques[2] that work just as well; it's just not the sql way. Basically, support for multiple set membership queries against a list of tags stored with entities is extremely powerful and if you index properly, you can do a lot of cool things [3]. Plus, you can do datastore queries in parallel [4], which means you don't have to denormalize as much as you think; just parallelize and memcache results; e.g for a complicated front page, you can fetch different types of content in parallel.

The local server behaves remarkably the same as the deployed server, it's quite rare I find a situation where something behaves differently in production. the entire datastore can be tested locally, including complicated schemas / indices / queries in fast running unit tests. This means when I do need to do something fancy with the datastore, I can fully test it with unit tests and be confident it will work when deployed.

Long running tasks can always be broken up using the task queue. the limit will soon be 10 minutes for individual tasks and cron jobs [5]

I agree that cold start is a huge issue, but looks like it is being addressed in the 1.4 release [5] where you can pay for 3 reserve instances at all ties. Lack of support for https on your domain definitely sucks too, but I don't see how he wouldn't have been aware of that before going with GAE.

Finally, there are a number of things that are a huge time / money savers: - really easy deployment process including support for multiple versions. This let's you have staging instances and quickly roll back to a previous version if there are any problems - a nice admin console with a number of tools, including comprehensive access to logs that are coherent across all instances - some really nice libraries for examining performance of datastore queries and other api calls [6] and getting daily email reports of any exceptions [7]. these are built using hooks available to you in case you want to build something similar (for instance I used hooks to have regression tests on the number of datastore queries each page requires). - the services and apis made available are really nice. for instance, the image hosting infrastructure that provides fast access to different sizes for a stored picture based on a url is pretty slick; they basically opened up the same infrastructure that is used by picasaweb to app engine users - virtually no hosting costs until you get a lot of traffic. thousands of daily visitors is still in the free range

That said, my biggest outstanding gripes:

- cold start problem (until 1.4 is out)

- datastore latency spikes sometimes. this has gotten a lot better in the past few weeks, but I'll still have this gripe until I see it more consistent for a couple months

- no support for incoming emails with attachments > 1mb (makes incoming photos from smart phones impossible since they are usually > 3Mp these days)

- no support for long polling (upcoming channel API seems to be more for chat rooms than for general purpose server push) [8]

[1] http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-A...

[2] http://fluffybunnysoftware.com/node/8

[3] http://code.google.com/events/io/sessions/BuildingScalableCo...

[4] http://code.google.com/p/asynctools

[5] http://groups.google.com/group/google-appengine/browse_threa...

[6] http://googleappengine.blogspot.com/2010/03/easy-performance...

[7]http://code.google.com/appengine/articles/python/recording_e...

[8] http://bitshaq.com/2010/09/01/sneak-peak-gae-channel-api

Cool! People know about asynctools and are using it! That was developed by a friend of mine at a previous job. Glad to see it's useful.

yeah, I've written a wrapper around it so we can post process the results and memcache that, but asynctools has been really useful!

To your point about incoming mail with attachments > 1mb. The limit is actually 10mb, so you should have no problem mailing photos from smart phones.

http://code.google.com/appengine/docs/python/mail/overview.h... http://code.google.com/appengine/docs/java/mail/overview.htm...

"the image hosting infrastructure that provides fast access to different sizes for a stored picture based on a url is pretty slick"

Any links? I am saving both pics and thumbs and that info would help me a lot.

yeah, check out "get_serving_url" on:

http://code.google.com/appengine/docs/python/images/function...

the next version will allow you to pay to keep three copies of your app loaded and ready to go at all times. that should eliminate the insanely slow load times for apps that don't get constant traffic (at a price yet to be determined).

google is also adding sql database capabilities to the platform soon.

and google apps for business will eventually let you talk https on your own domain (at a price yet to be determined).

i suspect they're about 6 or 8 months from becoming a solid solution to many problems.

Do you have any references to the claims about GAE offering sql databases support? Is it something that you predict or is it something they made a commitment to?

http://googlecode.blogspot.com/2010/05/announcing-google-app...

See under the section "Enterprise Features".

Note that this is a page about GAE for business, so I wouldn't be surprised if SQL databases require a billable account similar to the Blobstore - don't think anything has been explicitly stated one way or another as yet though.

It is available in "App Engine for Business" which is currently in preview. http://code.google.com/appengine/business/

#11 isn't true. Since a recent update, you can now retrieve more than 1000 results in a single call.

No, it is true. After the recent update you could use a combination of queries and offsets to retrieve more than 1000 results. To get 1001 results or more you need multiple calls.

I am sure you are mistaken.

The reason I'm sure is that I just tried it:

  entities = data.MyModel.all().fetch(1010)
  print len(entities) # Prints 1010

Then I thought that perhaps you meant the limitation still exists in the low-level datastore API — and it's worked around by the Model interface making multiple calls to the low-level API — but that's not true either:

  from google.appengine.api import datastore
  entities = datastore.Query("MyModel").Get(1010)
  print len(entities) # Still prints 1010

So: what do you mean?

I am mistaken. I apologize.

No, you can set limit in a fetch to whatever you please and the datastore will return all those results to you ... the 1000 limit was removed.

http://code.google.com/appengine/docs/python/datastore/query...

Where does it say that 1000 is the limit?

most of it isn't true

if you're using Django, check out http://www.djangy.com -- it's heroku for django (and eventually other wsgi frameworks)

:-)

I'm waiting for my invite - looks sweeeeet.

Any plans to do a RailsTutorial-esque Py/Dj/Git/Djangy tutorial (as opposed to Rb/Rls/Git/Heroku)?

I recently developed my first GAE site, and whilst I didn't run into the problems the author of this article did, I spent a lot of time hunting down best practises.

It's now been ages since I last looked at Amazon's offerings; does anything have any links to best practises / development strategies for either AWS or GAE?

Incredible how many points this article gathers considering the valid arguments made in the comments[1]. There must be a lot of bad experience (or fear) with this platform.

Personally, I recently had my fair share of problems with the DownloadError, but I knew it would come up (because I read the documentation - and I concur with the limit for scaling reasons). So I built myself a failover decorator relaying the failing requests to a VPS.

Otherwise, I love GAE!

1. For example the top commenter nl with 83 points: http://news.ycombinator.com/item?id=1928148

#3 isn't true for taskqueue tasks or cron jobs anymore, the deadline is now 10 minutes ( actually will be once 1.4.0 SDK is released in about a week or so )

Also at first glance there is no indication of how the author got to a value of 15k€. My best guess, and a guess at that, is that they put the value of a line of code at 1€ and had to migrate 15k lines, but I hope there is more scientific than than.

I hate products that suck, waiting for that magical upgrade that's just around the corner.

After using AWS for some time now, with things like CloudFront, S3, Elastic Map Reduce and managed Mysql instances and AutoScale (and hey, they've got servers in Europe), App Engine really feels like a half baked toy.

He also mentioned that working around GAE set them back 5 months.

This entire article reads like a tutorial about how to make poor decisions.

I mean, if I want to eat a cake, and I know how to make a cake, I'm not going to pull up a recipe for brownies and try to make a cake out of them. This guy made some really bewildering choices; the list of things he clearly wants to do is both silly and frustrating.

"I want to write a site using Django that uses full-text search and multiple-table JOINs so that it takes longer than 30 seconds to load a single page, and Google App Engine won't let me!"

Brilliant.

IMO, i feel this is a design flaw. Without understanding what's a NOSQL used for you can't except joins to work.

Shouldn't there be a checklist to show "What applications can be moved to cloud (appengine here)? including the myths, expectations, assertions?"

i think in general what we should take from this article is one user's experience with gae and use it as just another data point. many comments on here seem to suggest that they're upset about the bashing, but this is just feedback, a grain of salt, i find it helpful, although noted that you could probably find most of these in the documentation.

i originally also considered gae for my project, but decided against it, my impetus was that i wanted to use a homegrown best of breed stack: tornado, mysql, nginx, memcached, python2.7 and have more control over the environment.

Did someone document their progress, or overview the development of a medium-sized app (~10K LoC) on GAE?

I'm a little bit confused about the number of arguments against the OP's post, here and especially on his own site.

As someone who's considering whether going w/GAE or custom, the points he makes are totally valid and applicable.

Well, I don't know much, but I do know that memcache has a limit of 1MB per value. So that fact that he attributes this to be a problem with AppEngine tells me two things: 1. He doesnt read technical specs (forget the manual, just the basics of "what this library does"). 2. His complaint in this case is completely invalid and entirely his own fault. This leads me to believe (on an albeit small samples size) that the rest of his complaints are invalid. Here is someone who begins a project without understanding the tools available, and when he encounters their limits he then blames the hosting environment. I expect he will have a rude awakening.

http://code.google.com/p/memcached/wiki/FAQ#What_is_the_maxi...

Really, REALLY I can't understand how this can be #1 on HN now. Ah, Carlos, please go RTFM! Powerful tools... they are not for everyone.

This is the worst critic I've ever read about App engine.

One thing not mentioned is the development server slowness and need to constantly restart it. The live website actually ends up being much faster than the development site on your machine. Apparently, this is due to the fact that the development server is single threaded. TyphoonAE is one work around but then you're depending on a third-party to catch up with the latest API changes.

It is not even in Beta yet, as it states in the documentation from Google;

"This is a preview release of Google App Engine."

My biggest gripe about GAE (and Google in general) is that when a change is made on Google's infrastructure that causes large problems, no acknowledgement (or answer) is made until enough people complain.

GAE is web scale!

i'd curious to hear what you switch to and, in about a year, how well it is working out.

my buddies and i are working on a time manager for college students and we are having a hard time deciding between RoR and GAE. any insight would be helpful.

we are looking for: login manager database capable of up to 50 fields for each user sorting and search capability must cost less than $10/year/user.

ps. if you are someone interested in solving this problem, please contact me directly at hn@sahajsingh.com

Why is it mostly non-western developers that post those kind of ridiculous "please solve my problem for me" questions?

(btw: non western myself)

For all I know, GAE is really easy to develop on. I made this using Django in a couple of hours' time: http://bhoogolvidya.appspot.com/

Yes, I'm sure your one page, one form, application tells a lot of relevant stuff about GAE.

Reply on news.ycombinator.com