Why we’re really happy with AppEngine (and not going anywhere else)
grack.com
grack.com
This, in my mind, is the most insightful line in this post. When cloud hosts like Heroku and GAE are discussed on HN often there is cost comparison between using them and doing the sysadmin yourself on Amazon, or on your own hardware. But what doesn't figure in is that with Heroku and GAE (I suppose more with GAE) is that you aren't just getting out of doing your own sysadmin work, you are also getting out of doing a lot of scaling work yourself. This is expensive, difficult work. Of course it is the type of work that many of us here salivate over, but that's another issue... ;)
Scaling on GAE really works because of all the restrictions, and nothing stops you from restricting yourself in a similar way, e.g. in a similar vein the FriendFeed people have done: http://bret.appspot.com/entry/how-friendfeed-uses-mysql
The sysadmin part is also easy when you've got only 1 or 2 servers to worry about, the hard part of doing sysadmin is also when "scaling". Personally I can get an initial EC2 instance up and running in 2 hours tops, and then I can and have been automating that.
I've also worked on GAE apps, and from my experience the sysadmin stuff is replaced by at least 3/2 as much developer time, and this for trivial stuff.
Sorry for the analogy / bad language, but do you know what else scales? Fucking in the ass, i.e. no unexpected pregnancies, but that would be a stupid suggestion to make, wouldn't it?
It's good to have the options but there's no silver bullet yet.
That's the peace of mind I am talking about...
There are and have been some rough edges but overall it is a great platform and it was well worth it for us to invest time to evaluate and embrace the constraints.
Video is in development, but the system is pretty straightforward. We're integrated with Zencoder. Our clients (e.g. mobile/web) request an upload endpoint from our server (GAE). The endpoint is a pre-signed S3 upload url computed by the server. The client uploads to that url and then triggers a "process" API on our server which issues a request to Zencoder on the backend to pick up the videos from S3 and begin transcoding them. Zencoder sends back a response that allows us to associate the "video" entity in our datastore with job ids/output locations. You can set up a callback so that Zencoder will notify GAE when transcoding is complete. The output is available in S3, including thumbnails.
So I guess this is also an endorsement of Zencoder: it rocks and was very easy to integrate with!
Annoying Example: I realized that I needed to delete a lot of records from the table. I created a URL that I could ping periodically to delete a bunch, then get killed because it was taking too long, repeat.
http://googleappengine.blogspot.com/2010/07/introducing-mapp...
I've read tips in various comments on various websites, such as use filters instead of GQL, using task queues and handling storage exceptions, but very little in one place :)
http://www.youtube.com/watch?v=o3TuRs9ANhs
http://www.youtube.com/watch?v=AgaL6NGpkB8
Here's some random tips I can give:
It's worth spending some time and writing the ORM stuff by hand for your first application. The low-level datastore APIs are well-written and you'll learn a lot about how stuff works by being close to the metal.
AppEngine rarely throws storage or memcache exceptions outside of maintenance periods anymore. Unless there's someone here who's had a different experience, I would consider them rare.
Use as few abstractions as possible, no matter which language you are choosing.
If you don't need something that interacts with an external system done right away, stick it in a task queue, no matter how quick you think it will run. Assume that it'll succeed in your code (it probably will, eventually).
There's a lot of detail at the low level that you miss by using any layers of abstraction (on both Python and Java platforms). That detail is important for getting maximum performance out of your application and correctly dealing with AppEngine's unique transaction model.
The choice of web frameworks is tougher. Startup time is important right now for AppEngine (although slightly less important with some of the new features rolling down). You'll want to use the lightest-weight framework you can find.
On Python, I've had great luck with webapp + Django templates + raw Datastore APIs.
On Java, Guice + Guice servlets + raw Datastore API is really lightweight and fast. We've also developed our own Django-like templating system for Java that we hope to open-source at some point.
Anyone else with more GAE/Python experience want to chime in?
Any chance you could list viable alternatives, please? (I'm considering GAE/Java). Thanks for this interesting thread.
I like it barenaked and closer to the metal.
For a dir structure, I have admin, css, js, prog and view. Not hard to know what goes in every folder. Right now I have like fifty progs and fifty views, each for every specific task.
For every program there is usually one line to get the data from models and one line to render it using a template. Nothing else.
I really don't get what a framework would do to help me, I already have everything I need at my fingertips.
import app
class main(app.request):
app.session.start()
data=app.models.getContacts()
app.render('index',data)
app.run('/',main)
<html>
<body>
<h1>Contacts List</h1>
{% for item in data %}
<li>{{item.name}} - {{item.phone}}</li>
{% endfor %}
</body>
</html>
Using a minimalist framework is a pretty good match for app engine since other frameworks might assume something about the environment that won't be true of GAE.
And you can always mix in other ala cart libraries. For instance, we use wtforms. That combined with a tornado UI module to render the forms nicely has been a effective.
If I were to choose again, I might just use the built in web.py since they often include utility handlers for things like parsing multipart uploads; we've had to reimplement those for our tornado BaseHandler. Then I might use mustache for templating since we have started using that on the client side.
Kind of random thoughts I guess, hopefully somewhat illuminating nonetheless :)
One thing I'm curious about the is continuous deployment. AppEngine has a 1000 deployment limit (http://code.google.com/appengine/docs/quotas.html#Deployment...)
Is that a worry for you?
I don't think that's likely to be a problem then!
How do you do full text search?
The other two GAE topics posted this week had quite a few reports about datastore time outs. Has this been a problem for you?
I'm working on a moderately sized app that uses a fairly complex range of data, but I only have a few tables with a few columns each in the datastore. All the relational complexity of the schema is tucked away in blob fields and handled on the client. Besides greatly streamlining the back end code, it also cuts down on cpu cycles drastically by dumping most work to the user's machine, thereby saving lots of money.
I'm using this approach with a Flex UI, but it applies just as well to javascript.
I found RailsTutorial to be the easiest for my noobishness - though all were admittedly pretty good. GAE didn't seem easily portable at all compared to the other two - which was concerning, while DjangoBook's system seemed incomplete without teaching version control - like R-T does. For noobs, I like the easy way RoR/Git/Heroku are together, but I like Python better.
I'm anxious for http://www.Djangy.com to come out as the Heroku for Django and encouraged to see them use Git and not Hg.
HN member: endlessvoid94 is the guy behind Djangy - I hope he sees a lot of us begging for this!
This really isn't about good and bad... it's about right tool for the job. The guy yesterday whining about GAE picked wrong.
We could have done it the same way that we did with DotSpots: EC2 + MongoDB/MySQL. Instead we chose AppEngine to host our system and have spent very little time managing infrastructure.
An added bonus: we've also spent somewhere around $2.00 total to keep the lights on.
Of course, installing everything on more expensive EC2/rackspace machine / mysql service would take half of a day, ....
Using heroku would be way more easier, but heroku is not so cool and hippie as Google.
We spent three years working on DotSpots on the EC2 platform. While you can certainly boot up an instance of your application and get it running, there's still a lot more infrastructure work required to keep it going, scale beyond a single webserver, etc. On top of that, you've got to keep something like Pagerduty running to make sure that your instances aren't wedged or your MySQL box isn't running close to its storage limit.
The big benefit for me was being able to chose a language I'm strong in and just deploy something that automatically scales. This is particularly important during the early phase of the project where you are a lone coder doing all of the work.
GAE, through Java, can run a multitude of languages, basically everything that runs on the JVM. Not every language is a good choice for GAE, but if you want, you can.
You’re right about installing many things on EC2/Rackspace is becoming easier every day. But tuning the system to get the most out of the infrastructure requires advanced knowledge in all layers of the stack. On appengine, you just have to write your app, you don't worry about the infrastructure. It’s a fairly good thing for many people.
With the generous free quotas, you don't have to pay a cent until your project gets some traction, which is not the case with EC2 for instance, because they charge you by the hour and not by the amount of resources used.
GAE is not perfect, but it enables small teams to rapidly launch prototype and iterate until the project takes off. Whether you have 10 users per day or 10M, you’re running on the same stack, built by some of the brightest engineers at Google.
Of course Heroku can be easier if you're using Ruby/RoR, but again using Ruby means Heroku is the right tool. Heroku is useless for someone that wants to use Java or Python to build their site.
At the end of the day a real entrepreneur doesn't care what's cool or hip and instead one focuses on the end product and how he can deliver it.