Hacker News Reader

Comment by mmastrac | Hacker News Reader

I'm the original author of the post. If anyone has any questions about AppEngine or what it's like to develop on that platform, feel free to post here.

View on HN

I'd love to hear about your view on which frameworks to use for python (tipfy, webapp, tornado, django ports etc) and their advantages over each other. I've developed a webapp using django-nonrel, it was pretty quick to have it going but I have this uneasy feeling in my gut that something may be wrong.

I'm not as strong with GAE/Python as I am with GAE/Java, but my feeling is that your AppEngine experience will be far more productive if you use AppEngine's native Datastore APIs rather than any abstraction layer.

There's a lot of detail at the low level that you miss by using any layers of abstraction (on both Python and Java platforms). That detail is important for getting maximum performance out of your application and correctly dealing with AppEngine's unique transaction model.

The choice of web frameworks is tougher. Startup time is important right now for AppEngine (although slightly less important with some of the new features rolling down). You'll want to use the lightest-weight framework you can find.

On Python, I've had great luck with webapp + Django templates + raw Datastore APIs.

On Java, Guice + Guice servlets + raw Datastore API is really lightweight and fast. We've also developed our own Django-like templating system for Java that we hope to open-source at some point.

Anyone else with more GAE/Python experience want to chime in?

Webapp, webapp templates (just like django) and datastore.

I like it barenaked and closer to the metal.

Could you share some information about how your app code is structured? The main reason I'm using Django is because I like that it gives me a directory structure. The only examples I've seen of webapp are a ton of handlers in one main.py file, which doesn't seem maintainable.

Usually I put one handler per .py, to the point, what that particular module should do and nothing else. I leave the routing stuff to the app.yaml file.

For a dir structure, I have admin, css, js, prog and view. Not hard to know what goes in every folder. Right now I have like fifty progs and fifty views, each for every specific task.

For every program there is usually one line to get the data from models and one line to render it using a template. Nothing else.

I really don't get what a framework would do to help me, I already have everything I need at my fingertips.

Here is a sample index.py that renders index.html. I enhanced webapp with some utilities to make my life easier, I hate typing in excess.

    import app

    class main(app.request):
      app.session.start()
      data=app.models.getContacts()
      app.render('index',data)

    app.run('/',main)

And here the html in the template:

    <html>
    <body>
      <h1>Contacts List</h1>
      {% for item in data %}
        <li>{{item.name}} - {{item.phone}}</li>
      {% endfor %}
    </body>
    </html>

Some purists will burn me at the stake, but it works wonders for me.

How about a framework like objectify? Is that close enough to the low level datastore to get good performance?

I've heard really good things about it. It's written for GAE and GAE only, so there aren't any abstractions that insulate you from how the datastore works. It's still worth your time to write some code against the raw datastore APIs. It takes some time for entity groups, keys and parents to sink in and working with the raw API helps you see what's going on.

I started with the raw datastore API and was able to do what I wanted after quite a bit of research and trial and error. I was then turned on to Objectify and couldn't be happier. I know what is going on underneath and the limitations but I have a much easier way to deal with them. Objectify is highly recommended, but my app doesn't have massive scale so I am not sure if it adversely affects performance much.

> We've also developed our own Django-like templating system for Java

Any chance you could list viable alternatives, please? (I'm considering GAE/Java). Thanks for this interesting thread.

I did a study of the bigger templating systems available on Java earlier this year and the landscape wasn't that great: JSP, Freemarker, Velocity and a number of smaller projects. We decided that it was worth some time developing a system inspired by Django's lightweight syntax. If you send me a note (matthew@mastracci.com), I can let you know when we get our project up on github. It's rendering every page on gri.pe right now and has been stable for a few months now.

Have you looked into Google Sitebricks for templating?

We use python on GAE, and chose tornado, and admittedly this is partly because at the time it was the new hotness and I like friendfeed a lot. That said, even though with the wsgi adapter you've neutered what tornado is famous for (evented python framework), it is a nice concise web framework with a flexible templating system that is easy to grok and get working quickly, yet still has some nice features like UI modules, named url matching with reverse_url construction, xsrf protection and secure (in the sense that they are unforgeable) cookies. Its templating system doesn't auto escape, so we hacked it to do so.

Using a minimalist framework is a pretty good match for app engine since other frameworks might assume something about the environment that won't be true of GAE.

And you can always mix in other ala cart libraries. For instance, we use wtforms. That combined with a tornado UI module to render the forms nicely has been a effective.

If I were to choose again, I might just use the built in web.py since they often include utility handlers for things like parsing multipart uploads; we've had to reimplement those for our tornado BaseHandler. Then I might use mustache for templating since we have started using that on the client side.

Kind of random thoughts I guess, hopefully somewhat illuminating nonetheless :)

Anyone tried Flask/Bottle on GAE? These Sinatra-type mini frameworks seem like a good fit, but I never had a chance to try it out.

I used Flask in two of my currently running apps. It works nicely together. AppEngine limitation never gets in the way, at least not much as running Django on AppEngine.

Why did you chose Flask over tipfy? Because of Jinja2?

I can't remember exactly why, but it has something to do with how tipfy handles WSGI middleware back then (and it was very confusing, since tipfy also call their plugin system a "middleware"), and the framework being a bit too verbose for my liking.

My first impression of GAE (admittedly, many months ago) was that it was very painful to deal with loading or otherwise manipulating data in BigTable. Do you have any shortcuts or advice here? Has it improved?

Annoying Example: I realized that I needed to delete a lot of records from the table. I created a URL that I could ping periodically to delete a bunch, then get killed because it was taking too long, repeat.

Check out the mapper API. They've wrapped up all the difficult details behind a hadoop-like interface:

http://googleappengine.blogspot.com/2010/07/introducing-mapp...

That looks exactly like what I needed. I will have to come back and give them a try again. Thanks.

I am really glad someone wrote a response from another POV. I was thinking about doing the same if I get the time on the weekend. We have been very happy appengine campers and the 1.4.0 SDK release looks very promising.

There are and have been some rough edges but overall it is a great platform and it was well worth it for us to invest time to evaluate and embrace the constraints.

Nice writeup. I'd like to hear more about how you manage using S3 for video in conjunction with running the rest on GAE.

Hi, I work with Matt at Gripe :)

Video is in development, but the system is pretty straightforward. We're integrated with Zencoder. Our clients (e.g. mobile/web) request an upload endpoint from our server (GAE). The endpoint is a pre-signed S3 upload url computed by the server. The client uploads to that url and then triggers a "process" API on our server which issues a request to Zencoder on the backend to pick up the videos from S3 and begin transcoding them. Zencoder sends back a response that allows us to associate the "video" entity in our datastore with job ids/output locations. You can set up a callback so that Zencoder will notify GAE when transcoding is complete. The output is available in S3, including thumbnails.

So I guess this is also an endorsement of Zencoder: it rocks and was very easy to integrate with!

Thanks! our startup (realtimefarms.com) also runs on GAE (python though), and we use the blob store image serving to great benefit, but have been toying with supporting video uploads, helps to hear how you guys are doing it.

Thanks! And great writeup by Matt. You should do a blog post on your video workflow - GAE and a transcoding API like Zencoder. :)

Nice write-up Matt.

One thing I'm curious about the is continuous deployment. AppEngine has a 1000 deployment limit (http://code.google.com/appengine/docs/quotas.html#Deployment...)

Is that a worry for you?

Thanks, Nick! That 1,000 deployment limit is daily, AFAICT. Our quota dashboard shows 9/1000 deployments, which matches up with the number of builds we've pushed live today.

Oh. I guess that makes sense.

I don't think that's likely to be a problem then!

Not unless the biz guys go crazy making content changes (they can commit to svn and their changes go live via the same CD system)

That seems like an accident waiting to happen ;)

How are you doing backups of your database? How often do you do them?

How do you do full text search?

The other two GAE topics posted this week had quite a few reports about datastore time outs. Has this been a problem for you?

If you want to see the answer to your question turn show dead on. Someone made a mistake.

Do you know of any good links for GAE development patterns?

I've read tips in various comments on various websites, such as use filters instead of GQL, using task queues and handling storage exceptions, but very little in one place :)

I'm more familiar with GAE/J than the Python stuff myself. I don't have specific links to patterns, but I highly recommend watching the AppEngine videos from Google I/O. I watched nearly a dozen hours of video during development:

http://www.youtube.com/watch?v=o3TuRs9ANhs

http://www.youtube.com/watch?v=AgaL6NGpkB8

Here's some random tips I can give:

It's worth spending some time and writing the ORM stuff by hand for your first application. The low-level datastore APIs are well-written and you'll learn a lot about how stuff works by being close to the metal.

AppEngine rarely throws storage or memcache exceptions outside of maintenance periods anymore. Unless there's someone here who's had a different experience, I would consider them rare.

Use as few abstractions as possible, no matter which language you are choosing.

If you don't need something that interacts with an external system done right away, stick it in a task queue, no matter how quick you think it will run. Assume that it'll succeed in your code (it probably will, eventually).

Great tips, thanks! :-D

Reply on news.ycombinator.com