Google App Engine Datastore API proposal by GvR | Hacker News Reader

Google App Engine Datastore API proposal by GvR

neopythonic.blogspot.com

70 pointsbockris11 comments

""" if we squint a little it looks almost like a synchronous call..

you can take a piece of code that was written using synchronous calls, and mechanically translate it into a tasklet: just decorate your functions with @tasklets.tasklet and change your synchronous calls from

result = some_function(args)

into

result = yield some_function_async(args) """

I feel like this style of coding (e.g twisted) is stretching python generators a bit far... are callbacks really that hard to work with? And this is clever and all, but what happens when you forget the yield call, or when there is an exception somewhere down the line, how hard will it be to figure out what is going wrong? I would prefer the work went into better function literals in python along with a node.js style approach.

That aside, glad to see better asynchronous support coming to app engine, using:

http://code.google.com/p/asynctools

has already been a big help for constructing complex pages that require a few different queries.

I personally like the coroutine approach better (greenlets) since your code looks synchronous without squinting and usually performs well (there are various optimization in Stackless Python and PyPy has support as well). With libraries like gevent, and its API inspired by multiprocessing, asynchronous handling of code blocking on i/o just becomes another option next to using threads, processes or computing cluster.

Whereas with node.js/twisted (excluding @inlineCallbacks) approach the implementation detail—that is how do you talk with OS kernel about I/O—becomes the force shaping how you design your program and your API.

Sure, there's place for callbacks in Python. But as Python programmers, we can afford to reserve them for where they belong, like handling high-level events (i.e. observer pattern).

agreed. I had to use twisted this year, and I find the whole approach really wrong from an API POV. You have to force everything to use twisted-way, and this impacts a lot the design of the whole app. inlineCallbacks has its issues as well (it sometimes lead to wrong stack-traces, and you are shit of out luck trying to understand what's going on then, but maybe that's just an implementation flaw).

The use of generator has its flaw as well: as pointed already by other, it is difficult to compose them effectively, making refactoring quite a PITA.

Gevent seems like the best option in python, but I have never used it, so maybe it has its drawbacks as well.

I see a few advantages:

1. Requires much less modification to existing code than a callback style.

2. Code written in the sequential style is shorter, easier to understand, and just as explicit.

Edit, also from his article:

"""By decorating your functions with @tasklet and writing yield in front of blocking operations, you get the benefits of concurrent asynchronous I/O without the drawbacks of callback functions. Here's a blog comparing the two styles in the context of Twisted. The Monocle framework also promotes this style, and their introduction provides a good explanation of why coroutines are better than callback (scroll down to "The Big Idea")."""

Links:

http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-...

https://github.com/saucelabs/monocle#readme

You can compare the two styles in Tornado: plain Tornado uses callbacks, or you can code with generators using e.g. http://code.naeseth.com/swirl/

In practice everyone uses plain Tornado with callbacks. The problem with generators as coroutines to me is that you can't compose them nicely: you can't call a subroutine that yields, you must yield only from your top level. This and the fact that it's another layer of leaky abstraction.

good point, I think I ran into that when I was playing with coroutines to build a simple finite state machine to parse something; as soon as it got the least bit complex, I was like, "you mean I can't extract a generator!?" I was originally inspired by this post:

http://eli.thegreenplace.net/2009/08/29/co-routines-as-an-al...

and was thrilled with how clever it all seemed, but in practice, it seems to be stretching the capabilities of the yield statement too far

Actually, he does provide a solution to this, in the section "Tasklets and futures". It's not simple though.

I think inline async has its place. I'm hopeful this will speed adoption of PEP 380 which should cleanup the syntax a bit. I'd prefer C# 5's await keyword rather than yield from, but that's minor.

That said, I agree that Python needs better function literals. I'd also like a better solution to query syntax than all these variable__gt=5 hacks and strange Q(variable__lt=2) objects that encapsulate what is easily expressed as a python expression. I wish you could pass expression trees, something like db.where("age>5 and alive=True") without the quotes. Not sure about the best syntax...

I guess they could 'lend' the syntax from SQLAlchemy, it has exactly that. query.filter(User.age>5) and such.

It was originally from SQLObject, a much older ORM. DeJaVu also has its own variant on Python expressions transforming into SQL using lambdas and bytecode inspection. To my knowledge, the foo__gt=2 thing is a Django-specific concept.

I haven't read the entire documentation/proposal yet, but this seems to be a very good foray into asynchronous programming for Python programmers. I'll have to give it more attention later, and not just for the Datastore stuff.

Reply on news.ycombinator.com