Hacker News Reader

Comment by madflo | Hacker News Reader

Via Cron Jobs ? http://code.google.com/appengine/docs/python/config/cron.htm...

Quote from that article:

     A URL invoked by cron is subject to the same limits 
     and quotas as a normal HTTP request, including 
     the request time limit.

And I'm looking for some tips and tricks to do heavy processing (which is really needed when working with non-relational datastores).

The key is to combine cron jobs with Tasks. Tasks are am easy to use and powerful feature of Google Appengine that let you deal with the standard limitations. For example, rather than doing some operation on 1000 entities in one request, you spin off 1000 tasks, each handling the operation on one entity. Read up about them at http://code.google.com/appengine/docs/python/taskqueue/

You can also explore the new Mapper API at http://blog.notdot.net/2010/05/Exploring-the-new-mapper-API

Agreed, the Mapper API is the best way to deal with processing large numbers of entities from the datastore.

AppEngine requires careful thought of application design at this point. Everything is an HTTP request and subject to the same restrictions.

I've heard rumours that they'll be launching longer-lived tasks at some point in the future, but I have no idea when that will happen.

Reply on news.ycombinator.com