This is pretty interesting, I have a related project (plug, hadoopy.com). The way I went about this (in an experimental branch) is to use Celery running on Redis.