Very interesting writeup, that's a cool project.
Now to make it still cooler and Jarvis-like, you can add voice commands with the Raspberry Pi. Somebody did it here [1] using the Wit Speech API [2]
[1] http://blog.fmpwizard.com/blog/go-cortex-using-wit-ai-using-...