The idea I had was roughly to make some sort of extensible voice driven agent you could customise to do pretty much anything you want.

Did you get it working on a Raspberry Pi? Would be interested to know more.

Looks great. I was actually trying to do the same thing. Run an agent on a raspberry pi and use it to automate stuff around the house or at work. Thanks for sharing :)