The idea I had was roughly to make some sort of extensible voice driven agent you could customise to do pretty much anything you want.

Did you get it working on a Raspberry Pi? Would be interested to know more.