First prototype was on an Arduino it and eventually ran out of firmware space. So then I upgraded to the RPi and from there it was a breeze.
Suggestions:
(1) Use Wit.ai for NLP. There is some added latency but the capabilities far out reach Sphinx in the long run. It's free. Less code to maintain. Easier to deploy and distribute.
(2) Try to find a small mic so that you can put everything in a sleek package.
(3) Add support for bluetooth speakers (you're on a RPi, it's basically done for you)
(4) 3D print a custom case, throw some 3M tape on it and it's ready to be wall-mounted!