I've done something similar with Sphinx, which worked quite well too.

However, I feel the real challenge to productify such a technology would be to make this system recognise utterances spoken casually, over loud music, over friends talking in the background.

Also, how would the speech to text recognise the user's commands over some sound playing from its own speaker. Suppose I set up a text-to-speech system to read a book to me. I want to be able to make it repeat a paragraph (because my attention had drifted away). However, the input channel of the microphone would contain not only me shouting "stop", but also the output of the text-to-speech system.

We humans have a way of factoring out our own voice when we are listening to someone else. Is there a way to make machines do that?