With Picovoice you can use the voice control engine to accomplish this. Maybe something similar to this demo?
https://picovoice.ai/#voice-control-demo
The cool thing about this engine is that it is tiny. It uses less than 8% CPU on RPi3 and altogether it is less than 2MB (code, model, etc). Technically you can run it on something much smaller and cheaper than RPi.
Alternatively, the speech-to-intent engine could be a good candidate. More information on this along with an interactive demo will be released this weekend.