This idea was similarly proposed by a couple people when I posted my voice-recognition resume/website (https://benwasser.com) to Hacker News. Glad to see it taking off.

Also, I didn't see it mentioned, but I found that I needed to use SSL to get Chrome to not re-ask for permission after a brief timeout of silence. I haven't had a chance to test out their implementation, but my guess is that SSL is a big unstated requirement for decent usage.