I worked with TellMe (a voice reco company) right after they were acquired by Microsoft. My job was mainly to work on VUIs (voice user interfaces) for the Windows Phone, Xbox, and Cortana during its early days.

There are two things blocking voice interfaces from being more prevalent. The first is an action word. As you said yourself, all voice systems today require the user to enter in to a voice mode before actions can be recognized. It's sadly a necessity as devices today can't differentiate between when they're being talked to and when a user is talking to someone else. Even if you only allow voice commands within a specific context (i.e. an alert pops up, the user says "ignore", the pop up goes away), you're playing a very dangerous game. Take this scenario for instance:

Bob on his computer: "Have you seen this cat video? Look at how the cute kitty cat will completely..."

Phone: POP UP WITH ALERT ALL YOUR ICLOUD NUDES ARE BEING STOLEN

Bob on his computer: "...ignore..."

Phone: pop up goes away, never seen by Bob (unlike his nudes, which are now everywhere)

Bob on his computer "...everything around him! I would never ignore things that were important."

See the problem there? While it is a slim chance of happening, the effects of misinterpreting a command can be dire. The potential negatives outweigh the convenience. Maybe if everyone actively used voice commands, the net sum would be positive despite it destroying a select few peoples' lives, but that brings us to our other problem:

Voice commands are socially looked down upon and are rarely used by most people aged 25-50.

This group of people have two things in common - they typically know how to use a keyboard and trust its input, and they've all used the terrible voice commands from old phone systems in the past, which leads them not to trust voice. The lack of trust means few people use it, making it seem weird to do in public. This is getting better, and certainly more people are using voice commands now than they were five years ago. But it's still an extremely small percentage of users that are voice power users. Ask yourself this: how many people do you know that send text messages with voice? What percentage of your friend group is that? I know two people - one of them was a voice engineer who had to test voice input on phones constantly.

Voice will get better, but it's going to take time. More people need to use it before we can take more risks, and we need to develop systems that know when they're being addressed before we can get rid of action words.