It's not just trusting the software, but the implicit policy around its implementation and use and even more implicit social norms around what it is expected to do that develop and evolve in interesting ways as products and services entrench further and further into the woodwork.
Say an attacker captures enough of a homeowner's voiceprint to build a language model good enough to get past Google's or Amazon's voiceprint matching, then pipes "<hotword>, open the front door" through a wall or window (possibly ultrasonically).
What happens now?
Who is to blame?
Is it the homeowner's fault for using a product according to the use cases included in the instructions?
"Oh, no, that was just a serving suggestion! You shouldn't use <product> to lock your doors unless the doors were an effectively unnecessary decoration to begin with."
So then you have Buy N' Large and Big Brother getting into the home and contents insurance industry.
No. No you don't. The only reason this use-case got cleared by Legal is because someone figured out the circle of plausible deniability (weaponized portmanteau of "circle of competence") is currently wide enough to fit a lawsuit defence through. :(
It's like with Tesla and the whole self-driving thing. The implementation *happens* to be able to drive asleep "drivers" down highways at 100 miles an hour, and the company has somehow been able to corral itself into the evil-genius legal position of being 50% "always keep hands on the wheel" and 50% "look self-driving AI go brrrt" and somehow combine the liability protections of the former with the societal expectations of the latter. Solely in terms of overpromising and underdelivering and then wrangling the legal status quo to declare that okay, I do think it's a bit... not great.
Crucially, this problem is most definitely not limited to Tesla, Google or Amazon; it's an endemic permeability in the legal systems of ostensibly-developed countries that makes a joke of the proof-of-work foundation of good-faith technical investment, and institutes a fraudulent social contract that the infrastructure we increasingly depend on is fundamentally reliable and trustworthy ("set in stone") when it is not. To adopt these technologies is to adopt the ever-changing tech stacks that underpin them, and the risks associated with upgrading the proverbial engines on ever-larger digital airplanes while they're in flight. I think it's dangerous and irresponsible to dispel the tenuous fabric of social perception around these risks and give unsuspecting passersby the impression things are safer than they are.
Hmm.
I'm reminded of http://bash.org/?4753:
<xterm> The problem with America is stupidity. I'm not saying there should be a capital punishment for stupidity, but why don't we just take the safety labels off of everything and let the problem solve itself?
Maybe that's what's happening here?