This already happens in the industry, like heavy industry.
At home, it's held back by absence of an adequate server. Often a router box or a NAS would be fine to host something like OwnCloud. But to run ML middle, different and much more expensive hardware is required, which would sit idle 99% of time.
The idea is just to keep that within your own LAN.
IoT world mostly wants to create closed gardens so you buy other products around that garden.
Easiest way to make it easy for consumer is to connect your devices to cloud and so most decide to go that way, no need for end user to buy a hub.
And even if you need to (say devices use zigbee and you need a bridge), putting that in cloud means user's phone can always access it, regardless of whether they are connected to the home's wifi, without much fuss (as home internet will most likely be behind NAT/firewall so it can be hard to access directly
$20 for the SMD TPU isn't bad, but it's definitely at the top end of the BOM for custom PCB projects.
Ideally we'll see some competition with on-package options.
Sounds great in theory, but show me a place where you can buy Coral TPUs at anywhere near msrp. Unless something has changed recently, finding a real live unicorn would be easier.
Until we get something local powerful enough "cloudsourcing" LLM while keeping everything else locally is not a terrible compromise.