$5? You still need to store the weights, and with 100's of millions of parameters@32 bits each that still won't be cheap. It'll be a lot cheaper than a GPU though and it will use far less power.
The application NVIDIA targets their TPU at is clearly vision/robotics and while there are probably many vision/robotics tasks that can be done with very small nets I think the majority of them will require something a bit more beefy.