The performance issue stood out to me at first too, but with no UI and Dalvik really only being optional and Google's background services stripped out, this is mostly just a weird Linux, so I'm not sure why this would perform significantly worse than any other Linux-based OS.
Sadly I don't know enough about hardware economics to say whether hosting a whole OS is crazy or not, but we've been trading off CPU time for less developer time in every other area of development, so I don't see why it would be different here. To the extent that the OS itself is not what is causing you problems, you can keep optimising your app code as you go and release on cheaper and cheaper hardware if you want.