In answer to your questions:
1) Yeah, good cheap Java hosting doesn't seem to exist. That said, I think the Rackspace Cloud and Amazon EC2 are great solutions and great value for money.
2) Java is a high performance language, if you get performance issues it'll be your design/archiecture.
3) thingloop.com, successful from a technical standpoint, business-wise, the jury is still out!
On the whole question of which language, I'd say:
a) avoid religious arguments - my language is better than yours! They all have advantages and disadvantages.
b) a lot of it is down to what's cool. Ten years ago Java was the coolest kid on the block, now it's RoR. Thinking longer term, you might want to consider what the most available skillset is in your area for recruitment purposes.
c) if you've got a bunch of guys who already know a platform inside out, then stick with that. They'll almost certainly be way faster.