Control over network configuration and hardware is one of the big reasons. I hate getting bitten because my three new servers are halfway across the facility and stuck with somebody else who is maxing out the uplink between our shared top of rack switch and the core...
Edited: Also, your "$100/month" quote is pretty fanciful. Who is handing out 32GB with SSDs for that price? I'd really love to know, because the best I've been able to get for a database quality machine (RAID-10 with BBU, SSDs, and lots of RAM) is closer to 10x that.