I actually switched to using an EC2 Micro instance because my 256MB VPS didn't have enough RAM to compile some of the packages I needed. The compile took a while, but they were for libraries I rebuild maybe once a month.

As far as development goes I haven't had a problem yet with speed. I tend to do small incremental compiles, and when I need to a full compile I go get a cup of coffee.