The problem you describe has been fixed with precompiled wheels. As long as projects make the effort to package them, there is no need for compilers now.
Don't know why this was downvoted. NB you need to upgrade your pip to the latest to use this if you are on Ubuntu LTS though. (But you are working in virtualenvs so this isn't a problem right?)