Varnish does go a long way, on the HTTP side combining and minifying scripts helps a lot too. I think the heavy load is almost the nature of the system, but there are some ways to alleviate the pain!