I had similar nightmares to you for a long time with Apache/PHP/WP, then finally put Varnish cache in front of the whole thing. Every single page loads in sub 1 second, and even a massive traffic spike results in CPU load under 0.2.
Little dragon logo (maybe 15k?) and a CSS3 gradient? http://www.colorzilla.com/gradient-editor/
If your site was otherwise graphics-heavy I wouldn't say anything, but it is a really sparse/clean layout already.
For me, dealing with problem was pretty painful and took me much longer than I thought it would to get it right. VCL also has a bit of a learning curve, although I did find it rewarding and was glad I did it.