I don't host it for myself. Over the years I've collected a small blog network. I'm proud of the blogs I host, they're the some of the best and most influential in the Australian blogging world.
But by god do I hate hosting Wordpress. If I made more I'd push the whole thing onto WPEngine and be done with it.
Nowadays I have it running smoothly. The performance keys are:
1. Exorcise Apache. You don't need it. Nginx + PHP-FPM is the correct option.
2. Caching, caching, caching. Enable the MySQL query cache. Install WP-Supercache or W3 Total Cache. Have them spit out gzipped HTML to disk and tell Nginx to serve it directly.
3. MySQL should be on a different server. Annoying but it causes a dramatic improvement.
There are other tweaks I haven't applied such as serving static content from a cookieless subdomain, pushing stuff onto a CDN. If I get to the CDN stage it's time just to turn it over to WPEngine.
It's now sufficiently fast. During the Brisbane floods last year, one of the sites got > 100,000 visits in a few hours and I didn't even notice a slowdown (in fact I didn't notice the visits until a month later when I logged into Google Analytics to check something else). I am now preposterously over provisioned but at least I don't have to think about it any more.