Also running the database, webserver, mail server, etc. off a single machine is just asking for trouble.
nginx, PHP-FPM and APC, your caching tool of choice, done.
Sounds like "now you have two problems"? There is plenty of blog software out there but very few are so terribly, terribly slow as Wordpress. I'm pretty sure it's the slowest thing I've ever installed on my server (and uninstalled, obviously). Just use something else and there is no need for caching in the first place.
If anything, moving to an unproven blog framework is going to cause much more headaches than a worpress blog slowing down for seconds every time you make a post.
Until five years ago I ran that* on an old 2001 laptop with 128MB RAM, of which about 56MB went to Windows XP. Worked quite fine really.
* That, plus FTP server, SMB, DNS, VNC and uTorrent. I think that was all.
It's not like many other professionally hosted websites haven't done the same over the years, and my data remains in my control even when my website is crushed under the weight of enthusiastic visitors, so my self-hosting goal would be intact.