DigitalOcean.com starts at $5/mo for a VPS that is going to be faster and better than most shared hosts would be. Of course you can scale it up and down as needed. Their $10 a month server is actually really powerful for any sites that get less than say 10,000 visits a day. They even have one-click wordpress installs now among many other open source programs.

I use DigitalOcean now for almost all of my sites. I have abandoned wordpress lately and so I am using custom built sites for all my projects. DigitalOcean is awesome.

Low-end dedicated VPS runs around $5/month. See DigitalCloud or Amazon Lightsail.

If you want to save a bit more money and are willing to commit to a 1-year contract, you could even get a t2.nano instance from Amazon EC2 at around $3/month.