I do precisely this with Fastmail and it works a treat. Setting up Fastmail with a custom domain was a joy, I can easily filter based on the To address, and you can setup a wildcard identity so you can trivially send email from whatever name you like at your domain.

To me the only downside is the mobile app isn't quite as polished as Gmail. It doesn't work offline, and I notice occasional bugs or awkwardnesses. But it's still very usable, and I much prefer the Fastmail web interface to Gmail.

Edit: Also, the Fastmail importer didn't work very well on my large number of Gmail messages. It failed a few times and restarting it resulted in duplicate messags.