I also switched from Google Apps / Gmail to FastMail and I can't go back even if I wanted to.
Biggest difference between Gmail and FastMail in how they handle email is tags vs classic IMAP folders. This can be a turn off for people, but for my personal email I discovered that I want folders. This is because my personal email also has the role of backing up my work email. I've lost work email in the past when switching jobs and I don't want that to happen again. And with FastMail I can simply download my work email in its own folder, so it doesn't have to pollute my personal email. FastMail is also very flexible in its settings. Setting up filters is easy in Gmail but limited compared with what you can do with those Sieve scripts in FastMail. Speaking of the interface, I now like FastMail's interface more, as it makes it easier to switch between folders, or back and forth between email, calendar, contacts or notes.
SPAM filtering in Gmail is said to be extremely good, however a certain class of spam has been reaching my Gmail Inbox for the past 2 years with me being unable to stop it - I made the mistake of giving away an email address on my blog and now I get emails related to SEO marketing and bullshit. In Gmail I cannot setup a filter that automatically marks an email as spam if it contains the word SEO. But in FastMail I can do just that.
FastMail does CalDAV for the calendar and CardDAV for their contacts. I can sync my Android phone and it works out great. IMAP is also very responsive and standards compliant. And personally I feel good about rewarding a smaller player that plays nice with others because monoculture are bad ;-)
Another thing I did after that was to create 2 Google accounts. I'm using one Google account for Google Play purchases and nothing else. This is because Google Play purchases are not transferable. So for example I have an old phone lying around that I gave to my son for playing. Restricted accounts only work on tablets, don't know why, so my son ended up having access to my email, which is not cool. Plus I'm all for paying for apps, but I should be able to share my purchases with my wife. Like if I buy a GPS device, we can both use it, but if I buy a maps app my wife can't borrow it without borrowing my account, along with my email. And I don't like this kind of lock-in.
There are some downsides. I could not find a good email client that can do IMAP Idle. FastMail's app works on Android but I don't like the UI. On the bright side that app is simply exposing their web interface, so FastMail works very well in a mobile browser, something that Gmail doesn't do and it matters when borrowing other people's devices.
All in all it's been a great transition for me and best of all is that I can now try out other Google services and products without feeling remorse or fear of lock-in ;-)