Is there a way to only host the receive portion?
I'm happy to pay someone to handle all the idiocy around sending email and getting it through Google and Microsoft, but I'd really like to hold my emails myself.
Sure. Set your MX to your own SMTP server but pay a mail delivery service to send your emails & use their SMTP servers as your outgoing server. You'll have to setup SPF & DKIM appropriately of course.
It's not trivial to do this, but it should be possible.
Is there a paid mail delivery service that doesn't get marked as spam instantaneously?
Or is there a way to do this through something like Fastmail?
Sure. Set up your email server and have DNS of your domain point the MX record at it so you receive all email directly without any third party involvement. Then, set up outgoing email to forward to some third party which handles delivery for you. IIRC there are some free options even, if your outgoing volume is low.
That said, my email server handles incoming and outgoing email for a handful of domains for self and family, no problem.
All of that seems easier than setting up a server to keep your e-mails.