Yeah, I do the same, but without the catchall for exactly that reason. If I start getting spam, the e-mail gets disabled.
If you want to 'self host' on a provider, I thing cheap/free options are available from cloudflare, Google, and similar enterprise companies.
If you want to truly self host, I don't have experience, but this guy who does gave a great thorough answer for those who are interested: https://news.ycombinator.com/item?id=48073510