Been there done that. If you catch all e-mail you will also get a lot of spam to random addresses like say sven@marak.com, lollerskates95@marak.com and so on and so forth. So then you need spam filtering anyway, or you need to configure which addresses are valid.

I still host my own e-mail but I no longer do catch-all. There are only a few sites and services I care about. For those I have trusted them with my e-mail address. For all others I use 3rd-party throwaway mail services.

I get less spam now than I did with catch-all.