The only problem I've found with this method is the spammers that try to guess your email, so they end up sending emails to "admin@domain.com", "webmaster@domain.com", etc. The catch-all forwards them all to me.

The only way around this, I think, is to only have uncommon emails, like instead of admin@domain.com, use contactadmin@domain.com. Put a block on the common ones and you're good to go.