How do you handle spam filtering for self hosted servers? I know there are black lists for SMTP servers, but I mean the content based filters that are updated based on recent spam signatures?
As for inbound spam, I can use RBL's but the most effective thing has been to block email servers that have rDNS that is either mismatched or nonexistent. I also host at a provider that isn't known for having their IP's on the "forever banned" lists.
This is also a recipe for never receiving mail from the SMB you do business with. (The same is true for enforcing SPF hard fails, unfortunately.)
If you really want to be smart about this, use an SMTP soft error (greylisting) or an SMTP greeting pause. This effectively singles out spam bots, while leaving your regular mail traffic unaffected.
It was a great learning experience, and I got to see a lot of volume. Sadly I never had enough customers to make the jump to the big-leagues, but it was fun.
(I'd do proper SMTP-time rejection, but also archive rejected mails in a database for 30 days. Letting you browse / search through the rejected stuff - just in case I made a mistake, and also to see how good I was doing.)
Even then content-scanning was not the best, but I used a lot of heuristics and I was quite proud of some of them. For example I "defeated" fast-flux hosts, via heuristics to lookup the number of addresses for a domain. Even large domains such as gmail only have 1-4 address-records, though of course their anycast and multiplexed. If you got a mail from a domain with 7+ MX records it was 99% spam.)
At one point I documented damn near everything, after I'd shut down, but it seems I left the domains I used expire. I should see if I can dig out copies.
(I used qpsmtpd as my core, and wrote a ton of glue code and custom plugins. Later the qpsmtpd-proxy was ported to javascript as Haraka - but I didn't covert my code even as I ported some plugins back.)