I also use to manage exim4/spamd long ago. Part of the problem I remember is that your email (sent from your server) can be marked as spam on someone else's inbox (e.g. you are blacklisted without knowing it). So while you haven't had issues on your side, it's hard to prove that you didn't have issues outside your box. This was especially common with VPS datacentres, as their IPs routinely showed up on blacklists.