The entire point of the whitelisting was to allow our internal mail server to forward transactional messages to addresses on our own domain, so it was not even that it was ever reported as spam by anyone, or ever hitting external addresses where I'd understand if Google was more trigger happy.
It was ridiculous realising that they basically had no way for even paying customers to tell them that "that e-mail server we told you is ours and we always want to receive all e-mail from no matter how spammy you think it is, using the mechanism you say it there to do exactly that, well, we really mean it". Instead it'd randomly just stop working. [this is a few years ago; who knows, maybe they've fixed it]