Instead of getting into specifics, would simply listing them as "someone I would never work with again" suffice? If enough people do this then the specifics don't really matter... the warning sign is clear.

This seems to avoid the defamation issue because it is a personal statement (about yourself / your thoughts) instead of a statement about them. They could still sue you for defamation, but hopefully you could get it thrown out early at a low cost?