You could always respond to reference calls by saying, "I'd love to tell you all about him, but don't want to open myself up to litigation." That sends a clear message without you having to say anything.

Just say "He worked here. I do not recommend his services".

That's not a particular statement about the person, it gives no version of events. If pressed, explain that this all you can say. You simply don't recommend him.

(Of course, IANAL, TINLA)