You could always respond to reference calls by saying, "I'd love to tell you all about him, but don't want to open myself up to litigation." That sends a clear message without you having to say anything.