If no one will offer an opinion on a former employee, then you have the information you need.
As an obvious example, if a POC experienced frequent racism at a company they might leave a company without having anyone they would want to use as a reference even though THEY weren't the problem.