"Even if their refusal was based on the Fifth Amendment privilege against self-incrimination, the Court still may draw adverse inferences against them in this civil proceeding. Baxter v. Palmigiano, 425 U.S. 308, 318 (1976)"
The way I read it (again, not a lawyer) is that you can't convict someone based on their refusal to testify, but this isn't a criminal proceeding. In civil matters the rules are different.