Even more troubling if you think that, since "he didn't perform the work", then he could only have gotten to the server by nefarious means. Hence, hacking charges...
And they'd likely win with that argument under the CFAA. Even if they just handed you login credentials, they did so in the context of you fulfilling a specific task. From a technical perspective, those credentials might give you root access that lets you do anything. From a legal perspective, that's not the same as giving you authorization to do anything. For the CFAA, this would likely constitute "exceed[ing] authorized access." Not to mention that taking down a site (or just defacing it) is, on its face, clearly against a client's interests.
In any case, you're using the threat of damage to force payment. Even if the client owes you a valid debt, those threats can likely be considered extortion. The sort of jerks who will gleefully screw over a freelancer because, as they so kindly informed you, "no one will listen to you" are often the sort who have no problems with screwing you another way.