It's the "nice way" to go by replacing your stuff with the original design when you decide you're really not going to be getting paid. Can see just removing the page entirely, kind of. However, I don't think it's good for either side of the table if you do something like this.

If you take away what they gave or took down the site entirely you would be justified and perhaps quickly pay. Doing something classless like this is just going to make the client bitter and perhaps damage your personal reputation.