I disagree. The more you charge the more you put yourself at risk—nominating some insane figure to scare him off is just as silly as his request (or close to it).
There is nothing wrong with not taking on work or firing a client. It is a risk assessment, and the risks with a client who asks these kind of questions are huge.
PS: I actually find it concerning that someone has to ask how to respond. Some people need to learn to say "no".