You would take the Fifth Amendment and invoke your right against self incrimination.
"...nor shall be compelled in any criminal case to be a witness against himself..."
Technically speaking, you are only allowed to refuse to answer a question if you believe that answering will incriminate yourself.