Refusing to answer the question would be interpreted the same as saying "yes".