If they want to end the interview, it doesn't matter what question they ask, because they won't be listening to your answer anyway.
Yeah, that's like saying, 'hey, if you don't like the laws in this country, you can move out.' You sure can; you sure CAN. Disparity of "power". That's what interviews are all about. So you would pass up a great candidate because they disagreed with the relevancy of your line of questioning? It takes a lot more intelligence and self-confidence to consider a question and respectfully opt out of it than to blindly do your master's bidding and muddle through a through a question you don't even understand.