Yes of course. If the goal is to hire for attribute x and you decline to say whether you posses that, they will pass you over until they find what they're looking for.
Military service is the only one I can think of that's legal to take into consideration.
Never mind that it was a blatant violation not only of the Constitution, but of the letter of the very government policy that initiated Affirmative Action to begin with.
And actually, that means I was unfair about your first comment - I wouldn't want to work for a place that operates that way anyway, as another top level comment notes. I think I can safely discard secret illegal hiring practices as a concern when interviewing, which does solve my issue of whether to answer truthfully or simply opt-out of those questions.