Normally in a good interview environment the conversation will naturally lead to a situation where a candidate asks a tough question or an algorithm etc. Or as happened to me once, they asked me why we didn't use Y, to which my answer was WTF is Y and why would you suggest it as an option here? I was genuinely curious why he felt it might be a good fit and it led to some awesome discussions where he proved to professionally kick my ass in the details. We made an offer that evening to him because as we all sat and talked about his interviews he had made really great observations and was an awesome team fit.
So I think if done as part of the interview conversation fine, but if you are doing it to just try and leave an impression or have a ego contest it is a stupid idea. And I do think you are right to ask non-softball type questions of a potential employer, its a big decision on both parts so you should know what you are getting into too. It really boils down to how it is done.