That phrasing (at least without the "to you") is fine when you are genuinely open to saying more. It's not fine as a way to refuse to say more. So you're talking about a completely different context.

If they get aggressive and you then decide that you don't want to explain, that's fine. But that's not what happened here.