I see a few key points in what you said that I want to respond to in particular.
> I am sorry for how they feel. I am not sorry for what I said.
You don't have to be sorry for what you said to be sorry for offending or hurting the person. You can stand by what you said entirely while still being sorry that what you said hurt them. Saying "I'm sorry I said something that offended you" doesn't mean you're retracting your words but it does mean that you are saddened for having caused the other party pain, even if you don't think their pain is rational.
> but that doesn't change the fact that your outrage is [...] totally your fault
If you're concerned about "fault" in this case, I'm not really convinced you do feel bad. "I'm sorry I said something that offended you" isn't "I'm a terrible person who is totally in the wrong"; it's "I regret that I did something that resulted in you being hurt." If you stand entirely by what you said it might even mean something like "I wish it were possible to go back and time and restate that in a way that would achieve what I intended without hurting you."
> "I'm sorry" means I feel bad about something and I don't want to do it again
Ultimately, I think this is what it's all about. "I'm sorry you were offended" is (at its absolute best) "I feel bad about you being offended"; "I'm sorry I offended you" is "I feel bad about you being offended and I don't want to do it again."