Exactly. This is something I think most don't understand. Patents give you the right to sue. That's it. They don't provide any "protection" beyond that.
It's like freedom of speech. You have the freedom to say whatever you want but the first amendment does not offer any protection from the consequences of your speech. In other words, say the wrong thing at the wrong time in the wrong place and you might just get beaten into a pulp. You had the absolute right to say whatever you wanted but no protections beyond that.