But you can also have an invasion of privacy or invasive surgery. In that sense it is about unwelcome intrusion into one's body / sovereignty.
And people are entertained by news articles with titles like, "10 times countries accidentally invaded their neighbors." Clearly the intent to violate sovereignty matters.
I think you can argue that the Bin Laden raid was and invasion into Pakistan. Anytime a military forces enters uninvited, that's an invasion.