So, the only difference between the anti-English banter I experience and racism is that I don't get offended?

You had better hope I don't start being offended, then! :)

And, yes, this argument also applies to Scots living in England, or any other similar situation.