Because a lot of people who are not white are born in Britain, making them native Brits?

It's sort of a silly opinion really. The British Empire spanned the globe. It's akin to being shocked to see blacks in the US.

that's not the controversial part, it's the distinct overtones of "go back to India/China/etc" that comes with it.