Well, yes, that's the issue at hand. Probably legally you're just as correct as that issue recently where a bunch of tourists were stomping all over some hills that were sacred to Australian aborigines, but according to the Australian government were legal hiking paths. The most, legally, the aborigines could do was put up a sign saying "these hills are important to us for religious, please don't walk on them."
There's a lot of issues going on right now that draw the majority of my energy (climate change, propagation of hate, etc), so unfortunately I can't bring myself to look into or exert effort into the moral debate of natives' rights, but I think it's a massive issue with huge implications and not nearly so clear-cut as some of the other ones we're dealing with (i.e., climate change is straightforward easy to argue about).
The issue, as I see it:
Take Native Americans in the USA. Outside the written-law rights they have regarding their reservations, one could argue their sovereignty was violated if, say, a plot of land bought in the 1600s (off-reservation) that happened to include a burial mound was developed on. One could also argue that, yes, their sovereignty was violated, but they failed to defend their sovereignty, and so it was relinquished (I don't like this because I think seizing via violence is fucked). One could also argue that they aren't the prime sovereign claim to that land - one tribe might have warred and taken it from another some 500 years prior, so why should the most recent tribe be considered the morally "sovereign" authority on the land?
Which brings me to the point that really concerns me: Taiwan. Populated by natives that were mingling with mainland Chinese for centuries, before a total Japanese takeover. Japanese sovereign territory (by violence). But then, mainland Chinese take over, uh, "back again?" Maybe? Arguable. Either way, another violent sovereign takeover. And finally, a civil war in China between two political parties, with a huge swath of one side fleeing in defeat to Taiwan. Is that even a sovereign takeover, or a political shift? And what of the aborigines, during this time?
Now China is claiming sovereignty over Taiwan, and from one point of view, they kind of do have a claim - Taiwan was technically Mainland Chinese territory, given back after Japan lost the war, and the Communist party did overthrow the Democratic party. However, the Communists did not successfully completely terminate the Democratic party (which is great because who wants more death), so on the other hand, I can see why ROC(Taiwan)'s claim to the island is also legitimate.
Either way, I'm terrified PRC will enforce their self-proclaimed sovereignty over the nation (which is effectively non-existent at present) using violence, which I'm not sure ROC(Taiwan) would be able to withstand, and then I'm worried afterwards the rest of the world will let China get away with it for the slippery sovereign arguments listed above.