Indeed. Everyone should value their privacy seriously, much more than the general population currently does.
It is why after the U.S. kills or captures some foreign leader, they brag about figuring out their routes and daily habits. It is not a stretch to say that it could also be done, and probably has before, in the U.S.
Extreme penalities should be put in place for any location data access without a court order... And your location should never be allowed to be sold or shared with any non court approved third party. It really is that serious and if the public had the bandwidth to be concerned over another issue, maybe something would change.
Who knows, maybe all the public needs to take it seriously are some real life examples of location data being used illegally...