It's not in the general, constant sense that you have to worry about, but rather in specific applications. When you're out in public, "they" will have 3-meter-accuracy, more than enough. But when you're in stores, and malls, and other venues where UWB is set up, then that's where real problems begin. They'll be able to track which advertisements you linger around, and which sections you visit.

It's going to be a whole new category of passive location tracking.