No, it doesn't really bother me. Why would anyone care that I get up around 7AM on weekdays, drive to work around 9, stay there until 5, and then drive home?
On weekends they will see me going to Target and the grocery store. Sometimes another store. Sometimes I go to visit my family in another city.
I really don't care if people have that information. Many people (not me) post that information freely on Facebook, Instagram, or Twitter.
There are some future situations in which I might care that I am being tracked. If that were the case, it's highly unlikely I would bring a phone with me.
Would I prefer not to be tracked? Probably. Does it bother me that I am being tracked? Nope.
I suspect these views line up with the majority of people in the US.