tl;dr: Your phone gives off a Bluetooth signal that often times is unique due to variation in manufacture. A motivated attacker could isolate what your phone's pattern is (if it's unique enough) by following you to a few different locations and checking what pattern repeats.

This is a lot less powerful than I expected from the title but perhaps I'm not thinking big enough. A powerful adversary could set up sniffers all over the city and track the population, I guess, but I imagine the value of this method goes down the more phones that are being distinguished (variability among millions of phones probably means a lot of overlap, especially considering it doesn't seem that variation in signal is consistent).