The worry is not a personal one, or even a systemic one, but a concern over the general direction of data availability and societal fragility.
Take, for example, the drone scare in NJ right now. The problem isn't what the drones are doing but the potential they have to do any number of harmful or invasive things. Right now I can go to Costco and buy a drone with an HD camera and hover it outside your bedroom window. Or have it sprinkle anthrax on your head when you walk outside your apartment.
The problem is that technological advancement far outpaces our ability to reason or control its usage effectively. Regulation lags misuse and eventually something (drones, nanobots, whatever) is going to lead to a massive and irreversible calamity before we change our starry-eyed rush to embrace the "new".
Sometimes I think the Amish have the right approach, though a little extreme. They aren't 100% anti-technology but meet every year to vote on if something new should be adopted, only after considering all negatives and secondary effects.
I am a realist, though, so I just live my life and brace for the eventual impact.