You may not worry about this, but others might (and I can think of scenarios where this would be a very bad thing), and it certainly isn't apparent to most people that if your phone gets stolen, the thief has access to your location history.