So to access it, a person needs access to either your phone or computer. But if they have access to your phone or computer, is a cache of cell towers you've connected to really a significant concern compared to the other things the accessor could get ahold of from either of these devices?
Timestamped travel information generally equates to "hey, this guy returns home between 7 and 8:30 most nights, works in the office five days a week, and here's where his home is". It's no great stretch to equate that to "he will not be home during this time frame" and suddenly a lost phone turns into a much larger burglary.
There's also plenty of larger privacy concerns, but even just the out-and-out-crook scenarios should be concerning enough.
Please read my previous comment for more details:
You seem to be using this claim to dismiss the importance of the data, but I lived in a very secluded area, do I not matter?
1) Your cell number is going to be readily available from the device itself. Using this, he'll be able to look up your address information in the Address Book.
2) Unless you notice the missing device right away and change your passwords, this guy also has access to your email accounts. Ever order anything online? Your address is most likely in your emails.
3) Ever use Google Maps? They can look at your recent searches. "3 different searches originating at 123 Main St? Might be his home address"
4) Phone app. Recent calls/favorites. With a little social engineering, your address is known. "Hi, I found this phone on the ground, can you tell me where they live so I can drop it off?"
5) SMS. Ever text someone your address?