This is what bothers me. I think there is a lot of evidence Apple's claims that they keep as much of this data on-device as possible. Even so, Apple's increasing dominance in certain aspects of my life bothers me.
After watching Google progress from being a company I loved and respected into a gross corporation, I'm concerned will eventually follow suit. Even if you assume Apple's current management are a bunch of great guys, in 10 years Cook might get replaced by a corrupt douchebag.
It's concerning. But at the same time I appreciate the previous posters view as well, I'd rather not have my information painted across dozens of small companies who are easily acquired.