Metadata analysis can deduce quite a lot about you. Your sexual predilections or your political or religious outlook, for instance. Who you spend time with and where you spend it and what you do when you are together, for another. Who you care about, who they care about, who they spend time together with...
Not only do you have to trust that the people who have access to this data about you will use it only to advertise to you, but you also have to trust everyone who will ever have access to this data in the future, too.
You might dismiss that there is any inherent risk to total strangers knowing intimate details about you, but I personally think the risk of this is a "specific harm".
I think your stance is a version of "I do nothing wrong, therefore I have nothing to hide, therefore surveillance is nothing to worry about". With respect, it's naive.