I am not saying that I wouldn't mind at all if my phone was searched. But I can't think of anything in particular that I would be concerned about if it was. Sure in theory the agent could remember some personal information and come back later and use that info or pass it to someone for some nefarious purpose. But that's a pretty small chance event. It's more or less the equivalent of thinking you will get sick because you find a hair in your food at the restaurant. You don't like it and you send it back or demand a refund but the actual harm is more mental in nature.
Or what am I missing here?