I'd be more concerned if the intelligence agencies of the world WEREN'T doing this. It's their purpose. It's what we pay them for with our taxes.
Of course we should also always root for tech companies to stay one step ahead. But infosec is an arms race, and I sure hope my own country's intelligence agencies (CSIS and the CSE in Canada) are doing their best to stay ahead of, say, North Korea or Russia.
(Please note: I'm not saying violating our privacy is OK, any more than I'm saying it's cool to launch nuclear weapons. But if anyone's going to have the ability to hack my phone or launch a nuke, I want it to be the people on my own team. This seems like basic self-interest and survival strategy.)