It's not really a needle in haystack problem. If you record every moment of all your citizens. You can build a calendar application where you enter the name of the person and go through all the moments in question throughout the days of curiosity. Specifically when that person is under suspicions of some criminal involvement. I feel sorry for china because I can only imagine how much more data besides just phone calls are collected from citizens unaware.

Let us hope that you are correct. We would still need to fortify laws against data collection that is fiercely enforced (although already being a requirement in many constitutions in the western sphere). That would require oversight with severe consequences for any program breaking the rules. Personal consequences up the chain. Intelligence agencies have morphed to petty criminal gangs. Even if you can easily argue that they have been criminal before...

Conversely, their vacuum everything approach could have landed them additional evidence against Russian hackers.

I think a lot of people at the NSA are relieved to not be doing this anymore. These programs were started post-9/11 when we had no idea whether they would be effective but we felt like we needed to try them or at least try something because they might be the silver bullet that prevents another 9/11. People knew they were constitutionally dark gray and fundamentally un-American on day 1 but they went along because it might help. Many of the more senior people in the early 2000s were old enough to recall the Hoover era and everything that came to light after it. Now that we've proven bulk collection doesn't work we can finally put it to rest. With the increasing popularity of encryption it was getting long in the tooth anyway.

I also think that the established political parties are realizing how much power truly does lie in the executive branch and how long four years really is. While the current example being used to teach these lessons may be highly repugnant to the current political elite (on both sides) he is not doing anything that cannot be undone relatively easily.

The bigger the program the shorter the time it will stay secret, a nation scale metadata collection program is necessarily big. Congress will find out about it eventually at which point they will not be happy (Feinstein and a few die hard authoritarians excepted) because said program is all downside and no upside (as proven by PRISM's track record).

Nobody in Washington wants a time bomb like the calendar use case you described sitting around lest some future far left or right president use it in a manner that inadvertently starts a civil war.

That said, I also fully expect that they've moved on to some equally nefarious shit without telling us because that's what intelligence agencies do.