The thing is, you don't know about the abuse until its too late. Plus he's retired at the time, he'll do some 'consulting' and maybe speak for a lot of money but he'll have his security detail and he's done.
Kind of the ultimate abuse would be some future presidential candidate's records being gone through and perhaps in his youth he had some questionable friends or something. Our politics and politicians are not nearly civil enough to have these kinds of tools and information available. It's basically just a hope and a wish that the NSA folks with access take their jobs seriously and take the nation's security seriously and that they are otherwise trustworthy about it. Not long after 9/11, anthrax, anthrax that originated in a top secret US defense lab was sent to politicians in DC... In Fort Hood Texas, a member of the US military, a guy they vetted and indoctrinated, went through their background checks and they accepted shot up a bunch of people.. These were trusted people with trusted access.
The knives are already out, should we shut this stuff down, as soon as there is another terrorist anything the other side will use it for political gain. How do you undo it?