Off the top of my head, here are some possible concerns and avenues for abuse.
1) Today, the government needs at least the pretense of a warrant or at least legislation to allow them to legally snoop into your financial affairs within a bank. Spying on you would probably be a lot easier for them on some kind of central digital currency that they'd manage. Want to buy a 12 pack of giant buttplugs or a Quran? Uncle Sam is going to know about it in our brave new centralized currency world.
2) Cancel culture and removing classes of people from the economy for wrongthink views, or even just actions that their politicians take on their behalf is already a thing. It's not hard to see a future authoritarian government making financial life for their political opponents hell.
3) Programmable centralized money allows central bankers to have unprecedented control to impose policy that would have been close to impossible to implement before. Keynesian thinkers for example view the idea of money flowing through the economy quickly as one of the big avenues to prosperity. It's not hard to see them clamoring for effectively a tax on money that sits in your wallet too long without being spent to promote consumption, or any number of experiments now that the tool exists to make it possible.
You talking about "the government" as a unified entity seems odd to me here. The various bank regulators (FDIC, OCC, and yes, that same Federal Reserve) already have access to huge data flows. If they are already misusing it, it doesn't sound like this would be worse. And if they aren't, then I'm not seeing why that lack of misuse would change.
1) Even assuming they get all bank data (which can be assumed, but is uncertain) they don't get access to physical cash data, which might also be phased out in the event of a national digital currency.
2) Nothing about your comment addresses my concern about government power.
I think concern about government power is valid and important. Which is why I think it's vital to save talking about it where there's actually a material difference in government power. I don't think it is here, and you haven't given any reason why it would be.
A central bank digital dollar would be IOUs from the central bank. It's closer to cash.
However, it puts the central bank being the only bank that handles the clearance/settlement. So it has higher political risk for holders. For example, you can store your dollar bills into a bank which does not follow the order from US government. But if you have CBDC, the federal reserve can simply freeze your balance.
The C in CBDC is Central. The government wants this centralized not decentralized. They will be able to create any amount of CBDC currency.
The government will almost certainly push, along with this type of CBDC, for them to have far greater and more immediate control over your finances. In other words, any one of many government officials will be able to freeze your accounts (or drain them) at the push of a button. Hmm... I think that one of those three people did something bad - BAM - all three CBDC accounts are drained and frozen. Too bad, due to seizure laws, you can't ever get that back.
All of the advantages of having a physical note or a $ in a bank account disappear with this system. The government will control the ledger and everything that happens to it. There won't be money - just entries in the government's ledger. You won't be able to walk around with cash or even purchase an ice cream cone for your daughter without the government knowing.
That's in the best world. Just wait until the CBDC gets hacked like BTC or ETH. I doubt the government will return your govcoins.
PS. The document mentions "key stakeholders" in the CBDC. They aren't talking about people. You and I are not key stakeholders in this endeavor.
With digital cash it might be more difficult, potentially intrusive or costly to convert to physical assets (look at crypto). If I take $20 out of my bank account and buy donuts, it's going to be very hard to the government to trace that. That will not be the case for digital cash. They will know the exact wallet that came from.
I also believe the need for the Fed for a CBDC is forced onto them because of the inevitability of a negative interest rate on savings accounts (which carries negative consequences). Indirectly, CBDC solves that for them. This person explains it practically: youtu.be/UP0e9MGjyD4
A CBDC backed by a proof-of-work scheme is unforgable, meaning said currency could be transferred between multiple banks (even potentially abroad) without having to worry about whether said bank obeyed all the regulations correctly or not.