The main concern here is that it will create a narrow bridge (i.e, the digital ID system) between people and various services and opportunities which will make it an easy target for people who wish to wield power against someone.

Perhaps your digital ID is needed to open a bank account, get a phone number, sign up for insurance, etc. Now, suppose some fascist government comes into power. They could start cancelling the digital ID's of people or groups they do not like or are bigotted against. These people start losing access to critical infrastructure.

Now, this could already happen, even with imperfect paper IDs, of course. But by making everything digital, we are reducing societal resilience towards such kind of hostility.