The key is that there is a process for reconciliation. It IS NOT ENOUGH to simply have a ledger - you also need a mechanism for enforcing that the ledger matches reality. And reality is complicated, filled with reasonable disputes over terms and deals (in the best case) and outright fraud and theft (in the worst case).
A given party may be able to temporarily pull money from you with two numbers, but the process around reconciliation makes it so that you, the customer, are protected from the actions of the mediary (because at the end of the day, they're selling this service, and are responsible for their actions in relation to providing credit). This incentivizes those institutions to be careful, protect their reputation, and avoid taking on obvious risks.
Essentially - the system is structured in a way where incentives align to prevent abuse. And entry into the playing field is expensive and limited enough that institutional reputation matters.