Yes, I got this. But if your server is compromised, then the attacker can erase your backups, which ruins the point of doing backups in the first place. This is why there is an append-only mode.
And being careful is difficult, because in Borg, once you do any other backup command from a full access account (for example to do pruning) - it will automatically, no warning, go through the log and apply it. You should really really read up on that functionality first before relying on it, the way Borg has implemented it is close to anti-feature.
Regarding compromisation of strictly the server itself, I believe there are commands to check the state of the repository? Isn't that enough?
It mounts the machine to be backed up and runs Borg on that.
The backup machine does need locking down in of itself but that’s a lot easier to do than locking down something public facing
The snapshots are not deletable even with your full credentials.