If one person holds the signing key to do something critical in your system, they're both a single point of failure and a huge security risk all in one. If you distribute that key to, say, 5 different people you've mitigated the single point of failure. But now you have 5 folks who can act potentially unilaterally.
Using a 3 of 5 TSS setup, you've still mitigated the single point of failure (any one or even two folks can go offline and you can still operate) while also protecting against unilateral action. It's a mathematically-enforced version of the "two-man rule." Similar to the way Cloudflare's Red October tool used to work by splitting things between parties: https://blog.cloudflare.com/red-october-cloudflares-open-sou...
Or one that's checked into your version control (representing that it is your company's code that's running) and one that lives on the server (representing that it is a server your company controls).
Or to take your example - a key in the repo, a key from the dev, and a key from the build server.
Now, you need to compromise at least 3 safes to do anything on the HSM. If a token is lost, stolen or damaged, there are still 2 others which can be used. Loss or theft of any two tokens does not compromise the HSM. Any unauthorised access to a token can be detected due to the tamper evident bag.
This gives you strong protection and assurance against any malicious insider or attacker. Having a single key (even if protected in a safe) is much weaker, and carries the risk of key loss (so you would have to have token backups, multiplying rather than dividing the risk).