It's going to have to be on the blockchain. No way around that.

Any other way you end up with the same outcome. If one party has full control, it will eventually abuse it's position.