It sounds like this guy helped you out of a tough situation and I can understand why you've struggled to say no to him.

But just because he helped you out doesn't mean he owns you until the debt is paid.