When you stop using signal, you can log in to the signal website and say "I'm no longer using Signal"; that will solve the problem with your Signal-using contacts. I just checked and it looks life you have to deregister your account from within the Signal app (ie, can't uninstall first). I'm pretty sure it used to be more lenient: I remember friends being able to deregister their numbers after the fact.

It looks to me like it's a per contact switch, so I think that's what you want, yea :)