You may not like hearing this - setting boundaries is a skill, and a difficult one to learn. It's also one of the most valuable skills for you, especially you personally, to learn, based on this comment.

It has to be a really really small place for "you're the only person we can say yes with" to be a fair note on a request. That doesn't mean there aren't plenty of people stuck with such jobs at bigger places, but it doesn't make it any more reasonable an excuse and pretty much still boils down to constant fear of being dismissed if you say no in the end.