It sounds like you are a people-pleaser. This is a common problem that I shared. The thing to learn is to have, state, and enforce boundaries. Otherwise some people pick up on a lack of boundaries and will do what they wish with you.
Something that may help is to recognize that you are allowed to disappoint someone. You are not the sole cause of a situation. In a problematic situation, sometimes the best we can do is to have boundaries and see whether people want to adjust their behavior or leave. Simply having articulated boundaries will often cause people to be less bothersome (with you, anyway).