Exactly the same here. I think it's not really "being a dick", but just a dissonance between how the different types of people operate. I'm usually so passive that any sort "No, I want my free time after work" feels rude, and I'm probably not very good at being assertive and nice at the same time yet, haha.
The other developers always seemed to get the message that when I said "Maybe... I'm not sure if I have the time for it... I can see...", I really meant "No, but please don't make me say no.", while anyone else hears it as "Cool, he said he would see if he has the time for it" and then follow up with me a few days later "Did you make time for that project?"
I'm making an effort to flat out say "No" more often, but it's nice that other people are also beginning to realize that I only mean "Yes" when I actually use the word "yes".