They shouldn't invite you over for dinner again, but it ends there. They don't get to come for your job, your career, your legacy.
No. This is equivalent to a threat to strike. RMS without the GNU project maintainers is just a dude with a blog. And this approximates a vote of no confidence.