Because it’s no longer iMessage, but SMS. The other phones are sending to everyone they think is in the “room”. So you’ll be re-added. You’ll just have to tell them to stop texting you.