I'm sure it only took a couple rounds before people started noticing you (and some alums) in the cap tables and realizing what's going on. Word spread. I have no doubt that now it is considered a signal.
Investing is like dating. You need to be right for them, and they need to be right for you. Just because you don't want to invest in someone doesn't mean they're a bad investment. Smarter angels & VCs probably realize this, but there's plenty of perfectly good money that would not go to companies who might be a good fit amongst the less savvy.