Ironically, the quality of Community Notes is directly correlated with the toxicity and division on the rest of Twitter. That's because the notes algorithm explicitly maximizes the distance between the users reviewing each note, i.e. a political note should be reviewed by both blue and red users. So the Community Notes system becomes more robust the more users disagree with each other.
Personally, I actually think Twitter should allow users to opt into making these tribal associations even more explicit. One simple feature could accomplish that: let users choose the color of their checkmark (red, blue or gray). When asking them to choose a color, show a list of the accounts they follow that have chosen each color.