You are "flagging spam/not spam in those lists" but Google may then associate that list (not the content) as being spam or not. Google sees that mail comes from the list and decides based on that.

This is why I never mark mailing list email as spam, even if it is in fact spam.