Yes, primarily-political sites are downweighted equally. And yes, there are other kinds of site that get penalized. For example, most major media sites produce more lightweight articles than substantive ones, so are marked that way in our system. I should mention, though, that there are plenty of ways for such penalties to be overridden, besides moderators turning them off manually. For example, they don't apply to submissions by users who have a track record of submitting solid stories.
Some of this is algorithmically determined and some of it is manually curated. The more complicated a decision is (e.g. is site X primarily political), the less likely it is to be algorithmic.