I know it's perfectly fine, I'm trying to follow this (very common) line of thought and adherents never seem to explain it in a way I can understand.

For one thing you're not really 'censoring' anything but even if we say, for the sake of argument, that you are why should it have some cost different than the cost of 'promoting' something when you upvote? The mods' argument that meta is bad for actual conversation is also quite compelling. What's the counter to that other than stuff that mostly seems to boil down to some version of 'getting downvotes kind of feels bad' (of which calling downvotes 'censoring' seems like a particularly overwrought variant).