Who fact-checks the fact-checkers?
On the one hand, I certainly struggle to see how more crowds will improve upon what is fundamentally a problem with crowds to begin with.
On the other, Wikipedia seems to work reasonably well--and purely as a result of the community mores, not due to any sophisticated moderation algorithm or ranking structure or similar.
I'm not holding my breath, but it is interesting to see if a community-driven effort imbued with a Wikipedian-like spirit might succeed where more automated efforts have not.
Once your community or protocol "philosophically" rejects checksums it is only a matter of time before your system breaks catastrophically.
This philosophical freedom vs authoritarianism thing you're proposing is a view that is incomplete and small to the point of being false.
The actual underlying question is do you accept that grounded truth exists and are you willing to have a large and nuanced enough view to include all non false propositions and reject all false propositions.
But if you're squinting that hard, not a lot of light is getting through.
When you only have a hammer, every problem looks like a nail.
I believe that the social networks have built technologies that probably can't actually be made to work in a socially beneficial way, since they're utterly dependent on crowdsourcing and automation for economic reasons, but those tools aren't actually up the the problems they face.
The answer to "who fact-checks the fact-checkers" is "the fact checkers".
And we can use consensus, reputation, and checks against known facts to validate the checks themselves. If group A, group B, and group C all agree and group D disagrees, then we can say with relative confidence that D is probably wrong in this case. It is safe to go with the consensus.
However, if A, B, C are notoriously unreliable and have a history of distorting claims, then it is beneficial to see why D disagrees with all three. Or even not use A, B, and C as sources.
We can also judge a fact-checker's veracity by how they evaluate a claim. Do they provide sources for their claim. It's one thing to say "This will murder 40 people a minute" and quite another to say "This claim disagrees with a recent study by the CDC [link to study] that says it will only murder 39 people a minute".
The only reason to cast doubt on the process or activity of fact-checking is if you don't want people checking your facts.