What sort of arbitration process do you have for judging the quality of evidence and the degree to which evidence agrees with hypotheses?
What's to stop trolls from spoiling the process through the above methods, or through adding extraneous hypotheses to confuse the process?
Here are some thoughts on the points you bring up:
- You can currently dig into the evidence quality by looking at the corroborating/conflicting sources that users have added. Users can also tag sources based on their quality, e.g., state-sponsored media, secondhand information, etc. Eventually I want to arrive at some metrics for evidence/source quality.
- The degree to which the evidence agrees with the hypotheses is handled by the evaluation/assessment process. The site merges the evaluation/assessments from all the participants and highlights areas of dispute. In the future, we'll probably end up allowing users to mark which evaluators they trust (e.g., verified journalists).
- As far as trolls go, we're exploring the space of moderation and anti-troll measures. This will include automated measures like rate limiting, as well as moderation features like flagging evidence/hypotheses as irrelevant, duplicate, etc.
A lot of potential ideas are captured on the issue tracker: https://github.com/twschiller/open-synthesis/issues