Comment by arecsu | Hacker News Reader

Comment by arecsu | Hacker News Reader

Makes me wonder if an ELO-based system would work to mitigate these issues. People who merged PR successfully onto a project, that had real issues acknowledged, the quality of their responses measured by other users reactions or something, etc, multiplied possibly by the degree of importance of the project where their activity has been made. Won't be about human vs AI, but actual helpful effective being vs low effort/spammy contributions. Issues and PRs could be sorted and filtered by their ELO score. I'm saying ELO as analogy to "score based given the context", not really a 1:1 translation of the ELO system.

Negative score would be reports from other users because of spammy content or not acknowledged issues, with a middle ground of neutral score (+-0) or little positive score to issues or whatever with clear good intention, but couldn't reach a proper merged PR or were not issues (e.g. issue existed but wasn't the correct repo to be addressed, PR was good but needed other stuff to be implemented prior to it, maybe in the long run, etc)

ELO is shockingly easy to manipulate. For example there was a literal jail with a decent chess player in it. He created a pool of players who got great ELOs by beating him, then used them to boost his rating higher. Wash, rinse, and repeat.

Given any manipulatable scheme, AI will figure out how to manipulate it. For the OP, what happens if a single AI manages to get through to contributor? Then it starts elevating other AIs to contributor, and we're off again. There doesn't have to be a purpose to this. Trolls will troll, and trolls armed with AI bots can devote endless energy to doing so. The more you work to keep them out, the more fun it becomes for them.

I wish I had an answer for that problem. But I don't.

Reputation scores, review cartels. This all sounds familiar!

fix this problem by make the rating value tied to some paid currency - a repo owner would have to pay for the PR, and that PR contributor will now have more currency than previously. In order to have said currency to pay, the repo owner would need to have contributed to another repo whose owner have currency.

The totality of someone's currency is their reputation.

Of course, now the decision becomes...who is the central currency issuer that creates it?

It's the StackExchange model! This has bootstrapping issues, is hard to break into the community, and risks creating moderator cliques.

This is called proof of stake

ELO is a bad fit because it requires competition between submitters; but if the idea is interpreted as “contributor karma score” or similar (not everyone’s familiar with the mathematical nature of ELO), then the way to close the loophole is to only consider voting inputs from the human project owner. This project chose to have people lie to a webform rather than lie to a git interface about using AI, so I don’t expect it will be particularly successful at inhibiting AI use by project-involved humans, but certainly it’ll squelch a lot of noise from unattended/passersby.

I think they were saying Elo system as kind of a general ranking system idea instead of the actual algorithm.

You could probably use some kind of pairwise ranking algorithm (like anything based on the Bradley-Terry model) to rate human vs. AI contributions, but that would take a lot of manual effort. Google is using it to (supposedly) improve their searching algorithms. They give testers two different versions and ask them what's better.

contributors being able to grant contributor to other users seems like a problem

>what happens if a single AI manages to get through to contributor

Then they'll get removed by the humans? Its about cutting down work, not about eliminating the work entirely

The current approach removes about 99% of their overhead it would seem. If they have to do a few manual interventions here and there, that seems like a huge win overall

For those wondering what Elo means, it is a person's last name, not an acronym (not all caps). More info here:

https://en.wikipedia.org/wiki/Elo_rating_system

Thank you, big fan of ELO https://en.wikipedia.org/wiki/Electric_Light_Orchestra and I was a bit confused about the comments.

It's Elo not ELO. Elo is not an acronym.

https://en.wikipedia.org/wiki/Elo_rating_system

From what I've seen in the comments, it's definitely ELO, if not through ubiquity alone. Happens to the best of 'em!

Elo is nicer as it gives a nod to the inventor, no?

Some say the greatest compliment a mathematician can be paid is when a concept that bears their name is so ubiquitous that we no longer capitalise it. (E.g. Abel with the abelian group).

That’s a fun fact!

I have built something like this and in process of collecting the data.

Frontier users: 527,865 Light indexed: 527,865 Ready to queue: 9,083 Fast scores ready: 0 Activity events 24h: 30,266 Fast scores completed 24h: 19,123 Deep jobs completed 24h: 3,043 Fast-score ETA: n/a Deep-hydrate ETA: 69h Stale running jobs: 0 GitHub backpressure jobs: 19,113 High automation signals: 4,608 Medium automation signals: 1,327 Completed jobs: 74,714

Biggest challenge is Github's rate limits. At this pace it will take two more months to have 98% coverage. But after that the maintenance should be quite straight forward.

Sounds a bit like Mitchell Hashimoto’s Vouch: https://github.com/mitchellh/vouch

This would just hurt new users similar to how you are unable to comment on 90% of subreddits on Reddit as a new user, because you don't have enough karma points, or how on Stackoverflow your permissions are severely limited until you do certain jobs. The incentives aren't very good in systems like this. Bots can be made to easily game the system while regular users are discouraged from even participating.

Some kind of vouching or scoring might make sense to help qualify contributions and many people have suggested similar recently. If by "ELO-based system" you meant "some kind of scoring system (not based on Elo)".

The Elo rating system doesn't make sense in this context; it's designed around collecting zero sum game results for a given community of players and building a model around it.

The problem is you want the ELO score based on work on other community projects - you can't assume good faith here.

The problem with that is that there are certain kinds of users that like to take control of community projects. And then they take control of more, and bigger ones.

There are a lot of political tricks that get used.

What is scary is that one of those kinds of users are malicious state actors. Like North Korea and Russia...

I think you need trust circles, not ELO.

Reply on news.ycombinator.com