Scoring Open Source Contributors in the Age of AI Slop: Finding Good Eggs

Jeff, the author, here. We built a tool that scores PR authors by mining their contribution graph from the GitHub API. Every input is a merge/reject decision a human maintainer already made. It doesn't look at PR content or try to detect AI usage. It just answers: has this person gotten code accepted into projects before, and how relevant is that history to your project?

The scoring is graph-based (bipartite user-repo graph, personalized ranking, 180-day recency decay). Scores are context-specific, so the same person can score differently against different repos. The post walks through how Guillermo Rauch scores MEDIUM against his own company's Next.js repo because he has zero merged PRs there, and how v2 rescues that with merge rate and account age.

We validated on 5,129 PRs across 49 repos. Three features survived statistical testing, four didn't. The most surprising failure: text similarity between PR descriptions and project READMEs predicted lower merge rates. We published all of it, including the failures.

More detail on the Substack post.

Repo: https://github.com/2ndSetAI/good-egg (MIT, pip install good-egg). Runs as a CLI, GitHub Action, Python library, and MCP server.