Hacker News Reader

Comment by danicgross | Hacker News Reader

Pioneer founder here. As noted by almost all commenters in this thread, we do rely on expert voting. It's in our FAQ. We take pride in that! I believe mixture of moderation _and_ quantitative metrics are the correct path towards accomplishing our goal.

We've all seen what happens when you let the algorithms run free.

We've got much to fix in terms of making the game more _fun_. Point systems are best when you _understand_ the underlying mechanism. There's a tension between that and moderation. Losing should feel educational, not frustrating. We've got some ideas on how to fix this, and we're burning the midnight oil implementing them.

(BTW, we're very open to any product suggestions here! In case you're wondering what the ruckus is about: the Pioneer idea is to make an online game that rewards productive, creative behavior. Which beckons the question: how do you quantify productivity in different domains?)

View on HN

When one component of the system has infinite power over the other (moderation over quantitative metrics here), you can't call that a mixture IMO.

I would suggest these three things: (1) separate the concept of "experts" and "organizers/developers". When I hear "experts" I assume "independent experts". (2) change deceiving language on the landing page. Quoting: "All you need to do is convince other participants that your project is worth doing." and "Every week other participants will give you feedback and points. The more progress you make, the higher your score will be." No mention of experts, yet alone organizers at all. (3) Make the points transparent. That includes breaking down voting vs experts vs whatever and disclosing the details of the scoring and matchmaking systems. Doesn't have to be exact spec/formula obviously, but general details.

Thanks OP! What I find curious is why we couldn't continue our existing exchange over email.

(noprocrast correctly suggested I get back to work, so made a new account.)

Well, I wrote to you (Pioneer team) multiple times over this week. I actually sent all the materials for this article yesterday.

Edit: re-read your comment more carefully. To answer: because the exchange was not productive. I only saw general comments equating to "you don't understand our opaque system" without even acknowledging the arguments I was making.

To add to my previous comment. What separates "expert voting" from "leaderboard manipulation" the most in my opinion is the access to the leaderboard state. If experts vote in isolation, based solely on the project info and reports it's a "mixture of signals". If they have unlimited power, and change the scores to shape the leaderboard to a desirable state, it's manipulation.

I agree that pioneer is super sketchy about it's exper scoring system. I think in practice it's more like peers float your app to the top of the applicant pool and then experts don't bother themselves with the ones ranked lower, so it's not as off the peers are useless. At least thats what I figured when I applied last year, and gave up when I realized that the experts were not really selecting "outsiders"

Protip : never try to hide your shortcomings. Embrace criticsm and scrutiny. If you do it right, these instances should only help you become a better team and build better products. How you respond to scrutiny pretty much gives an idea of how you handle yourself when the going gets tough, and comments such as these just don't inspire confidence.

Do experts take into account previous performance and overcompensate current score to ensure previous top players are pushed out the top 10?

Data seems to suggest this, which is going to extreme lengths beyond “expert votes”. And is more akin to changing the rules at the last minute to avoid living up to the promises put out by the platform.

If this is not the case then perhaps you can release more data to illuminate the oddities highlighted by the article?

You should probably put the FAQ link somewhere other than the very bottom of the page, it's quite easy to miss as-is.

Reply on news.ycombinator.com