How Crowdsourcing Turned on Me (2014) | Hacker News Reader

How Crowdsourcing Turned on Me (2014)

nautil.us

100 pointsgrkvlt29 comments

So the author decided to rely on "Crowdsourcing"... without actually building any of the features needed to make crowdsourcing work... and was surprised at the outcome?

He appears to have supplied the task as a wide open Drawball or Google Docs type interface. This is great if you have some lasting leverage over the participants (such as "I can punch them in the face" or "I can fire them"), but otherwise some form of advanced moderation (Wikipedia with moderator bots) would be step zero, and a validation process would be step 1. The possibilities for, eg, anonymous drawings of dongs, are early and obvious in any open system, to the point that I start out collaborative document editing sessions in gaming by telling people to get it out of their systems at the beginning.

Mechanical Turk is great, but the cost of paying people <$1/hr is you have to assume some percentage of them are unreliable or bots just trying to make money (if not actively malicious), and designing this expectation into your workflow is nearly the whole task at hand. There will be some multiplier where you pay for c*N hours of labor in order to perform N hours of work. Randomized peer review methods combined with long-term engagement can keep c quite low, far lower than management expenses in an enterprise, but it will always be greater than 1.

It would have been interesting to add in a validation layer, letting the crowd evaluate each others moves, creating a virtual trust score for each player. Those with high trust scores get to meta-moderate others moves before those moves can be applied to the board. You can also try to catch trolls here by presenting them with a known good matching and the system blackholes those who reject it. Similarly, if their overall trust score dips below a certain threshold, whether due to poor skill or malicious activity, they get blackholed then too and any subsequent change they make to the board is only visible to them privately though they think they're causing havoc.

Combine a system like that with a requirement that N (some reasonable number of randomly chosen people) must agree when combining pieces or splitting pieces before the change is applied to the master board. And add the ability of the project admins to scan the board and lock in place obvious good solutions. That might work.

We have such systems in place - see Wikipedia and/or Stackoverflow. Both platforms IMHO have become stagnant messes. A group of people who spend waaayyyyy too much of their life on those platforms and by now identify with it way too much get power-hungry. It's "their" platform now. While still being able to deliver some value a lot of the fun and "feeling easy" about or when using the platforms actively (instead of just passively looking at what they produce) is gone.

It's equivalent to managers or politicians who are in power for too long. Basically, the 0.1% who contribute the most are the ones who should actually get thrown out of "office" occasionally.

I disagree. Moderation plus meta-moderation would work well for the singular task under discussion: putting shredded documents back together. And because there's no usernames or discussion involved (displaying "User happykitten92 made this move. Was it correct?" would defeat the purpose of preventing troll collusion), everything is done anonymously so nobody has a fiefdom they are protecting; the value is in accomplishing the goal.

To add to this, the concept is not exactly new when it comes to crowdsourcing. Casting Words uses these approaches with Amazon Mechanical Turk for validating others audio transcription for accuracy, formatting, and so on.

You can also validate actions anonymously, by obscuring the user name at the point of validation. We do that on our site.

> A group of people who spend waaayyyyy too much of their life on those platforms and by now identify with it way too much get power-hungry.

Though, what is "waaayyyyy too much"? Someone who consistently responds to a question on SO every day or checks 10 pages on WP for language issues every day?

And are long timers really power-hungry or just fed up with discussing the same things over and over again? (Not saying the latter is better.)

I'm sure what you say is true for some, but I think the vast majority is actually civilized and most of the time quietly doing their thing for a project.

What I am missing with most "power awarding" algorithms is a time factor: Respect has to be earned again and again and some "powerful" person who has been inactive for a year can easily come across as condescending to someone who has been very active and engaged in a project for just the last 9 months and thus does not have more than baseline respect for the old timer because s/he never saw him/her in action. An example of a, in my opinion, better solution would be to lose 10% of your karma for every month of inactivity on HN (just an example, I don't think HN suffers from the problems described) or removal of the "auto patrol" flag for users who have been inactive on WP for some time.

Idk about stack overflow but wikiedia degenerated into a bureaucracy, especially evident in the smaller communities (i.e. not the english)

my definition of "waaayyy too much of their life" in the context of this topic is, people who don't have much going on IRL due to social adjustment impairments and personality disorders discover these online communities and have their emotional needs met by fitting into an online community.

Nothing wrong with all that, some hermit toiling over research into an obscure topic can do great work, but the same sorts of neuroticisms and anti social behaviors that can plague real life can more easily over plague online.

Choose the lesser of evils. It's not possible to maintain the easy and fun atmosphere of a tiny community once it grows beyond a certain size.

Personally I don't share your opinion on Wikipedia and Stackoverflow. Sure they've lost the new car scent, and there's a lot of noise and bureaucracy, but they're still tremendous resources.

Most crowdsourcing systems hinge on trust. Too much trust, and exploitation and sabotage will occur. Too little trust, and nobody gets excited.

You cannot just give trust away like hugs. Trust must be earned. Trust is a social construct, and if that's not built into your system, your system is bound to fail.

Almost everyone will be honest and genuine, but that's not enough. It takes just one asshole in a thousand to burn down the building, if they have nothing to lose.

This story is interesting because of the hate generated in a small number of people around the perceived cheating.

Or maybe the stuff around anonymous crowds will always contain destructive assholes.

People might be interested in the previous discussion of this article (only 25 comments) https://news.ycombinator.com/item?id=8499452

Here's a different post about the 2011 challenge, with some interesting comments: https://news.ycombinator.com/item?id=9021383

And here's another one with 50 comments: https://news.ycombinator.com/item?id=3164466

Even non anonymous crowds contain destructive assholes - see plenty of anecdotes and write-ups of businesses. Identification however lets traditional social mechanisms handle the issues to the extent they are able to handle them in real life. The other examples in the article mentioned friends recruiting friends based on utility which tends to be a really good way of preventing dodgy behaviour. The same effect has been used in micro financing in South America where social reputation and therefore stigma is used in lieu of traditional loan guarantees.

The previous submission had exactly the same URL and title, so why did the duplicate detection system allow me to submit this? Still, there's some interesting discussion there, thanks for pointing it out.

Just for clarity: I don't care about duplicates. When I post links to past discussion it's only to point people to comments that they may find interesting.

I think the dupe detector is weak to allow interesting articles more chance to get substantial discussion. This is an interesting article, and it would be (in my opinion) a shame if it got caught in the dupe detector. (But obviously I'm not a mod!)

Because it was 584 days ago. HN lets you repost something after a while.

If a link was posted that long ago maybe the original year could be automatically appended.

Usually the year means the year it was posted on the source site. That doesn't necessarily mean it'll be posted here the same year.

The shredding sabotage seems very interesting. There is probably a lot of unexplored space in creating resilient (self repairing) crowds and/or the tradeoff between opening up the crowdsourcing process and security/safety concerns.

Yes. I wonder why they didn't just roll back the attacker's moves? It seems like the problem is one of those NP-hard type situations, where the solution is very complex (hence crowdsourcing as a brute-force solution, rather than anything algorithmic) but checking a proposed solution is tractable. You might be able to validate a move (or perhaps a block of moves) and either keep or reject.

After reading the attacker's note, which I'm very thankful the author included, I'm inclined to agree with the negative perception of "crowdsourcing" in contests as mentioned. There's merit in researching the effectiveness of groups, but, I mean, sort of harnessing a mass of people for individual ends can be altruistic or simply a mechanical turk like the examples, but it can also be used for mob-like tactics. What I'm saying is that it's simply such a wide platform that I agree it has the potential to solve most any problem thrown its way, provided the conduit leader provides the right incentives.

As for those who might take issue with the style of the writing, hey it's just got some flourishes. A bit above and beyond the usual newspaper dryness. I can appreciate the effort to put together something with more than just a one sided, personally glorifying take. This felt a bit more in touch with balance on its own.

I'm not sure I agree with the tactics, but I agree with the "attacker." It seems pretty obvious that the intent of the challenge was to find an efficient automated way of reproducing the shredded documents.

The author even points out the movie where a bunch of people put together shredded documents. So everybody already knows it's possible with enough people working on it. But it's boring, labor intensive, very expensive, and doesn't scale at all. Sure, it's faster on 5 documents over a few days when everybody's excited to be competing, but how many people are going to stick around working for free for 50, 500, or 5000 documents when the system is used in real life?

The tradeoff between verification/validation and openness is definitely a tricky one. I'm the co-founder of a crowdsourcing platform prolific.ac and we settled on system with minimal verification (via sms) with background screening and shadow banning. It's been effective so far at limiting bad actors though I think there are definitely some more effective ways to address this problem.

One could look at the crowdsourcing strategy as a darwinist approach... It works in nature...

It's hard to see how the author thinks that "harnessing" humans can be a legit solution for this type of challenge.

Think about it... putting harnesses on humans, if you take it literally, actually sounds pretty disturbing. I realize it's not literal, but the mind-numbing work he has envisioned for large numbers of people is not something I would wish for as a future solution for anything.

Well, that's not a very good point.

Reply on news.ycombinator.com