The ‘invisible’, often unhappy workforce that’s deciding the future of AI
unite.ai
unite.ai
> ‘[A] large majority of crowdworkers (94%) have had work that was rejected or for which they were not paid. Yet, requesters retain full rights over the data they receive regardless of whether they accept or reject it; Roberts (2016) describes this system as one that “enables wage theft”.
So, what was the source of wisdom in rejecting the results?
You can run the same task 3 or 5 times and take the consensus option rejecting the others. But from experience unless the taggers have had special training in the current task the quality of annotations is crap. Then you have to do it all over again with someone in-house and gain nothing from the whole crowd sourcing ordeal.
Outside people tend to think anyone could label examples, but practice shows that it takes a special kind of person to do it well. Probably this is why many get their work rejected.
It's interesting we've built a system where people are allowed to be given a job and then told they're unqualified to perform it. If a construction site hired someone as a heavy machine operator and then immediately decided to fire them, they'd still be owed some form of wages.
AI needs to face up to its invisible-worker problem https://www.technologyreview.com/2020/12/11/1014081/ai-machi...
That being said, this one covers two recent research works that are quite interesting (Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation , The Origin and Value of Disagreement Among Data Labelers: A Case Study of Individual Differences in Hate Speech Annotation)
This is designed to detect workers who either did not understand the instructions, or those who don't care about those and answer randomly. But this works against intentional sabotage as well.
What ends up happening is that some labelers learn what the pre-determined questions and answers are and share these via Facebook and Discord to other labelers. That way, the other labelers can stay on the task longer while providing garbage responses to the non-predetermined question/answer pairs.
It's an arms race with labelers on one end, trying to make a quick buck, and data labeling platforms on the other, trying to get quality labeled data.
It turns out that they are a drop in the bucket. Not only is there low RoI but also the group is too weak.
I want to highlight this phrase, because I think it is critically important and frequently missing from the worldview of the technical workforce. It is a serious risk that is created by the narratives around STEM education - especially engineering. Good technicians are incredibly valuable.
My first job out of engineering school was in a semiconductor factory. From the day I walked in, I was absolutely dependent on the technicians who worked on the tools to do my job well, as was every other engineer. I could do things they couldn't, but they absolutely could do things I couldn't. There were two types of early career engineers there (1) those who valued what the technicians could teach them and built those relationships and (2) those who did not. That second group struggled. They struggled because they didn't know how to listen to someone who they perceived as having less knowledge/education/value compared to them. Some of them got really upset when they found out senior technicians (typically with an AS) earned more than junior engineers (BS/MS). The reality was, the young engineers were a hell of a lot more replaceable and much easier to automate.
Google recently published a paper on this topic[0] and it is now a required read in the basic and advanced statistics courses that I teach, because a lot of my students are very excited about getting into machine learning roles. Unless you have quality data and information, the complexity of analysis you do on it largely doesn't matter.
As if I haven't outed myself to any of my students reading this...we do it on the first day in my stats class:
Spend the day collecting data using the board game operation. I tell you the categories of data to collect, I don't tell you exactly how to collect the data. Then you clean that data for homework...the discussion about whether that data is good quality creates a lot of insight.
Data labeling is one job that seems to lend itself well to gig work. There were many times where I would have gladly spent a day labeling images in exchange for $50, no need for benefits
I fundamentally disagree. Its not the size of the dateset that determines performance but the quality. Sure. Bigger is better. But simply better is *way* better. I can do more with 1000 high quality examples of the thing than I can with 40k low quality ones.
I've gone as far as bringing on 40 digitizers at once full time. I would never have been able to accomplish that project with turks. I would never have been able to call a team meeting and explain a very specific shift in interpretation we would be doing regarding the relationship of feature apple to feature banana, and when and where it would apply.
The difference between thinking you need oom more examples over a few quality ones I think depends on how in the thick of it you've been with regards to ML on theoretical grounds versus doing the boots on the ground work to get a product out. Having control over digitization and labeling and being willing to pay for high quality label's has allowed me to get results that directly meet my clients needs as opposed to sitting in some uncanny valley where the results are close, but no cigar.
1) I spent $100 on 1000 labels
2) I spent $100 on 500 good labels and 500 bad ones and can’t tell them apart.
If I spent $0.20 per label on 500 labels with good better accuracy I end up spending less and getting more.
The only way what you said makes sense is if we ignore the quality of the data as a consideration.
For example:
1. Most data labeling systems don't allow you to communicate meaningfully with your workforce. In contrast, we prize two-way communication; your data labelers are the ones going through tens of thousands examples, so they often have amazing feedback for how to improve your data and design your tasks better. And of course, you often have questions for them as well.
2. Context matters. I can't label Spanish hate speech; someone from Mexico City often can't label Madrid slang either.
3. The majority vote isn't always the best one. Real-world data is often personalized and subjective; your opinion on a funny or angry story may not match mine, and that's okay. Our training sets and AI models should reflect that. We just wrote a blog post on the subtle nuances when considering majority votes and inter-rater reliability metrics: https://www.surgehq.ai/blog/the-pitfalls-of-inter-rater-reli...
4. Curating annotator pools. Our product is designed around helping you build custom labeling teams that you trust, who learn the nuances of your domain and stay with you over time.
I'd like to know more about the people and the culture there - not because it's easy to miss some culture thing here- but not knowing small things, the quality raters could be pushing some type of content into X category and other things into Y - and not because it's in the tons of pages of text they are supposed to follow, but because - whatever religion / other culture things are a thing there.
This increases the cost of labor, and therefore cost of goods across the economy. Its a big part of the reason why US based manufacturing is increasingly impractical.
Your point expands to all social services our government has outsourced, from retirement benefits to transportation, too.
And the taxes could be spread more progressively, e.g. to match the rate of wealth accumulation people get from having passively having wealth
The 'American Dream' is a stable job, a modest home with 'plumbing' and 'electricity' and maybe one car, plastic tupperware, maybe a dishwasher, and a safe community with basic constitutional protections and a justice and democratic system with integrity and legitimacy, decent public schools and some access to higher education.
All of that is revolutionary in the grand context of history.
The 1950's 'America Middle Class' is the dream, as it is for many migrants to the US for whom those things are like heaven.
The '$100K job, BMW and Corner Office' - that's something else, more like icing on the cake.
People forget how hard it was for the world to create the basic '1950s middle class standard of living' and frankly how delicate the system we live in is.
Within a generation we've come to accept that as more of a 'basic' condition of life, rather than something aspriational, which is a bit tragic. Most immigrants 'get it' though.
I'd guess a factor in ending this has been the rise or at least growth of an administrative or managerial class in most businesses.
An apprenticeship model lets people grow, and to some extent a similar model used to apply in business, where the executives and managers were all career industry people who started on the shop floor.
The current system is closer to (closer, not the same as) a feudal system where there is a group of nobility that manages a group of serfs, and there is not much movement between the groups, at least at the same company. Admission into the nobility is based on education and other pedigree, and can never really be had by work experience.
Interestingly, that might be the case for Fortune 500. But among the fortune 10, most CEOs/C-Suite are engineers with long tenures at the company.
Actually, if you look back at some of the most influential companies in the Valley at their peak, that's been a rule of thumb since the Fairchild days. Andrew Grove is an example of that [0].
Then along came the laser/UPC/computer system and deskilled the job. No longer did you need to know the difference, the computer printed up a natural language description of the item. This allowed stores to get much larger, because no one could know all the prices in a large store (early department stores handled the size by making you pay for items in each department, using store balance accounts to not create too much friction). This fundamentally reshaped the relationship between capital (who owned the stores and the computer system) and labor. It's actually the exact same alienation of the individual from their labor that Marx was writing about in the 1850's, just for jobs that didn't get the same attention.
Similarly, an in-law worked as a teller at a bank. 50 years ago that job would have more upward mobility because banks managed their risk locally: after many years as a teller and then as a loan officer you could move up to be the local person making the decision on loans. Nowadays large banks have centralized that with fancy computer systems to guide the decision, so there are entire paths of advancement that essentially don't exist anymore because computers spit out the answer. There is only shift and then branch manager, with once again the owners of the computer system using it to de-skill jobs and cut off promotion opportunities. (See, e.g. Better.com, using poorly paid contractors to do all the tedious stuff that can't be automated, and then letting the computer do all the high skill work.)
This is, by the by, one of the reasons that computer programmers continue to get paid well, because we are enabling capital to de-skill and bifurcate jobs better and better.
Alternatively put, we've commodified human hand eye coordination and basic speech recognition and intent resolution. This view is definitely supported by the current state of the gig economy.
That is, management was declaring that there was no value in experience whatsoever (until you got to management levels, naturally), and that they could replace anyone with a new hire without loss of performance. (After all, the way you got to be a highly paid retail person was to stick around for a long time and be pretty good at your job.) And while most managers in a retail setting aren't quite as obvious about it, that sure seems to be how they feel: employees are completely fungible, and it isn't worth paying them enough to get them to stay.
(Costco is a notable exception in the US: notice how the standard Costco name badge includes their start year on it, that is one of the ways Costco signals that they do value experience. They are also significantly more profitable per sq/ft than Sams Club, their competitor in the warehouse segment operated by Walmart. I strongly suspect that those two facts are related.)
The American dream is not about starting low and making it to the top, it's about upward, intergenerational class mobility.
Much of the world views the American Dream as America as a place where one can come, work hard, be treated fairly and be rewarded regardless of social class or circumstances of birth.
The extrapolation of this concept into a larger ethos that includes upward, inter-generational class mobility makes sense, but is just one element of an ethos based on the basic principles of "life, liberty and the pursuit of happiness."
So here is mine, from a French who chose to emigrate in China instead of the US: the idea was that you'd come from your bum-shit country into that vast new land of opportunities where everything was to build, you build it and are rewarded in a fair meritocratic system in ways your rotting homeland could never have provided.
But now, you have to get your Twitter account scanned to be allowed entry under a temporary visa (or hey, try a green card lottery), risk upsetting an untreated schizophrenic while crossing the streets, to go to a gun-heavy anti-police protest anarchists use as an excuse to empty their Molotov stock while the police, too happy to finally have a purpose for their military weapon stocks, shoot at random. And if you are at the wrong place at the wrong time, you can be sent to a private jail. All the while, any accident you would have would not have been covered by any sort of socialized insurance but you'd be under one of the most punitive international tax system. And if you're unlucky, you can always take a payday loan, and join the merry mass of indebted grassroot people who have negative equity for no reason a rational mind could understand.
The American nightmare, it seems to me :s
Mighty ironic of you to go on such a lengthy rant to bash the US off your Western European high-horse (white male too I presume?) living in China like a priviledged westerner, but where the stuff you bashed the US for is amateur level stuff.
Be careful with all those Winnie the Pooh memes or you might loose your social credit points, comrade. Could you please point out the free nation of Taiwan on the map, in public, for everyone to see please? My geography is a bit rusty. Also, how are them force labor death camps this time of year over there?
What are the odds of that?
Everyone wants the job, and there's only one of them to go around.
This is the reason you see such similar personalities occupying extremely high positions in companies. You need a certain mentality and dedication to set everything to the wayside for the pursuit of climbing the corporate hierarchy.
no, everyone does not want that job. People who do want that job reduce the equation that way, though
But yeah, I guess low-level employees should do without healthcare, they just shouldn’t be poor AND sick after all :)
If companies have to provide healthcare coverage for their fulltime employees, but not their gig workers and contractors, which type of employment do you think they are going to prioritise?
If healthcare costs were applied uniformly through tax (as is done in most countries with universal healthcare) then there would be less reason for employers to prefer one type of employment contract over another for low level jobs.
Your eagerness to shit on America made you entirely miss the point. Believe it or not, it is extremely obvious to many that universal healthcare would be a cheaper and preferential option.
It's possible German stocks are well priced, in a regulated, slow and rational market that cares about fitting the price with the value of the company, and would hold most of their current market cap much more than Apple, were it liquidated.
LOL, well regulated and rational my ass. The Deutsche Bank and Wirecard scandals (puls numerous more) proved the German government is just as corrupt when it comes to manipulating the market and threatening honest journalists, so that some rich and well connected scumbags can get even more obscenely rich. I've worked in several western countries but never saw more high-level corporate corruption than in Germany.
Oh yes, the regulation (BaFin?) totally was out of order on that one!
A lot of people here do that but at least they make good money doing it. Gig workers don’t get the money.
An advantage is that you can set your own hours and drop the work whenever you like and not think about it, because someone else will do it. Combined with working from home, it doesn’t seem like all that bad a way to make a few extra bucks? At least, if the problems in the article can be fixed.
It seems like a decent fit when something else is more important in your life and the job comes second or third.
I personally think that the fight against hate speech in the name of minorities is far more offensive than a visit to the darkest corners of 4chan. Perhaps objectively so, since the latter group is more diverse (international audience) and inclusive (low standards) than contemporary academia, which proposes these metrics as a gold standard.
But the technical challenge is not solvable as soon as there are different standards of allowed content. Everybody has to decide those limits for himself, parents in case of minors. At least that is true for universal platforms. Some people live up when they get headwind, some people might feel inhibited. The quest against hate speech is intuitive one but still very foolish.
In my experience is that people search out offensive material to be offended. There are cases of harassment that are separate from this and often people seem to misjudge this. But AI is far away from making competent choices here.
An annotator service like suggested here would mainly be a service to select the sample group to spit out predetermined results. Maybe that is a service Google like to provide. They are also just people pleaser I guess.
You could strip people of their anonymity and they will get happier in polls, more content with management. How would you evaluate the change in the data you see?
The cue here for me was "You could strip people of their anonymity and they will get happier in polls" as an unexamined backing into the feedback topic.. real people have feedback that sticks
What "feedback that sticks" is natural in digital realms, and how can it evolve, not dictatorially?
But I am not that angry and I don't think that can be read out of my comment aside from general disapproval.
I would assume that gig workers did not care as much about hate speech as some academics do and did not flag content as expected. This discrepancy is declared as bias. Fine, be that way...
> The Google researchers suggest that ‘[the] disagreements between annotators may embed valuable nuances about the task’.
On that I agree with the researchers, but would propose that any annotation (hate speech yes/no) would have to fall back to the 'no' and solve the dispute. Otherwise only asking the target will provide any additional understanding. Perhaps asking as supreme court too, but that is not feasible and not even the highest courts are infallible.
Lots more, it seems, seek out people to try to deliberately offend / intimidate others.