Its worth hiring people full time to do high quality labeling. Benefits, 401k, the whole 9. A good technician who can interpret nuance has value.
I want to highlight this phrase, because I think it is critically important and frequently missing from the worldview of the technical workforce. It is a serious risk that is created by the narratives around STEM education - especially engineering. Good technicians are incredibly valuable.
My first job out of engineering school was in a semiconductor factory. From the day I walked in, I was absolutely dependent on the technicians who worked on the tools to do my job well, as was every other engineer. I could do things they couldn't, but they absolutely could do things I couldn't. There were two types of early career engineers there (1) those who valued what the technicians could teach them and built those relationships and (2) those who did not. That second group struggled. They struggled because they didn't know how to listen to someone who they perceived as having less knowledge/education/value compared to them. Some of them got really upset when they found out senior technicians (typically with an AS) earned more than junior engineers (BS/MS). The reality was, the young engineers were a hell of a lot more replaceable and much easier to automate.
Google recently published a paper on this topic[0] and it is now a required read in the basic and advanced statistics courses that I teach, because a lot of my students are very excited about getting into machine learning roles. Unless you have quality data and information, the complexity of analysis you do on it largely doesn't matter.
As if I haven't outed myself to any of my students reading this...we do it on the first day in my stats class:
Spend the day collecting data using the board game operation. I tell you the categories of data to collect, I don't tell you exactly how to collect the data. Then you clean that data for homework...the discussion about whether that data is good quality creates a lot of insight.
Data labeling is one job that seems to lend itself well to gig work. There were many times where I would have gladly spent a day labeling images in exchange for $50, no need for benefits
I fundamentally disagree. Its not the size of the dateset that determines performance but the quality. Sure. Bigger is better. But simply better is *way* better. I can do more with 1000 high quality examples of the thing than I can with 40k low quality ones.
I've gone as far as bringing on 40 digitizers at once full time. I would never have been able to accomplish that project with turks. I would never have been able to call a team meeting and explain a very specific shift in interpretation we would be doing regarding the relationship of feature apple to feature banana, and when and where it would apply.
The difference between thinking you need oom more examples over a few quality ones I think depends on how in the thick of it you've been with regards to ML on theoretical grounds versus doing the boots on the ground work to get a product out. Having control over digitization and labeling and being willing to pay for high quality label's has allowed me to get results that directly meet my clients needs as opposed to sitting in some uncanny valley where the results are close, but no cigar.
1) I spent $100 on 1000 labels
2) I spent $100 on 500 good labels and 500 bad ones and can’t tell them apart.
If I spent $0.20 per label on 500 labels with good better accuracy I end up spending less and getting more.
The only way what you said makes sense is if we ignore the quality of the data as a consideration.