Comment by echen | Hacker News Reader

Comment by echen | Hacker News Reader

Parent Full thread

I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a startup solving this exact problem (https://www.surgehq.ai), and previously built the human computation platforms at a couple FAANGs (precisely because this was a huge issue I always faced internally).

We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis, text categorization, etc), but also a lot more advanced stuff (e.g., search evaluation, training the new wave of large language models, adversarial labeling, etc -- so not just distinguishing cats and dogs, but rather making full use of the power of the human mind!).

Good news for you: being your target audience, we actually did have you guys on our radar.I

For the scale of our project, however, the price point was prohibitive.

We ended up building a small cli tool that interactively trained the model, and allowed us to focus on the most important messages (eg those where positive/negative sentiment was closest, the labels with the smallest volume, etc).

EDIT: If I now look at your website, it seems like you’ve also just provide good tooling for doing these types of things yourself? If that were the case, I wouldn’t mind having paid $50-$100 for a week of access to such a tool. But $20/hr to hire someone who classifies data which we would still need to audit afterwards was too much for us.

$20/hour to classify data sounds reasonable though?

If you have more time than money it might not make sense, but at that price point I could save myself a lot of time by just working a few extra hours doing SE and let someone else do 3x that amount of labelling.

I fully agree $20/hr is reasonable, it just was too expensive for us at that time.

So in the end the whole problem boils down to “quality is (more) expensive”; but MTurk is a special case since they’re so heavily positioning themselves as “the” solution for this and they’re terrible.

Reply on news.ycombinator.com