Show HN: I made a tool that lets you collect large amounts of human inputs fast | Hacker News Reader

Show HN: I made a tool that lets you collect large amounts of human inputs fast

app.rapidata.ai

5 pointsmaalber7 comments

Along with some friends I have developed a tool that lets you send out a small task or question to thousands of people and have the responses trickle in within a few minutes. We have developed the tool with data annotation for AI in mind, since this is typically a huge time sink for researchers. However we imagine it can be used in many other scenarios like user, or market research, or to determine which pictures to upload to your dating profile. Please share other ideas you might have and feel free to give it a try. Most of the effort so far has gone into building a solid distribution pipeline, so the UI still leaves something to be desired. Any feedback and suggestions are very welcome. Since what you upload will be distributed to thousands of real people worldwide, we unfortunately require a sign-in before the upload stage and you can expect a short delay as the content gets approved. We are hoping to commercialize the tool, so if you or someone you know has a use case for it, please feel free to reach out and we would be happy to consider pilot projects.

Understanding how/where the humans were recruited is probably important info to many potential users, for language and cultural reasons.

Thank you for the feedback - and sorry for the late response. People are not recruited as such, rather the tasks/questions are distributed to appear in-app on smartphones and similar devices. The web-interface allows you to directly select which country/region the responses should come from, exactly to be able to take into account regional or cultural differences and biases. Additional metadata is also available along with the results. I am currently looking into other demographic data - do you have suggestions on what to include in this regard?

How do you verify that the "humans" are humans? (and not bots or sockpuppets)

For verification of humans we primarily rely on two approaches. One is about sporadically verifying answers on clear examples that we know an objective truth for, and keeping track of the results for each user. The other is through third party tools as part of the distribution chain, which has already invested heavily in this area.

Can I use this to e.g. benchmark two image generators against each other? Cause their output is usually kinda subjective...

Interesting!

I just tried your tool. I asked people in the US who they'd rather work for and put five images of Musk, Zuck, Bezos, Cook and Huang. I guess the ranking mode does pairwise comparisons of the images if I interpret the resulting .json file correctly.

This what I got (win rates):

1. Elon Musk (67.7%) 2. Mark Zuckerberg (52.5%) 3. Tim Cook (51.6%) 4. Jensen Huang (46.4%) 5. Jeff Bezos (31.4%)

(also as a bar plot here: https://imgur.com/a/M0NDD6Z)

Took me a while to figure out the UI though...

Reply on news.ycombinator.com