Human in A.I. Loop | Hacker News Reader

Human in A.I. Loop

sanealytics.com

75 pointssndean21 comments

"What we need to build is a gamified quick truther thingy that all our 200 employees in main office can log in during their lunch break and rate a few items."

Lunch break? Or you could just, I don't know... go with me here, I know it sounds crazy at first..., pay them for providing this value to your company.

You're already paying them. Simply tell them to do this during any downtime they might have. Obviously don't encroach on their lunch/home time, though.

Or, strange concept, the employees benefit from their efforts to improve the company, and idea of working during lunch on a trival task makes sense if they don't have anything else worth doing.

>the employees benefit from their efforts to improve the company

The traditional benefit for this activity is known as a "pay check".

Be wary of someone who tells you that changing the world should be its own reward when it looks like they'll be keeping the money.

its no longer a break if you're still working.

And the break is worth doing. Do you want your highly paid managers, engineers, salespeople, etc. properly recuperating? To better tackle the nontrivial tasks you pay the big bucks to tackle?

Or do you want really really expensive mechanical turks?

"In the loop" is a reference to the OODA loop, which is the decision cycle of observe, orient, decide, and act, developed by military strategist and United States Air Force Colonel John Boyd. "In the loop" has a specific tactical meaning, which is that you're able to do a full OODA loop cycle faster than a comparable AI/man system which gives a strategic advantage and maybe used against oppositional forces in a variety of ways; most interesting is applying it to PsychOps.

"Man in the loop" reference can be found in use by the US AirForce going back to the 80s for weapon systems design; specifically between fighter jet systems and pilots. I've heard that US AirForce used brain implants as early as the 90s in fighter pilots.

Here's an example of the US Army reads soldier's brain waves to speed up image analysis:

http://newatlas.com/us-army-eeg-brainwave-image-analysis/403...

I'm pretty sure it's just a reference to the phrase "in the loop", which dates back beyond the 80s:

http://idioms.thefreedictionary.com/in+the+loop

Yes, "in the loop" does date back further, but as used to reference human/systems integration earliest reference I was able to find was the 80s which was the same time that Boydian like systems started popping up.

Looks like what CrowdFlower calls "human in the loop computing" which is a nicer name for it.

http://www.computerworld.com/article/3004013/robotics/why-hu...

Also called human computation. The past paper awards from this conference [0] gives a good flavor for the field.

[0] http://www.humancomputation.com/2016/past%20meetings.html#aw...

It seems that the name of the concept this guy is looking for is: Mechanical Turk service.

The concept is actually more commonly called "human computation" (which includes stuff like digitizing books using CAPTCHAs).

It has academic journals/conferences under that name: http://hcjournal.org/ojs/index.php?journal=jhc

Not sure if MT works when some specific knowledge is needed (I think their employees know or have quick ways to find out the correct matching dress given a photo)

Yes perhaps for training the AI.

Applications of human-in-the-loop systems like this are exactly why we built Scale API (www.scaleapi.com, YC S16).

Unfortunately, Mechanical Turk is often a bad choice these days for these types of applications—it's extremely difficult to guarantee quality, not to mention hard to use. Using employees also works well to a point (we did some of this at companies I've worked at too), but it doesn't scale, and should be done sparingly.

Great application of humans in the loop—this is the type of thing that really excites me. We're trending towards more and more human-augmented software, and it opens up so many possibilities for what software can do!

Does anyone know any good tools to make that sort of page quickly in Python? I mean, I could cobble together a web app, but I feel like dataset annotation is so common that there should be something out there already.

Why Python? Why not just focus on finding the best open source data annotation framework?

(Love Python, but normally don't bias my selection of a solution by the language it's written in.)

You're right, though I do need a way to interact with it programatically from python, since that's what I prefer to write my scraping & munging code in.

It would probably be fine if it was a solution that had an API that was easily callable from Python without a silly amount of overhead, specifically being able to dump data into a named dataset is what I want.

Python would also be nice since it would be easier for me to customize; though I could deal with Java or Node. I've avoided Ruby for this many years and would prefer to keep it that way.

Something like this, perhaps:

http://labelme.csail.mit.edu/Release3.0/browserTools/php/mec...

we wanted flying cars - instead we got tagged couture.

Reply on news.ycombinator.com