Microsoft Research turns Kinect into sign language reader | Hacker News Reader

Microsoft Research turns Kinect into sign language reader

blogs.technet.com

65 pointstellarin23 comments

I'd be very curious to learn about the speed difference between a competent typist and a fluent signer, especially if visual cues can be used to accelerate the latter (ie, an equivalent of auto-complete).

When writing prose (as opposed to code), I could actually see signing being very speed-competitive, and possibly a more natural way to communicate text than a keyboard. (Other advantages might be use with Glass, or the ability to pace around a room while "typing".) Maybe someday, kids will learn signing instead of keyboarding.

When writing prose (as opposed to code), I could actually see signing being very speed-competitive

How so? Signing isn't as "complete" as written or spoken language. You would say/write "The dog ran in the park yesterday" but the signing would be more like "YESTERDAY, DOG RUN [THROUGH] PARK."

You're not going to get David Mitchell out of ASL, but then again you can't get motive music videos in Dejima either. They each have their place and meaning, but they aren't quite interchangeable.

Right, as a non-signer it didn't even occur that syntax would be different, implied, or occasionally non-existent. In theory AI could try to auto-generate, but yeah, maybe not as rosy as I hoped.

My son is 2 and recently found to be deaf. One usage case I imagined (while we are going through the process of learning sign-language and learning about the forthcoming Cochlea Implant he will receive) is exactly this, so it's gratifying to see some development in that sphere.

Sign-language is different to English in the same way German is different to English.. you can translate, but it's not quite the same.

Regardless, I did wonder - given how cretinous most Human / Computer Interfaces still are - if learning signing would be a better way of interacting with a machine. It would certainly have the side-benefit that users could talk to deaf people more easily (and possibly would help reduce the stigma that being deaf seems to still have, even today).

Having studied ASL at University (almost got a minor in it) I have a bit of a hard time believing this is fully possible at all. Signed languages aren't simply a series of words where each word has a given sign. It's a very complex and nuances language, and the regional dialects alone would make it tough to really decide what sign means what noun or verb.

Probabilistic graphical models like Markov networks can yield surprisingly complex behaviors, if they're complex enough. I don't have much trouble believing that it's possible.

Considering that sign languages regularly invent signs, e.g. for names, I doubt that. The interpreter might find out that there is a name at that place, but not know what the written equivalent of that name is.

Nobody is arguing complete all-knowing recognition of human movement. It's just recognition of a sign for "table" being table and a sign for "hat" being hat, etc.

And PGMs make it so you can use context to disambiguate things, so it can go beyond 1:1 sign-word recognition. Not sure if the researchers here are doing that yet, though.

Except that is the very problem: drive 100 miles, and the signs for table and hat are different. Change the conversation, and the signs may be different. I can put an arm low and flat, and it means a general flat surface. It doesn't mean table until I mime setting a cup down on it, for example.

Like I said, complex and nuanced. You have some incorrect assumptions in how signed languages work.

Everything can be deciphered completely contextually based on location though. None of these are insurmountable problems.

I wouldn't claim an ASL recognizer would suddenly identify BSL or anything wacky like that. Systems can even be trained (or generalized) down to a per-signer level. You could even have "Joe Pikachu's Generic Gesture Translator" too.

Think solutions, not problems.

Sign language also has other complexities. I don't know about american sign language, but german sign language (which I learned, I wouldn't say speak) is very depending on context and direction. For example, meanings can differ depending on what you are looking at. It is not "just recognition", it is "parsing the whole scene within context".

Computer vision ASL recognition is actually an old field: http://www.cc.gatech.edu/~thad/031_research.htm

I wonder how it would distinguish between the many dialects in use in the US. Granted, a lot of the adult population does use ASL [1], but lots of times children grow up using one of the Signed English[2] variants, which vary from state to state.

[1] http://en.wikipedia.org/wiki/American_Sign_Language

[2] http://en.wikipedia.org/wiki/Signing_Exact_English

[Edit: links]

Regional Language Packs?

Well that's gotta be the first useful application of the hardware that I've heard of! I've tried Kinect before and it might be just me, but I find it to be horrible for gaming. Give me physical keys and I'm all happy. But of course that doesn't appeal to the bigger audience of casual gamers. Nintendo got that right with their, now dust-covered, Wiis.

There have been a few other interesting applications, such as use by surgeons where physically using a computer would be unhygienic (e.g. see http://www.zdnet.com/blog/health/xbox-kinect-helps-surgeons-...).

Kinect is used a lot in rehabilitation and assistance of the impaired. Research conferences are full of Kinect applications and commercial applications are about to come on the market.

Just today I found out about this great company which allows to control the mouse with head movements and facial expressions. http://kinesicmouse.xcessity.at/

PS: In my opinion the uselessness of the current Kinect for gaming is mostly due to the latency (up to 1s!) and to a lesser extend to the accuracy of the tracking.

None of that is gesture based but:

- Forza head-tracking is really immersive and extends your screen.

- Ally control by voice is great too because it's a lot more expressive and parallel than buttons (and it frees up buttons).

During one coruse project we had used Kinect as a Semaphore (the flag signalling) reader.

I got to play with a prototype simulator at a Navy base where a Kinect was used to allow cross training of helicopter pilots and those signaling guys who direct the helicopter from the deck of a ship.

The signaling guy stands in front of a large wrap-around projector screen with a Kinect to capture his arm movements, which were sent over the network to a central server which ran the simulation. The pilot in the heli cockpit simulator would see the guy on the ship. The pilot and signal guy could be training on different sides of the country (or world).

It is a shame that things like this aren't made into shipping products.

Maybe at some point they will integrate research into the product groups so that we can still really cool (self driving car cool) projects coming out of Microsoft

Another win for the NSA against the menace of deaf terrorists!

Reply on news.ycombinator.com