Hacker News Reader

Comment by jackblemming | Hacker News Reader

Why can’t they also make glasses that translate sign language into English audio or text then?

Translating sign languages to spoken languages is really, really hard. Sign languages have a lot of features that don't really show up in spoken languages.

Let's take an example: the sign for "send email" in ASL (http://www.lifeprint.com/asl101/pages-signs/e/email.htm). If I point at you at the end of the sign it could mean "I will send you an email." If I point at myself it could mean "You should send me an email" or "Did you send me an email" depending on my facial expression. If I point off into space it could mean "I'm sending an email." If I start by pointing at you and then end the sign by pointing off into space it could mean "You should send an email." So your translator AI needs not only to understand the facial expressions and movements of the signer, but also the spacial relationships of everyone in the conversation. And that is just one aspect of the difficulty - there are many other features of sign languages that are just as hard to translate.

Perhaps this is the sort of thing that future AI systems could do. But it is quite complex.

Not an expert in either technology but I would imagine a lot of the difficulties of speech to text (bad recording conditions, variety of accents & pronunciation differences, etc) also have analogues in computer recognition of hand signs (camera alignment is bad, cut off, lighting is bad, someone's hand signs are lazily performed or slightly different than textbook ASL, etc).

Speech to text was "solved" a long time ago but I've seen it take many years to become as usable as it has recently. And it still regularly is frustrating to use for me!

My experience of speech to text has been terrible. As soon as someone has a bit of an accent, it becomes completely incoherent, even if all participants understand the accent perfectly. Even for good accents, using any kind of technical terms or proper names throws it off. And even when none of these are a problem, it still has at least a few percent error, even for English.

true. But all transcription tools are terrible transcribing technical language if someone has accented English.

Microsoft Research had early implementation for this kind of visual sign language translation with Kinect & ASL recognition in 2013 or so. I expect that with the death of Kinect in the market it stayed lab-bound.

Sign language (speaking with ASL in mind, unfamiliar with others) uses a different grammatical structure from American English. To boot, a signer can adopt different body languages to imbue their message with different tonality. Current ML models (as far as I’m aware) struggle with capturing that tonality in speech-to-text, and there’s _far_ less data for sign language recognition, regrettably.

Sign language is a complex language, so something has to learn yet another language. Perhaps made all the more complex as it's spatial and visual instead of verbal or written.

Not every deaf person uses sign language and there are many different sign languages in the world. American Sign Language(ASL) is but one of these languages.

> Sign language is a complex language

Specifically, sign languages are not visual representations of existing languages (e.g. ASL and English) but completely different languages altogether.

And even then, there are different dialects of ASL.

Exactly. Though Visual "representations"(for lack of a better word) of English do exist in dialects like SEE(Signed Exact English), which is obviously not the same as ASL.

I'm sure other signed languages have rough equivalents in their regional areas as well. I know Mexico has a few different signed languages, though I only have passing familiarity with 1 of their signed languages and it's definitely not a representation of Spanish.

I was waiting for someone to say this. I have taught Humanities in a program which included a significant number of deaf students. Communication was enabled via translators. The difference in language culture plus the fact that everything was mediated through a third party made the whole experience profoundly unsatisfactory. Nuance was completely lost. Perhaps if I had recovered training in sign language things world have been different.

I'm sure that's something someone could do! And it sounds like a very fun project. It probably hasn't already been done because there already exist very good speech-to-text models and not many (any?) sign-language-to-{text,audio} models

Reply on news.ycombinator.com