SignWave: Program to transcribe text, audio files into a sign language animation | Hacker News Reader

SignWave: Program to transcribe text, audio files into a sign language animation

github.com

24 pointsthunderbong11 comments

I learn signlanguage for 1.5 years now (ÖGS - Austrian Sign Language) and this project is really awesome. I wonder how they fixed the sentence structure problem because many fill-words just don't get a hand movement and the words are also quite scrambled.

For example "what is the capital of austria" would be "austria capital what".

From the screenshots it looks like they just translate English to sign word-for-word, skipping words that are harder to translate. So it's not translating to (American) Sign Language, which has its own grammar and is not just English with function words removed.

It might be neat if they fingerspelled the missing words. IDK as to irl applicability either way tho

I speak a bit of Swedish sign language and we have a version called TAKK which is signs with spoken grammar. This is used in schools for special need kids since it’s much easier to teach staff. ASL probably has something similar.

Sad to see additional languages doesn’t seem to be in the pipe for the project but it’s a good initiative.

I recently met someone who is deaf and communication was a real struggle.

We were pointing phones in each others face to listen and translate to text.

They called up a really great service called convo which has deaf translators live to translate.

All this worked and surely must be leaps and bounds better than 10 years ago. Even so it was far from natural.

Anything to make life easier for deaf people is good. I had it in mind for a badge that both people can wear, it flashes or glows when the wearer is talking and instantly translates what they are saying and shows it on the badge display.

This idea of translating to sign language also sounds great though in the context of conversation i don't think it would be more helpful than text.

Are there any example videos how this works? From looking at the source in repository, this is not sign language. The best way to describe this thing I can come up is implementing thanslation from, say, chinese to finnish using a single pass list of text replacement rules.

If you are going to show english word for word, well, subtitles are a thing already.

Any insights on what this is meant to accomplish? This is probably third or fourth such word-for-word project I have come across recently. It seems implausible that no one who works on them know what sign language is.

Not familiar with ASL, but there is a video here: https://devpost.com/software/speechtosign

In the "What's next for SignWave" roadmap section:

> Improving translation to more accurately reflect ASL sentence structure

Also note the devs seem to be high school-aged students[1].

[1]: https://jamhacks-7.devpost.com/

I translated the following message from russian to english by replacing letters (and skipping those that have no replacement in my translator). In upcoming versions, I will improve the english grammar of the results:

Nelz sobrat samolet iz solom i vaki. Solom ostanets solom to b v ne delali.

Wrt them being kids — imagine their software would present you with a medical diagnosis based on symptoms you input, using similar text replacement. Would them being kids make it ok, or would it make that even worse? I mean, obviously there were people giving them advice and oversight in the loop.

Face expression and hands position relative to the body are big information carriers on sign languages. At least in Spanish there are some signs with the exact same hand configuration and different meanings only because the face expression. Maybe they could aim for a whole body model on next iterations. Great effort still.

Kind of related, but does anyone have info on the best way to transcribe some hour long audio files of conversation?

Especially if it can detect different people? Would love to have timestamps for some sentences or be able to ‘jump around’ the audio file too.

Haven’t kept up with new AI tools around this :/

OpenAI whisper

That doesn't do diarization, unfortunately. Try https://github.com/MahmoudAshraf97/whisper-diarization or something similar instead.

Reply on news.ycombinator.com