ASL ⭤ English Translation with MediaPipe, PointNet, ThreeJS and Embeddings

We need to bridge the communication gap between Deaf and hearing individuals.

Hi! I'm a Grade 11 student at Burnaby South Secondary School, also home to British Columbia's Secondary School for the Deaf. Over the last few months, I have been working on an ASL ⭤ English translation tool that respects and preserves American Sign Language (ASL) as the primary language.

Most existing ASL translation tools are built on the misconception that ASL is the same language as English. They view Deafness as a disability and only seek to overcome the inability to hear, but not to translate to the language of ASL itself.

Watch my YouTube video here https://youtu.be/uuPxMWQRoXc?si=7J5FAdoa9_xby9nf

For ASL reception, I augmented over 100,000 images of ASL alphabets using Google MediaPipe and trained a PointNet model to classify handshapes fingerspelled by Deaf individuals. For ASL expression, I augmented over 9,000 videos of ASL signs, embedded their corresponding words, and then used ThreeJS to sign words said by hearing individuals. I also use LLMs to improve accuracy and translate between English and ASL grammar.

I'm still learning more about machine learning, and I would appreciate any general feedback or opportunities related to this project or technology! Let me know if you have any questions :)

Check out the GitHub here https://github.com/kevinjosethomas/sign-language-translation Check out the LinkedIn post here https://www.linkedin.com/posts/kevinjosethomas_deaf-deafcomm...