First you need a whole lot of data to train the system which doesn't really exist (unlike the case of speech to text or one language to another(text to text)). You would need to set up a big data collection project.
Secondly there are a whole bunch of different things to track. Movement of multiple fingers, +palm, +entire hand, +facial expressions. Current state of the art would be tracking a single thing like a ball or a player for sports and that still requires a number of cameras and a couple people sitting behind the scenes fixing issues.
I think Microsoft's Kinect came close to achieving something akin to sign language recognition at a very basic level. But from what I remember reading they spent a whole lot of time and resources training their system just to get there.
Maybe 3-4 years away I'd say.
In the long term I can not think of anything why it should not be possible but here a few difficulties that come to mind:
1. Sign languages are in general not yet as well understood as spoken languages (the grammar, the vocabulary). For example the University of Hamburg is running the "DGS-Korpus" project[1] in order to create the first real dictionary for DGS, currently there is none. i.e. even humans have not a full explicit understanding of the language (DGS) and afaik it's similar for other sign languages
2. As video recordings are not older than 100 years and the community of native speakers per language is much smaller than for spoken languages the amount of recorded speech is much smaller
3. Large parts of sign languages are very productive. That means that the speaker/signer is not limited to a fixed vocabulary and a fixed grammar but allowed to reenact a situation he is telling about ("constructed dialogue" and "constructed action"[2]). Eg when retelling a conversation the storyteller can shift her body to the left or the the right in order to embody multiple people. Instead of saying "I drove the car. I had to turn left and then I saw her on the right side" the signer can just move as he was sitting in the car, holding the the steering wheel, moving it to the left and turning is head to the right. I would compare that to making "phewww... ohhhhh, whoaaa" sounds in order to tell how the plane you sit in took of. Additionally the signer is allowed to allocate locations around her upper body to establish references to previously mentions objects and refer back to them via pronouns.
4. There is no agreed upon notation for writing down signed texts. Sure there are notations for various hand forms and movements directions but especially the productive part of language mentioned above is hard to formalize. E.g. spacial locations that can be used for placing references are not predefined. Some person still learning the language may only be able to use two positions "left to his body" and "right to his body", but a very eloquent signer may be able to establish even 10 references points in front of him and still enable the audience to keep track of them. Such arbitrary reference points can not be written down in a sensible notation.
5. Various sign languages differ much more from each other than one might initially think. Each sign language has its own grammar and vocabulary. For german sign languages it's not even entirely clear what marks a sentence.
6. Sign languages are not sequential but simultaneous languages ie it's possible to express one thing with one hand and another thing with the other hand in parallel
So some first steps might be:
* find a formal grammar for sign languages
* find a formal notation to write down arbitrary sentences
* collect a large corpus
* Train some AI to translate video data into your notation
* Train some AI to translate the notation from one sign language to another (and to spoken/written language as well)
* Train some AI to translate spoken/written language to your notation for some sign language
* Build an avatar that accepts that notation as input and translates it into movement
[1] http://www.sign-lang.uni-hamburg.de [2] https://www.signteach.eu/index.php/podcastsall/item/construc...
Might not be necessary. Modern statistical machine translation techniques use formal grammars as an auxiliary signal at most.
> find a formal notation to write down arbitrary sentences
Some sign languages might not be representable in SignWriting, but most should be covered. https://en.wikipedia.org/wiki/SignWriting
> collect a large corpus
Many (all?) European (and other) countries have laws requiring news broadcasts, parliamentary debates etc. to be made available in sign language. Similar to the EUROPARL corpus, that could be used as a starting point for signed <-> spoken translation. Also similar to EUROPARL, colloquial language would be underrepresented.
Of course there won't be any SignWriting transcriptions, so the corpus would be essentially unlabeled.
> Build an avatar that accepts that notation as input and translates it into movement
I'd do that before/while attempting to create a large number of transcriptions, both to ensure accuracy and to enable unsupervised pattern mining in the later steps.
> Train some AI to translate video data into your notation
What I mean by unsupervised pattern mining is that the transcription AI would be trained to output a transcription that makes the avatar ouput something like the original video, as well as reconstructing the avatar's input from its movements.
> Train some AI to translate the notation from one sign language to another (and to spoken/written language as well)
It'd probably be easiest to start with translating between the signed <-> spoken pairs in the corpus and then leverage that into signed <-> signed translation.