b-3-n

Thank you for pointing this out. I overlooked this. I presume that on top of that what also could be relevant is the movements of arms, facial expressions and maybe also general body posture. Please correct me if I'm wrong as I'm not too familiar with sign language.

Thank you for sharing this creative idea. "so that the fingers can touch (or almost touch) the screen" I think this is a big advantage of this approach since you can only achieve this with the back facing camera. On the flip side, with a back facing camera you either have to place the camera in between yourself and the screen which might be awkward or you have to ensure a placement of the camera behind you that isn't prone to occlusions (e.g. your head or chair might occlude your hands from the cameras point of view). The latter might also make calibration more difficult or impact precision since you might have to mount the camera with some elevation causing a less optimal camera angle.

Thank you for the feedback.

Thank you for the feedback. I can confirm there seems to be an issue with iOS/Chrome and created a GitHub issue for it (https://github.com/handtracking-io/yoha/issues/5).

Note that if you were trying iOS/Safari and not iOS/Chrome there is nothing that can be done due to a limitation that is documented in the section "Discussion" here: https://developer.apple.com/documentation/webkitjs/canvasren... Will document this.

Thank you for the feedback. I would like to fix this but I neither own a Chromebook nor does it seem like I can use a platform like browserstack to reproduce the issue (didn't find Chromebook as available device there). If you would like to help debugging the issue you can open a GitHub issue here: https://github.com/handtracking-io/yoha/issues

Thank you for the question. Since a similar question was asked already let me refer you to this comment: https://news.ycombinator.com/item?id=28830943

Thank you for the feedback.

Thank you for the feedback. You are right, the home page should probably be enriched with more information and maybe I can make the information you were looking for stand out better. As a side note: There is a link to GitHub in the footer. The language ("TypeScript API") is also mentioned in the body of the page. But I see that these two can quickly go unnoticed.

Thank you for your feedback.

One can build this pretty easily for a website that you are hosting with the existing API (https://github.com/handtracking-io/yoha/tree/master/docs).

However, you likely want this functionality on any website that you are visiting for which you probably need to build a browser extension. I haven't tried incorporating YoHa into a browser extension but if somebody were to try I'd be happy to help.

Thank you for this inspiring question. For interpreting sign language you need multi-hand support which YoHa is currently lacking. Apart from that you likely also need to account for the temporal dimension which YoHa also does not do right now. If those things were implemented I'm confident that it would produce meaningful results.

In contrast to similar works there is no dedicated paper that presents e.g. the neural network or the training procedure. Of course ideas from many papers influenced this work and I can't list them all here. Maybe it helps that the backbone of the network is very similar to MobileNetV2 (https://arxiv.org/abs/1801.04381). Let me know if you have any more questions in that regard.

Thank you for the feedback. Indeed such a functionality would be nice. One could solve this via another hand pose or in some way also with the existing hand poses. E.g. make a fist for say 2 seconds to clear the whole screen. Anything shorter will just issue an "undo".

YoHa uses tfjs.js which provides several backends for computation. One indeed uses WASM, the other one is WebGL based. The latter one is usually the more powerful one.

Thank you for the feedback. You are right that the project is not open source right now. It's "only" MIT licensed. That's why I also don't advertise it as open source (if you see the word open source somewhere it would be a mistake on my end, feel free to tell me if you see it somewhere). I wanted to start out from just an API contract so that it is easier to manage and get started. In general I have no problem open sourcing the JS part. But first there is some refactoring to do so it is easier to maintain upon open sourcing. Stay tuned!

As a side note: The wasm files are actually from the inference engine (tfjs).

Please let me know if you have any more questions in that regard.

That's an interesting idea. I have not tried to build something similar but a humble word of caution that I want to put out is that no matter what kind of ML you use the mechanical version of the instrument will always be more precise (you likely are aware of it, just want to make sure). However, you might be able to approximate precision of the mechanical version.

Two hand support would be nice and I would love to add it in the future.

The engine should work well with different skin tones as the training data was collected from a set of many and diverse individuals. The training data will also grow further over time making it more and more robust.

Thank you for your feedback and for sharing this potential use case. I think it is a very creative idea.

Thank you for the feedback. Such an integration would be nice indeed.

Hey, I believe there are multiple things you could have meant. From the top of my head one thing that might be interesting would be an application that allows conductors to conduct a virtual orchestra. But there are other possibilities in this space too I'm sure! If you had something else in mind feel free to share.

I have not explored this space much so far as my focus is rather to build the infrastructure that enables such applications rather than building the applications myself.

Wish there would be some politicians on that list ;)

The post is very insightful and well written. The title is provocative which is very common nowadays. However, there should be a "serious" section that puts things into perspective. As others have pointed out it leaves a bit of a taste otherwise.

"Give us your data, you can trust us". Remember how that went?

"Give us your money, you can trust us". Hm...

I was a bit disappointed to see that the "constant time" was a click bait. Should be "fixed time" - or similar - instead.

Firefox is playing its trump, the privacy, very well lately. This is very smart as the competition has no good answer. Equalizing on privacy level would go against their business model so they won't ever do it wholeheartedly.

Trying out Firefox now...

The real problem is that regardless how much we trust Google and Chrome nobody can make guarantees about future government or corporate politics so it should be really important to us to keep the browser market competitive.