Tensorflow.js: Machine Learning in JavaScript

I have a hard time sorting through the ml logic vs the game logic, it would really help my learning (and others too) if the game logic was decoupled from the ml logic into 2 separate files.

Re-training! The generalised models are... Generalised. They have learned to pick up details from a vast, varied data set.

You can reuse that, and fine tune to the model to your specific data set, which could save you days worth of computation, and also decreases the data set size requirement on your end.

There is a process called transfer learning where you take a model that has been trained on a large data set and then modify the last few layers of the neural network by training them on your specific/smaller data set. Not sure if there are any good resources for beginners but I learned about the technique from fast.ai videos.

Here is an example thing I made using one of the pretrained mdoels (MediaPipe Facemesh):

https://github.com/paruby/snake-face/

I'm sure that many people would not class this as interesting or novel, but it demonstrates that at least some of the building blocks can actually be used to make actual things.

I don't think pretrained models are "beginner building blocks". They provide a kind of "base model" that can be fine tuned to your specific need. The advantage of not training the full model means you can save many resources (in design time and computation).

There are many boring but meaningful tasks in which this can be used. For example, I'm sure many industries could be benefit from image classification for very specific cases (e.g., fruit categorization). In those cases, you are not interested in the classification of "general" objects such as car, person, bike or horse (as provided by MobiletNet pretrained model), but you can use that model as base to classify different categories of fruit.

Therefore, you are right that they might not be very useful to build new ML algorithms or network architectures. But they are useful to build specific (and novel) uses of current neural networks.

You are correct that all commoditized AI currently cannot do anything completely novel.

They ship a network to convert landscape photos to landscape paintings, for example. You can retrain that with limited effort and knowledge by using their Collab notebook. That way, you can make an AI to convert photos of cats to photos of dogs. But you wouldn't be able to create an AI to convert between cats and rabbits, for example, because those are too different for that AI architecture to tolerate.

Similarly, they ship a classifier to detect objects of some categories based on surface texture. You can retrain that to detect different categories of things. But if someone paints a chair to look like elephant skin, the AI will be confused and short of inventing a new architecture yourself, there's nothing you can do to fix it.

Plus in any case, you will need to have large high quality datasets for your retraining. The biggest cost in building a bird classification app isn't the one ML employee that creates the final AI, it's hiring biologists to correctly tag thousands of photos to train the AI on.

That's why many AI research nowadays tries to offload the data generation to random people on the internet. For example, recaptcha, or crawling Flickr for that faces dataset.

Edit: since it was brought up in the comments, the retraining that I mentioned can be done only on parts of the model, which is then called transfer learning and can save you some computing time.

But, you'll need to determine if the first feature layers of the original model are suitable for your new classification task. Even experts sometimes guess wrongly here, so as a beginner, I'd say you can only pray and hope.

Sure. The innovation often comes from how you use the model, not trying to build new ones (although that happens too).

They have pose detection: https://github.com/tensorflow/tfjs-models/tree/master/posene.... One day during a talk with someone who works in aged care I worked up a demo of fall detection using this.

There is a difference between consuming models and training them yourself -- I could be wrong but with tensorflow.js I think it is much more geared toward consuming models in the browser, rather than training novel ones yourself. Seems like you can customize an existing model via transfer learning, which may or may not be good enough for your usecase, but even then I feel like basically the training side of it is second rate. But you could potentially make _predictions_ with a really sophisticated/cutting edge model in JS -- its not the prediction part that takes enormous compute, it is the training part

It depends on your definition of "novel." I've seen some of these base models for tfjs used in several neat hackathon projects ranging from music generating web apps to "hot dog, not a hot dog." I've seen tfjs run simple RL game agents in-browser, too. The models are re-trained, tweaked, or slightly augmented - but not too much, since you only have 24 hours.

Yes, the idea is that the model is local to the user and runs in their browser (pushing the computation to the edge). Here's a project demonstrating pose estimation from Twilio's engineering blog: https://www.twilio.com/blog/pose-detection-video-tensorflow-....

Re: bandwidth. Do you mean the weights? If so then yes, you'd have to think about distilling large models to smaller ones. There are techniques for doing this. Here is an explanation of distillation from Floyd Hub's engineering blog: https://blog.floydhub.com/knowledge-distillation/.

I used it in the wild for some week-end project [1][2]. To get around the bandwidth issue, I used it as a browser extension. This means that weights are downloaded and installed only once during the setup. But I also used it in the past in a website for real-time audio processing when model are small enough [3].

Novel models will probably need to have some custom layers, which are quite painful to write. The weights for the web version will probably be a low quality of what you achieve on a more powerful machine. And you don't need to provide the code for the training, nor the dataset used, so you will still have an edge over people copying you.

The main problem I have with tensorflow.js is that it's not production ready yet. It's probably 10 years ahead of it's time. Most people don't have recent GPU or recent video drivers. As it's doing some tricks with the GPU to have some acceleration, a fraction of your clients will encounter random bugs and crashes. I even got some bad review for causing reboots on Firefox :). For some people it will be so slow it will feel unresponsive, and others will find that it's unimpressive because you would have to use the minimal model. In a day and age where you need 99% satisfaction to exist, picking tensorflow.js is a mistake.

[1] chrome : https://chrome.google.com/webstore/detail/colorify/pipnfjhpb...

[2] firefox : https://addons.mozilla.org/en-US/firefox/addon/colorify/

[3] Wisteria : https://gistnoesis.github.io/

The Teachable Machine FAQ (https://teachablemachine.withgoogle.com/faq#Saving-&-Exporti...) says yes. See the "Can I use Teachable Machine models with other libraries and platforms?" question.

Yep! It looks like that's exactly what the new version of Teachable Machine is for: https://teachablemachine.withgoogle.com/

I might have misunderstood your comment, but if you've just started learning to code, I'd probably start with simpler tutorials and projects - just to get a handle on the basics. Depending on your background it could take a good 6 months of daily practice to get comfortable with the basics. It took me and my friends at least that long - similar time frame to learning a spoken language like Japanese, say, to a basic level. Learning to code is absolutely worth it though - 6 months is a very cheap price in my opinion. I'd probably start with a tutorial like this one: https://www.youtube.com/watch?v=yPWkPOfnGsw Or the Khan Academy computer science course.

This is fantastic. Any way to see and learn from source?

Yes, I'm interested. If you have notes please share. The more documented examples there are of how to use Tensorflow.js the better.

if you want to make use of your GPU you need an NVIDIA graphics card

I'd wait for the RTX 3000 series if possible, which would be around September or October. Seems like it will have better ML performance, and more VRAM, too.

If you can wait, wait as the other commenter stated. However, if you're just interested in playing with machine learning, you don't need to get the latest and greatest tech! Even the raspberry pi can do some machine learning experiments or inferences.

If you just want to play with Tensorflow.js in browsers then any video card will do. If you want to use Node then I think you can get accelerated performance using a NVidia video card.

If you want to train your own models then you'll need to use Python (or in theory Swift or something..) and you want an NVidia video card.

I played around with tf.js a few weeks ago and learning to use python-trained models was the first hurdle. I made a minimal working example of this here, feel free to check it out:

https://github.com/paruby/mnist

See model/mnist_js.ipynb to see how the model is trained and exported to a js readable format, and you can see in lines 13-15 in index.html how the model is loaded.

Note: I have a decent amount of ML experience but almost no javascript experience, so YMMV

there is a specific tool to convert from various formats to a tensorflow.js model:

https://github.com/tensorflow/tfjs/tree/master/tfjs-converte...

If used in a third party context, sure.

That's only relevant to ads. Hopefully no one is putting AI models in adverts.