nsthorat

Lilac co-creator here :)

Lilac is an open-source tool that enables AI practitioners to see and quantify their datasets.

Lilac allows users to:

- Browse datasets with unstructured data.

- Enrich unstructured fields with structured metadata using Lilac Signals, for instance near-duplicate and personal information detection. Structured metadata allows us to compute statistics, find problematic slices, and eventually measure changes over time.

- Create and refine Lilac Concepts which are customizable AI models that can be used to find and score text that matches a concept you may have in your mind.

- Download the results of the enrichment for downstream applications.

Out of the box, Lilac comes with a set of generally useful Signals and Concepts, however this list is not exhaustive and we will continue to work with the OSS community to continue to add more useful enrichments.

Check out the demo on HuggingFace: https://lilacai-lilac.hf.space/ Find us on GitHub: https://github.com/lilacai/lilac

I cant tell if this is research about generative AI, or AI generated research.

The new internet sucks

How could a bear population quadruple in a month of lockdown when the gestational period of a bear is 200+ days?

This backend work has already begun (and can run posenet, albeit still slower than WebGL): https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-...

Unfortunately there is no attribution, but this tool was created by Daniel Smilkov, who also built TensorFlow Playground and who is a cocreator of TensorFlow.js.

https://twitter.com/dsmilkov

There are many reasons to do it in JavaScript:

- Many companies and projects have their entire server-side stack in JavaScript and Node.js, and often they want to simply make a prediction through a model. It's quite a lot to ask them to pull in a python runtime just to make a prediction. TensorFlow.js with node bindings to TensorFlow C enables this type of inference with minimal overhead.

- Privacy. You can make predictions locally, or send embeddings back to a server without the raw data ever leaving a client.

- Flexibility of JavaScript / TypeScript. Dynamic languages are great for scientific computing, TypeScript allows you to define your own level of type safety, from raw JS on one end, to strict typing support on the other end.

- Interactivity / education tooling. See tensorflow playground for an excellent example.

- No servers for applications. Making predictions in TensorFlow on a server can be expensive in the long run. Hosting static weights on a server is much much cheaper.

JavaScript and Python ecosystems for machine learning are not mutually exclusive -- they both have their strengths and weaknesses.

This probably won't buy you anything. The API you still have is WebGL, and JS is not the bottleneck.

We've done some initial tests ourselves. WASM doesn't yet support SIMD so WebGL tends to be 5-10x faster. SIMD is actively being worked on by many smart people in Chromium / other browsers, so I would expect to see huge wins in the near term future. When that happens, deeplearn.js will have a WASM backend. WASM has a much better memory management story (destructors on the C++ side) so I'm super excited about its future.

Come build a WASM backend for deeplearn.js :)

Or you improve your algorithms and use the existing hardware (think distributed computing on cheap HDDs).

Why wait?

It doesn't work in node yet, a relevant issue: https://github.com/PAIR-code/deeplearnjs/issues/234

You're right. Some history:

We wanted to do hardware accelerated deep learning on the web, but we realized there was no NumPy equivalence. Our linear algebra layer has now matured to a place where we can start building a more functional automatic differentiation layer. We're going to completely remove the Graph in favor of a much simpler API by end of January.

Once that happens, we'll continue to build higher level abstractions that folks are familiar with: layers, networks, etc.

We really started from nothing, but we're getting there :)

WebGPU conversations are ongoing: https://en.wikipedia.org/wiki/WebGPU

WebAssembly is coming along quite nicely.

And SwiftShader is a quite nice fallback for blacklisted GPUs. They simulate WebGL on the CPU and take advantage of SIMD: https://github.com/google/swiftshader

Often times researchers train huge models, but don't think about model size (because they don't have to). We've seen ~200MB production models get down to ~4MB and not lose much precision. I'm quite confident we'll continue that trend.

Don't forget that folks were saying this about the web when images / rich media were becoming prevalent!

There is lots of work being done in model compression (quantization, simple factorization tricks, better conv kernels like depthwise separable convs, etc). We won’t let that happen!

We call ourselves deeplearn.js, but you can use it for general linear algebra! Our NDArrayMath layer is analogous to NumPy, and we support a large subset of it (we support many of the linear algebra kernels, broadcasting, axis reduction, etc).

This is just the beginning :)

Author of deeplearnjs here. We hear you, and we 100% agree. Stay tuned.

Author of deeplearn.js here. A quick summary:

We store NDArrays as floating point WebGLTextures (in rgba channels). Mathematical operations are defined as fragment shaders that operate on WebGLTextures and produce new WebGLTextures.

The fragment shaders we write operate in the context of a single output value of our result NDArray, which gets parallelized by the WebGL stack. This is how we get the performance that we do.

kinda, ya

It works on mobile, it's just slow. Every time we read and write from memory we have to pack and unpack 32 bit floats as 4 bytes without bit shifting operators >.>

Training a neural network on top would require a "proper" training phase, and finding the right hyperparameters that work everywhere turned out to be tricky. Actually, this is what we did originally, in the blog post we'll try to show demos of each of the approaches and explain why they don't work.

KNN also makes training "instant", and the code much much simpler.

We're using SqueezeNet (https://github.com/DeepScale/SqueezeNet), which is similar to Inception (trained on the same ImageNet dataset) but is much smaller - 5MB instead of inception's 100MB - and inference is much much quicker.

The application takes webcam frames and infers through SqueezeNet, producing a 1000D logits vector for each frame. These can be thought of as unnormalized probabilities for each of ImageNet's 1000 classes.

During the collection phase, we collect these vectors for each class in browser memory, and during inference we pass the frame through SqueezeNet and do k-nearest neighbors to find the class with the most similar logits vector. KNN is quick because we vectorize it as one large matrix multiplication.

I'll go deeper in a blog post soon :)

deeplearn.js author here...

We do not send any webcam / audio data back to a server, all of the computation is totally client side. The storage API requests are just downloading weights of a pretrained model.

We're thinking about releasing a blog post explaining the technical details of this project, would people be interested?

yes

"Historically it was unsafe to fly in an airplane so you shouldn't now"

This will be fixed in ~2 weeks, max.