Impressive!
LightOn hasn't received much discussion on here before. Some links have been submitted, and this is the only one I could find comments on: https://news.ycombinator.com/item?id=27797829
Maybe you aren't aware, but half-precision (16 bit float) is already well-established in the AI community: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format. In context single-precision isn't all that puny!
And there have already been successful experiments with stronger quantization, like 8-bit neural nets, or even 1-bit (!) neural nets. There is a lot of evidence that neural networks can be very resilient to quantization noise.
I don't think there's any point at which PCs have ever abandoned 8-bit and 16-bit data types, or ever appeared to be on a trajectory to do so. We've just had some shifts over the years of what kinds of things we use narrower data types for.
Inference is cheap, training is not.
And drones, trivially agree: Megaphragma mymaripenne has 7400 neurones, compared to the 71/14 million in a house mouse nervous system/brain.
Best estimate I have for total cell numbers in mouse brain—about 75 million neurons and 35 million other cell types. This estimate is from a 476 mg brain of a C57BL/6J case—the standard mouse used by many researchers.
Based on much other work with discrete neuron populations the range among different genometypes of mice probable +/- 40%.
for details see: www.nervenet.org/papers/brainrev99.html
Expect many more (and I hope better) estimates soon from Clarity/SHIELD whole brain lighsheet counting with Al Johnson and colleagues at Duke and team at Life Canvas Tech.
Joking aside, it is interesting how much noise and quantization these neural networks can work with. I think there’s a lot of room for low precision noisy computation here.
It may require some architecture changes to make training feasible, but it's far from a nonstarter.
And that is only considering backprop learning. The brain does not use backprop, and has way higher noise levels.
Maybe it’s application dependent, maybe NNs or other matrix-heavy domains can tolerate low precision much more easily than scientific simulations. It certainly wouldn’t surprise me if these “LightOPS” processors work well in a narrow range of applications, and won’t improve or speed up just anything that needs a matrix multiply.
For example path tracing (ray tracing) doesn't need to be very accurate because multiple samples per pixel are used.
A gaming card that uses less power is very welcome in laptops for example.
Maybe another way to ask is- we’re trying to simulate a real world “analog” scene - maybe using an analog processing technique could actually be quite faithful for generating it?
Or not. Like I said I don’t understand too much of this.
Now I'm not sure what this 6/8 bit precision or analog precision means so I can't say with confidence, but if your scalars are that low precision you can't really do much. You could technically encode it with some fancy tricks like instead of storing coordinates for each vertex you store the delta from previous one etc. but I think this wouldn't work if the device was just some dumb analog matrix multiplier with baked logic.
Also having low detail shadows creates visual artifacts, see this for example [1]
[1] https://digitalrune.github.io/DigitalRune-Documentation/medi...
> … leverages light scattering to perform a specific kind of matrix-vector operation called Random Projections. … have a long history for the analysis of large-size data since they achieve universal data compression. In other words, you can use this tool to reduce the size of any type of data, while keeping all the important information that is needed for Machine Learning.
So it sounds like the whole point is to reduce the data size and then feed it into GPUs like normal ML. Kind of a neat idea.
(*) disclosure: my company helped build the simulator