RaisinLoaf69

We’re not holding anything back, you should be able to recreate our findings from the paper. More broadly, there are a number of ways to recreate the network using the relatively simple learning rule we provide.

In theory, this learning rule will continue to decrease your error forever (in our simulations our error goes down to machine precision). However, with any physical learning network you’re always gonna hit an error floor based on the precision of your components. With our current variable resistors and network size that floor is around 10^-3. With more precise components (like we mention at the end of the paper), that floor will go down significantly.

This is strictly not a neural network, so there is no backpropagation. Credit assignment is done on each edge using a local rule (Eg. using only its current state and the state of touching edges). To scale the network you just have to add more edges (no limit on the amount). We have a design for a tiny version of this network using transistors that could have order 10^6 edges on the size of a microchip.

Since we have no global processor, each edge is changing using only local information, a lot of stuff that makes sense in our network doesn't really make sense in ANNs and vice versa. For example, in our newest paper (https://arxiv.org/abs/2201.04626) we desynchronize the updates of our edges. Instead of changing the entire system all at once, we change random parts of it each training step. This doesn't really make sense to do in an ANN where you require global information for every edge update.

The shape of the network is actually inspired by jamming solids (we're a soft matter lab), but is completely arbitrary. We've done a ton of different shapes and sizes in simulation.

Sweet! The basic learning comparison (Vc-Vf) can be implemented in tons of physical systems (memristors, springs, water pipes). So seeing it in other mediums would be pretty cool.

It is in fact just a digital potentiometer. In the experiment shown in the paper I linked we’re using a 128 position one. In new work we’ve actually shifted to using transistors which are better for a number of reasons (smaller, faster, nonlinear, continuous).

Thank you! Definitely a group effort. Two big things that contributed to the choice: We knew we didn’t want the system to have to store memory on each edge so that eliminated a lot of our options. In our experimental learning rule we only had to compare voltages which is actually easier to do if we have two simultaneous networks.

I actually work on this project! Feel free to ask any questions.

Also, here’s an arxiv link to one of the papers if anyone is interested: https://arxiv.org/abs/2108.00275