Comment by visarga | Hacker News Reader

Comment by visarga | Hacker News Reader

> This extremely energy efficient chip, which uses the data to learn and make inferences, gets smarter over time and does not need to be trained in the traditional way.

Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.

If this was a news site, I'd understand - the same writers cover too many topics, they can't be experts in all. But when it comes straight from Intel, it's weird to see such mistakes from someone who signs as Dr. Michael Mayberry.

> Researchers have demonstrated learning at a rate that is a 1 million times improvement compared with other typical spiking neural nets as measured by total operations to achieve a given accuracy when solving MNIST digit recognition problems.

MNIST is a joke already, why don't they show off the ImageNet numbers? And spiking neural nets are less efficient compared to normal neural nets, so the 1 million speedup might not be as impressive as it appears at first hand. They don't compare with GPU run TensorFlow or CNTK here, but with other spiking neural nets which are seldom used, mostly as a proof of concept.

> A total of 130,000 neurons and 130 million synapses.

So the network size is medium - you can do a lot with 130 mil weights, but there are models on GPU that go up to 1 billion weights. Even the "cat detector" unsupervised neural net created by Google in 2011, had 1 billion weights, and that's ages ago in AI time.

> Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.

Nope, "trained" is correct. The current state of deep learning involves "training" and "inference" as distinct components where entirely different kinds of math are applied to each. GPUs can do training as training is kind of "dumb" and repetitive. CPUs, however, can do training and inference (and also can do allocations among resources dedicated to training and inference).

Sure, GPUs are the trendy thing being pumped by stock market day traders these days, but they are only about as efficient as the black Model T Ford in the "traditional" sense.

> Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.

There are two ways to build spiking neural net architectures without training. The [Neural Engineering Framework](https://medium.com/@seanaubin/a-way-around-the-coming-perfor...) and [Deneve's approach which I'm more uncertain about](https://forum.nengo.ai/t/robustness-of-deneves-networks/320?...).

To be clear, when I say "training" in this context, I mean they don't require iterative rounds of stochastic gradient descent. They just need to be given the function that you want to approximate and then they're done.

Yes, maybe it uses something other than gradient descent, but why? Nothing works nearly as well by comparison. In a chip we can do things we can't do in the brain, such as sending gradients back on the same connections.

> Nothing works nearly as well by comparison.

Depends on what you mean by "well". It would be foolish of me to deny the advances of deep learning. However, there are limitations and the NEF can act as a bridge in those cases. For example, if you want:

1. To approximate a dynamic system 2. To use symbolic computation 3. Use the neural networks as an initial guess at a solution to a problem and then want to use gradient descent to figure out the actual solution

I talk about using NEF as a bridge in [this blog post](https://medium.com/@seanaubin/deep-learning-is-almost-the-br...), if you're curious. Unfortunately, I don't know of any examples using the "initial guess" paradigm that I describe, but that's more of a "not enough people using Nengo" problem.

What do you mean by "give the function"?

If you can do that, either by writing a deterministic algorithm, or even defining it analytically, there's no need for machine learning, neural networks or etc.

> there's no need for machine learning, neural networks or etc.

There is if: 1. You want an energy-efficient distributed representation of the function. 2. You want to use the function as an initial guess and then learn online afterwards.

"in the traditional way" - maybe it is in nontraditional way.

Reply on news.ycombinator.com