Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this?
I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.
http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
A key assumption I don't share with Detmer is that we must match the brain's raw computational power to have a chance at human-level AI. From an evolutionary point of view, most of the brain's computations surely must be superfluous to intelligence and necessary instead to a vast host of other biological functions.
I'm not sure how you get that. My point is that artificial NNs may well be more efficient than biological ones because they are punching so far above their weight based on what you would expect from a naive comparison of parameters, in which case it's not 17 iterations but a lot less. Further, modern NNs don't typically max out the memory of a single GPU (since we want to do minibatches of at least n=10 for the training phase and researchers care a lot less about the forward passes or deployment), and groups like Google Brain or Deepmind have shown the ability to use ~1000 GPUs, so that's 4 orders of magnitude right there (doable with asychronous training like synthetic gradients).
> I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size.
What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
>What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
This seems more impressive than anything any one NN can do, but less impressive than what all NNs can do: https://en.wikipedia.org/wiki/Bee_learning_and_communication
[1]: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
You obviously cannot reach human-level performance on ImageNet or do anything at all with a single neuron, so such an absurd equation refutes any such calculation or attempt to use a comparison as an upper bound.
The assumption that only when we have comparable number of parameters to the number of brain cells we can have nearly human performance is no longer true. It gradually loses traction once people discover that stochastic connections may be more effective than merely increase layers/parameters.
I also think neural networks could save alot of parameters if they use sparse connections. That just doesn't happen to work well with our current hardware.
Transfer learning demonstrates that the 'specialization' objection is moot. The CNNs are learning large parts of what goes into general-purpose vision, not super-narrow specific things, otherwise it would be impossible to take a classification CNN and reuse only a tweaked version for image captioning, localization, tagging, segmentation etc and find the intermediate layers learning semantically meaningful things.