clicking through the neural networks api:
ANEURALNETWORKS_RELU6 The output is calculated using this formula: output = min(6, max(0, input))
is that a joke/easter-egg?
also: is there an official java wrapper for the neural networks ndk api?
ANEURALNETWORKS_RELU6 The output is calculated using this formula: output = min(6, max(0, input))
is that a joke/easter-egg?
also: is there an official java wrapper for the neural networks ndk api?
The original paper detailing the function is here (PDF warning):
http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf
"Our ReLU units differ from those of [8] in two respects. First, we cap the units at 6, so our ReLU activation function is:
y = min(max(x, 0), 6).
In our tests, this encourages the model to learn sparse features earlier. In the formulation of [8], this is equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an infi nite amount. We will refer to ReLU units capped at n as ReLU-n units."
And even points to a paper: http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf
I'm not sure why they didn't make it a variable maximum though, or what the advantage is over multiplying a sigmoid by 6.
https://stackoverflow.com/questions/47220595/why-the-6-in-re...
It's an arbitrary value, but 6 fits best into 8 bits.