Hacker News Reader

No, the last layer is just a linear layer, there's no non-linearity. That's what's strange about their definition. The depth-three network only applies a non-linearity twice, which would conventionally be labeled as depth two.

View on HN

You're wrong. See section 5.1. I've drawn a graph of it that shows it to be a classical NN with a single output node.

Yes, it is a classical NN with a single output node. I'm not disputing that, I just think their calculation of depth is strange. The network only applies the sigmoid function twice, and would ordinarily be regarded as having a depth of two. The third linear layer is fixed to multiplying by 1, which is what I meant by "has no effect". (Did you miss that I was talking about the third layer?)

Here's a diagram [1] of a one-hidden-layer NN. It has 2 activation functions. Their NN is of the same type.

[1] - https://raw.githubusercontent.com/qingkaikong/blog/master/40...

The question is, how deep is that network?

Reply on news.ycombinator.com