Hacker News Reader

Comment by Estragon | Hacker News Reader

Surely that's not true until someone achieves human-level performance with their entry.

Human level performance was surpassed earlier this year:

I won't be surprised when human level performance is surpassed, but isn't that paper based on work that has the advantage of knowing the full test dataset in advance? You can say "yes but they only trained on the training data" but that doesn't rule out tweaking across several experiments and measuring them against the known test data, then cherry picking the experiment that was best overfit to that data, right? I'm not saying there are shenanigans here, just wondering how you know there are not.

Did they train also the human on the training data ?

The test labels are not known to them. The test labels are held by a different team that tells them their test error rate. Furthermore, multiple, separate teams have surpassed human performance.

FWIW, "human performance" is not as meaningful as most people make it out to be, because this is such a narrow task.

The paper mentions it's unfair to humans to have to tell say a coucal from an indigo bunting (both blue coloured birds). I've never heard of either so I guess I would have got that wrong but it's not the fault of my visual system.

But you only have to guess right once in 5 tries!

I replied to abrichr's link. Did you look at it at all?

They used a dataset from 2012. Are you saying that the test labels from a 2012 dataset have been kept secret until now? Really?

I am familiar with the ImageNet competition. Let me know if you find the ImageNet test labels. (Hint: they are not released, since they are re-used for competitions).

Interesting, did not know that. Do you think that proves this team was not able to get and exploit some inside knowledge? Judging from recent competitions, ethics does not seem to be a strong point in some cultures.

Reply on news.ycombinator.com