Hacker News Reader

Comment by lidHanteyk | Hacker News Reader

I don't understand why the author thinks that the PAC learning framework is not useful. PAC gives quick back-of-napkin math for estimating how long it might take to train a network, what kind of quality level is acceptable for inputs, etc.

View on HN

I disagree, PAC bounds are usually way too loose to be useful in practice when it comes to NNs. I think part of the reason is that the way we measure model complexity for NNs does not correspond very well with their ability to generalize, as we've seen with recent papers like deep double descent [1].

Anecdotally, I was of the main authors of the AdaNet framework [2] where we used (approximate) Rademacher complexity (closely related to PAC) to bound the complexity of our learned ensemble models. We never got good results using the complexity measure and would basically turn it off for any practical problem we solved.

[1]: https://openai.com/blog/deep-double-descent

[2]: https://github.com/tensorflow/adanet

Correct me if I'm wrong, but doesn't PAC on real data usually just tells you that "it doesn't know the answer"? Like, it says the equivalent to: "I think this is the answer, but it might be miles off and I give it 50% it is even more than miles off." I found that it is able to give hard guarantees, by being very loose on the bounds. Do you have a real-world example where it actually gave something useful? I'm genuine curious, because I discarded it early on, but am willing to revisit it.

PAC does not provide these types of estimates in most practical situations.

Reply on news.ycombinator.com