Hacker News Reader

Very interesting idea! Reminiscent of this robust learning paper[1] from 2020, where they do something even simpler: zero out elements in gradients with inconsistent signs, before averaging them.

[1]: https://arxiv.org/abs/2009.00329

View on HN

Hey thanks! Ya we tried similar strategies to this and could not beat cosine distance < tau, average, else skip. It was too much to put in the paper and we may put it in the arxiv version but we tried Sign AND gating and zero’ing out if the signs don’t agree, we tried L2<tau, etc but nothing beat cosine distance.

I'm also reminded of top-_k_ training/sampling, where you throw away the inconsistent samples: https://arxiv.org/abs/2002.06224 "Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples", Sinha et al 2020.

Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?

Reply on news.ycombinator.com