Hacker News Reader

The main advantage of Big Bird is its linear complexity in sequence length. If it were to be trained on the same corpus as GPT-3 what would be the advantages/disadvantages?

I am thinking maybe longer context window, faster training and less memory use, but what about performance, will it measure up?

View on HN

This is something we want to explore. BigBird just replaces the attention mechanism in BERT.

We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.

Reply on news.ycombinator.com