Could someone explain the article to layman engineer?
Instead of training GPT-3 with 178B weights, you train a 25x smaller model and allow it to retrieve useful snippets from a large text index as additional information.
This solves the problem of very large models and the problem of updating an already trained model, as you can swap the text corpus with a newer one. The model learns mostly syntax, burning less trivia in its weights than a regular LM as it can simply copy the relevant information from the index.
This development was bound to happen as large LMs are expensive to use and it was an obvious idea. We've had these semantic search text indices for a few years already[1], they just weren't combined with text generation.
For many tasks it wouldn't be helpful because the input is small enough to be covered by the context already, and for summarizing and question answering tasks, you want it to repeat information from other documents, but not from earlier in its own output.
It might be interesting for a long-context task like "given the first parts of this book, complete the next chapter".