Did anyone else read the generations? They’re… really poor quality, right? Im not sure whether I’m misunderstanding though.
Actually, as far as I can tell - the RETRO arch itself isn't trained in this article. It focuses more on how to build the retrieval system with a fast KNN index over all of the Pile.