But inside on epoch there is a lot of duplication already.

By duplication I mean if context length is N there is many sequence of N word that are not unique.