This maps a little to what we are doing research on what we are calling as shape of stories[1].

We can clearly see in 2D space itself how different "concepts" are explored.

Using the shape of stories for semantic chunking we can clearly see in multiple articles how we can chunk by "concepts". [2]

Now we are trying to see if we can just use these chunks and train a next "chunk" predictor instead of a next word predictor.

In the paper, they take a sentence to mean a concept. We believe that a "semantic chunk" is better suited for a concept instead of a sentence.

[1] https://gpt3experiments.substack.com/p/the-shape-of-stories-...

[2]https://gpt3experiments.substack.com/p/a-new-chunking-approa...