4/5 of today's top CNN articles have words with periods in them: "Mr.", "Dr.", "No.", "John D. Smith", "Rep."
The last one also has periods within quotations, so period chunking would cut off the quote.
The last one also has periods within quotations, so period chunking would cut off the quote.
https://github.com/KnowSeams/KnowSeams
(On a beefy machine) It gets 1 TB/s throughput including all IO and position mapping back to original text location. I used it to split project gutenberg novels. It does 20k+ novels in about 7 seconds.
Note it keeps all dialog together- which may not be what others want, but was what i wanted.
But as the number of files to ingest grows, chunking speed does become a bottleneck. We want faster everything (chunking, embedding, retrieval) but chunking was the first piece we tackled. Memchunk is the fastest we could build.