Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet.