Approximation of the Claude 3 tokenizer by inspecting generation stream

The Claude 3 tokenizer seems to be slightly broken for code. Claude 3 gives really good answers but sometimes long code snippets seem to be corrupted with missing chunks and weird indentation.

Might try this out and see what’s going on