This web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers.
Everything runs in-browser thanks to Transformers.js.
Everything runs in-browser thanks to Transformers.js.
No comments yet.