OpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding.
"If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js."
with the links:
https://huggingface.co/docs/transformers/model_doc/gpt2#tran...