Tokenizer benchmark comparing 16 language models pre-trained from scratch
old.reddit.com
old.reddit.com
> Summary of Findings:
> - Comparable (50256-strict-nocapcode) TokenMonster vocabularies perform better than both GPT-2 Tokenizer and tiktoken p50k_base on all metrics.
> - Optimal vocabulary size is 32,000.
> - Simpler vocabularies converge faster but do not necessarily produce better results when converged.
> - Higher compression (more chr/tok) does not negatively affect model quality alone.
> - Vocabularies with multiple words per token have a 5% negative impact on SMLQA (Ground Truth) benchmark, but a 13% better chr/tok compression.
> - Capcode takes longer to learn, but once the model has converged, does not appear to affect SMLQA (Ground Truth) or SQuAD (Data Extraction) benchmarks significantly in either direction.
> - Validation loss and F1 score are both meaningless metrics when comparing different tokenizers.
> - Flaws and complications in the tokenizer affect the model's ability to learn facts more than they affect its linguistic capability.