You might reuse simple LLaMA tokenizer right in your Go code, look there:

https://github.com/gotzmann/llama.go/blob/8cc54ca81e6bfbce25...