I'm curious: why can't you do tokenization in linear time? And why not use incremental tokenization for an editor?
And the go extension is a thin wrapper around standard go tooling, we weren’t tokenizing ourselves just converting between their tokens and ones we could process; a large part of that was converting from byte offsets to UTC-8 character offsets.