A simple loop that would have worked with the 70s era of programming languages, would go through each character and once the boundary between two tokens has been identified, write out a token to a one-dimensional list. This would be a mostly stateless loop, tracking enough state for the current token in hand only. The next phase would go through the tokens and pair up brackets, etc.
A C# tokenizer can't do that. It needs to keep a stack of state. When it sees a '}', it needs to know if that's a "normal" brace or the } that resumes a interpolated string literal.
I was writing a tokenizer myself and I wanted to have something similar to string interpolation. I very quickly realized my simple loop that I would have written for my CS degree isn't going to cut it and I had to start over.