$"A{new List<string>{$"B{"{C}"}D"}.First()}E"
This means that a C# compiler can't start with a simple tokenizing loop. That compiler phase would have to keep track of state in a stack, recording what each } character means while its still looping through code character-by-character.
Now we're adding {{ and }} into the equation. Yay.
Not true, it just means that the parts between double quotes aren't bunched into a single token.
To figure out how the compiler makes sense of this code, try
https://roslynquoter.azurewebsites.net/
You'll see how it tokenizes the string.
A simple loop that would have worked with the 70s era of programming languages, would go through each character and once the boundary between two tokens has been identified, write out a token to a one-dimensional list. This would be a mostly stateless loop, tracking enough state for the current token in hand only. The next phase would go through the tokens and pair up brackets, etc.
A C# tokenizer can't do that. It needs to keep a stack of state. When it sees a '}', it needs to know if that's a "normal" brace or the } that resumes a interpolated string literal.
I was writing a tokenizer myself and I wanted to have something similar to string interpolation. I very quickly realized my simple loop that I would have written for my CS degree isn't going to cut it and I had to start over.
C# has never had a "simple tokenizer". Indeed, even the first language has complex lexical constructs that are part and parcel of the language. For example, our comments can store structured data in them (like xml).
> A simple loop that would have worked with the 70s era of programming languages
Yes. But 70s era compilers had to deal with things like not having enough memory to even store basic amounts of data. It also had to work in spaces where things like a 'stack' was just not tenable. We're literally 50 years from that point, and having a compiler do stuff like keeping a stack is not an issue anymore :)
Although you did deny me an excuse to not have to start over. If the C# team didn't bother, why should I? (Except you did bother.)
Hi, i'm the lang designer and feature implementor here :)
The complexity of lexing/parsing did not get worse here. We actually just lex/parse this stuff the same way that interpolated strings have always been lexed/parsed. This has been supported in the language for almost 10 years at this point :)