Hi, I'm one of teh C# language designers, and I work on the compiler implementation as well.

C# has never had a "simple tokenizer". Indeed, even the first language has complex lexical constructs that are part and parcel of the language. For example, our comments can store structured data in them (like xml).

> A simple loop that would have worked with the 70s era of programming languages

Yes. But 70s era compilers had to deal with things like not having enough memory to even store basic amounts of data. It also had to work in spaces where things like a 'stack' was just not tenable. We're literally 50 years from that point, and having a compiler do stuff like keeping a stack is not an issue anymore :)