That’s why cases are treated differently - they’re different in Unicode.
This is also the only way to teach a model how to properly capitalize things (since there are no human defined rules).
[0] https://towardsdatascience.com/byte-pair-encoding-subword-ba....
It's hard to argue that this information isn't (a) being captured by GPTs and (b) important. If you just threw it away, GPTs would have less information available to absorb.
A good example is the initially released BERT-multilingual-uncased model back from the first BERT paper, which (without even mentioning it anywhere) not only collapsed the case but also removed diacritic marks from latin characters, thus killing its performance on those languages which heavily rely on them.