* NUL terminated strings (and now, non UTF-8 encoded strings on input/output)
* Using LF or CR or CRLF as line terminators, and pipe/comma-delimited fields when there were other unambiguous ASCII characters that could have been used (eg, GS, FS, RS) that would have made the encoding/decoding of line termination an I/O thing keeping HT/VT/CR/LF/FF as literally print related codes.
I suppose you could document that it's unsupported, and just drop or reject such values, but then the system couldn't be used to handle test data for such systems, for example.
Otherwise you need to have some sort of escape mechanism, exactly like quoting strings in CSV. In fact, there's an ASCII code "ESC" for entirely that purpose. :)
The problem is that those characters are non-printable, which means if you're just dumping the file out somewhere, you can't see them.
NL Next line (from EBCDIC?)
LS Line separator (invented by Unicode)
PS Paragraph separator (same)
The Unicode standard says that in addition to CR, LF, CRLF and the above, vertical tabs and form feeds should also be treated as line separators.
UTF-8 on stdin/stdout works perfectly fine (unless you are on Windows of course, which is stuck in in the early 90s when it comes to international text encoding).
> Using LF or CR or CRLF as line terminators
This is also an operating system convention, and it would be better if programming languages wouldn't try to "guess" the correct line endings, since this causes more problems than it solves - but again, this is mostly a Windows specific problem, and it's Microsoft's job to finally bring Windows into the current century.
Unix used LF, Apple used CR, Microsoft used CRLF.
They are all ASCII carriage movement codes, which is about driving the paper feed and print head of an ASR-33 or equivalent.
So they all made the "wrong" decision about what to store in a file.
They just chose different wrong characters.
Unix followed Multics. Multics chose right. ASCII/EMCA-6/ISO646 drafts discussed this at least as early as 1963¹: “For equipment which uses a single combination (called New Line) [...] NL will be coded at FE₂ [Field Effector 2 = 0x0A].”
¹ doi/10.1093/comjnl/7.3.197
I'd say Multics/Unix was technically correct, except this was still the wrong decision for I/O ever since.
The Record Separator is the logical character code to use to indicate the end of a line of text and print position characters, assuming that a line of text is a "record".
Apple hasn't been using CR since the release of OSX (26 years ago). Microsoft could have made the switch at any time too (just as they could have switched to UTF-8 as universal text encoding on Windows), they just choose not to.
In the end it's not the job of programming languages to clean up Microsoft's mess ;)
Why is it Microsoft's fault? They just stayed on their legacy implementation, Linux and Apple chose to move from the legacy implementation to another legacy implementation. That seems dumb.
The issue is that none of the print carriage movement ASCII characters should be used internally to indicate "end of line", because each of the chosen possibilities are used separately to indicate different carriage movements.
The logical decision would have been to choose one of the "separator" characters to indicate "separation of one line from another" and then allow the I/O drivers to decide what to send/receive to/from a particular device.
The fact that both Apple's and CP/M codes came out roughly at the same time, both on microcomputers, shows that it was probably just a design decision.
Some believe Gary Kildall picked CRLF for CP/M since he used DEC TOPS-10 to develop CP/M. see https://www.quora.com/Why-did-CP-M-stick-with-the-CR-LF-stan...
Last time I had to handle CSV files in bash, I converted them internally to RS and FS.
CR is the only wrong choice. There's never a reason to go to start of line without erasing the line or moving to next line in a file. And even user interfaces will have smarter ways to do that. It's a completely useless concept outside of typewriters.
Well, CRLF (or worse, LFCR) is also obviously a wrong choice because it's pointless to demand two characters and create problems when one of them is missing when one totally unambiguous character will do.
I would just use UTF-8 everywhere.
Take into account that some Unicode codepoints work together to combine to form a grapheme which then links to a glyph for display.
If you use UTF-8 internally, you will be expanding out to full 32 bits when scanning the vector anyway.
So if memory isn't an issue (and most of the time it's not), indexing a vector by codepoint (ie 32 bits) makes more sense from a processing POV.