Unitools – A suite of tools for working with Unicode in the browser
unicod.es
unicod.es
Unicode is ideal for storing text exactly as the user wanted it. Which may be crazy.
But when I am writing a program for internal use, or creating a communication protocol, or trying to parse one program's output into another, or trying to make a 7-segment display show some characters, I don't want to have to handle these crazy possibilities. Just text is fine, thank you!
As long as there's no BOM, anyway.
Any ASCII text is ipso facto UTF-8 text.
UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII.
Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode).
When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning.
Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
(I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII?
Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.
In what SPECIFIC way is an only-ascii string not a UTF-8 string for ALL intends and purposes?
You throw some words like "semantics" but no actual arguments.
Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc).
The difference isn't the representation; the difference is the interpretation that we attach to the representation.
And Unicode has a non-negligible footprint.
I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU).
Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ...
I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.
You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
Yes. String processing often occurs in a loop (like for trim, contains, split ...).
You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into.
Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now.
It was... horrible.
Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text.
It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.
For example, I can easily verify that it has no control codes, or is "plain text".
There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
For Unicode, I don't even know where to look up how to do it.
Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.
That's not because Unicode is not "just text" whereas ASCII is.
It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain".
We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding).
(I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.
Unless I read the Unicode spec and expect my inputs to also have read it.
It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji).
And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
> For example, I can easily verify that it has no control codes, or is "plain text".
> There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> For Unicode, I don't even know where to look up how to do it.
That's not a problem with unicode.
> Then there are these special encodings in UTF-8 to shelter binary data which failed to decode.
There are no special encodings in UTF-8. What you're talking about is error-handling strategies
> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
The exact same question exists about ASCII, since \xFF is not valid ASCII.
> Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
It's painful that your random manglings don't make sense?
> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
They don't.
> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
Which is already an issue in ASCII.
> So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII
It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did.
And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
I just want to add to what you said:
> > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> > For Unicode, I don't even know where to look up how to do it.
UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4].
In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document.
1. http://www.unicode.org/reports/tr14/
2. http://unicode.org/reports/tr29/
3. http://site.icu-project.org/
4. http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)
I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up.
I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y.
I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not.
I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.
ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.
And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic.
Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.
> That's not a problem with unicode.
So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider.
>> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
> The exact same question exists about ASCII, since \xFF is not valid ASCII.
Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible.
> It's painful that your random manglings don't make sense?
If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks).
>> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
> They don't.
They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong?
>> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
> Which is already an issue in ASCII.
Some common sense helps here. Show me what serious fonts don't support the ASCII print range.
> And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
I'm totally with you.
"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky
http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two...
> The use of "i" resulted in an SMS with a completely twisted meaning: instead of writing the word "sıkısınca" it looked like he wrote "sikisince." Ramazan wanted to write "You change the topic every time you run out of arguments" (sounds familiar enough) but what Emine read was, "You change the topic every time they are fucking you"...
After all, the turkish encoding wasn't ASCII in the DOS/Unix years but ISO 8859-9 (and various other, e.g. an IBM one etc).
OK.
When I was in PHP, I often used "htmlspeciachars()" and it does this kind of transform. That's why I tried that.
http://php.net/manual/en/function.htmlspecialchars.php
I had some problems too with " " because it's 160 in ASCII and not a whitespace (32).
Edit: 160
Yes! You're right.
>... I'd say they're a far nicer choice these days.
It was a long time ago for a scrapping project (garbage HTML).