Picking Apart the Crashing iOS String
manishearth.github.io
manishearth.github.io
I am not surprised there will be errors/bugs in the typesetting algorithm, as the OP demonstrates, this stuff is extraordinarily complicated to do for every language/all of unicode. But if you have something so complicated that bugs/errors aren't surprising -- you'd think you'd want to make sure they didn't cause hard crashes.
Any Apple people want to chime in? Why wasn't the general bug -- where there could be even the possibility of a "text crash" -- fixed?
You answered universally for all contexts, ever, everywhere. That's a usually quite a foolish thing to do. In ObjectStudio Smalltalk, there was actually a place where you could define an empty lambda as the "top-level" exception handler. Once you did that, all Smalltalk exceptions did nothing. There you go: a "don't ever crash" fix, in a language many would call "unsafe." You are now technically wrong, which is the best kind!
Exceptions can be caught in C++, Objective-C, and in Swift. You can even do this for C. There is apparently a history of similar bugs where certain data crash processes in iOS. Depending on how you count, this is either #3 (strings) or #5.
https://www.theverge.com/2018/2/15/17015654/apple-iphone-cra...
Given that, why wouldn't Apple take steps to make sure that certain critical processes are architecturally immune to this sort of thing? Springboard going away is pretty horrendous. Messages app, given that it's a core functionality for a phone, is almost as bad.
if they're using an unsafe language, there's always the possibility of more such issues.
There are software projects where certain things simply can't be allowed to happen, ever. Apparently, Apple isn't operating at that level.
It's the not even 2nd rater who gives up on something which causes such a huge hole in the user experience, saying to themselves, "Uhhhh, there's no such thing as a never crash fix." You're not a 1st rate programmer if you only think one step ahead and say, "Uhhh, you can't guarantee no crashes," then leave such a huge hole in your system. The 1st rater engages in a bit of lateral thinking. The real problem isn't to eliminate crashes. The real problem is to eliminate the hole in the UX! Anything you can detect, you can "fix," and sometimes a guaranteed "fix" every bit as good or better.
(It's exactly this kind of mediocre thinking that led to the hardware quality doldrums in the 90's. The OS crashed so often, hardware manufacturers started to make cheap machines that could only stay up for a few days anyways.)
2nd Sentence of the article says text box but not sure what "other places" are:
"Basically, if you put this string in any system text box (and other places), it crashes that process."
From the article: "I’ve been testing it by copy-pasting characters into Spotlight so I don’t end up crashing my browser". "I can cause this crash to happen more reliably in browsers by clicking on the string".
So, not text-box specific and happening on display for a wide variety of applications, including web pages.
The fix would be to rewrite CoreText in a memory safe language like Swift. This would be “hard”. Or put CoreText in an XPC container. This would both be “hard” and result in terrible performance.
For more details on how hard C, memory management, systems programming, and operating system development is please refer to your local copy of Modern Operating Systems by Andy Tannenbaum.
Simpler languages like Fortran can turn into even faster code than a C implementation. UB optimizations aren't that relevant for real-world performance.
Code generated by C compilers for C64, Spectrum, Atari, Atari ST, Amiga, Mac, CP/M, MS-DOS, Windows 3.x, Nintendo, MegaDrive,... systems meant many times the code would be 80% like this:
void some_func(/* params */) {
asm {
/* actual "C" code as inline Assembly */
}
}
The code that current C compilers don't generate, many times is related to taking advantage of UB.
They also generate extra code for handling stuff like floating point emulation though.
Just as an example, IBM did their whole RISC research using PL/8, including an OS and optimizing compiler using an architecture similar to what LLVM uses.
They only bothered with C, after making the business case that RISC would be a good platform for UNIX workstations.
>> The code that current C compilers don't generate, many times is related to taking advantage of UB
Compilers are really smart in optimizing things that aren't relevant to the real world.
For example, this code would reduce to "return 32" in most modern compilers:
int return32(){
int x=1;
for (int i=0; i<5; i++){
x*=2;
}
return x;
}
>> They also generate extra code for handling stuff like floating point emulation though.
Not necessarily.
To clear up the myth among young generations that C compilers always generated fast code, regardless of the platform.
As for something more modern, in 1998, C code quality was still at a similar level to other system's languages, before they started to fade away thanks to the increase in UNIX, Linux and BSD adoption
For example, given that Delphi and C++ Builder share the same backend, their generated code was quite similar, even if it would require disabling some of the Delphi's security checks.
> Not necessarily.
Sure, it all depends on the CPU being targeted.
Even better, when you failsafe you plan for the (unknown) future.
That's why we have circuit breakers, hydraulic and electric fuses, pressure relief valves, etc. Because no one thinks they can know all things that can go wrong in the future (with catastrophic consequences) and plan for that
It’s also a recognition that in complex systems, something unanticipated is going to go wrong sometimes, and rather have a plan for handling the failure than pretend that the system will never hit a really bizarre failure mode.
Your circuit breaker analogy made me think of this.
The big issue is that when CoreText crashes right now the kernel panics and the device restarts. If CoreText itself could crash safely, get restarted, and the OS continue running then these bugs would go from "significant" to "annoying." Even if CoreText crashing caused individual apps to also crash, that would be a big improvement the current situation.
Obviously we'd all like bug free fonts and text rendering, but if we call that goal aspirational (read: impossible), the best we can hope for today is handling the fault cases better than they're handled today. Bootloops are a pretty lame user experience.
Maybe I missed something though.
Pissed my niece the hell off though, that I could remotely disable her Messenger.
On the other hand I had the same thought (but better self-restraint than you) - however I was thinking "nah, scrubbing bad messages server side is just an s/badstring// and I am sure the major non-encrypted messenger apps (where the server knows the strings) added that server-side, so people couldn't crash their contacts' apps, which the app company might get blamed for. This kind of hotfix shouldn't have negative effects, I'm sure there are already a few server-side manipulations of text (stuff like adding a space to very long lines, maybe a blacklist of certain malicious URL's that sort of thing.)
So I'm surprised your message was delivered as sent (if it's not encrypted end to end), unless you did this right when the news broke.
If it's possible to write "Your message could not be delivered" when messages match the blacklist (even leaving the sender to guess at what they did wrong) it would be better.
As a practical matter if you haven't built the infrastructure into your clients to tell the sender that their message won't be delivered, none of the choices the platform operator has seem great:
- Silently drop a few kinds of messages without informing sender. Seems bad for the reason you outlined.
- Silently modify messages before delivery, modifying them so they won't crash clients. This seems potentially very wrong.
- Deliver messages even if you know for sure they will crash the client upon view
Doesn't seem great to me either.
I guess the real solution is to have robust forced-upgrade on the client (after all, it's your software, you're responsible for it and if you build it to include updates that it is on you) but some users object to that and I suppose they could be justified - it is also a massive responsibility.
I guess there really aren't any perfect answers here.
In text layout you do a lot of indexing into various arrays -- like the array of code units of the input string, or an array of metadata collected per-code point, or an array of data collected per-grapheme. Oftentimes those arrays are all the same length (like in simple text like Chinese) and mixing up which index to use where is no problem. And then when it goes wrong you're violating array bounds which is a crashable offense.
I agree that it's stupid to have code that causes crashes here. My only defense is that I had a lot of other work to do and complex test pathological cases only affect a fraction of users, none of whom were the ones yelling at me about other bugs. I am rooting for Servo, where they are using a language that defends against bad programmers like me.
PS: If a web page wants to crash, it easily do so by allocating memory in a loop, so making web pages that crash isn't as exciting as it is in Core Text in general. Of course crashes can often be escalated into RCE, but the Chrome sandbox was there to mitigate that.
In an exception based language you might forget to deal with the error and have it crash "higher up" in the code.
I also suspect that there might be a performance benefit but I could be completely wrong about that
Checked exceptions are explicit as well, though I'm not aware of a language that _only_ has checked exceptions.
> I also suspect that there might be a performance benefit but I could be completely wrong about that
Yes, this is a big advantage. Stack unwinding is very expensive.
Depending on compile-time options, a Rust panic can either cause an immediate crash, or do something similar to throwing a C++ exception, complete with stack unwinding.
Whereas for this bug it's quite possible that it may be exploitable. Especially given that the crash backtrace doesn't always appear in the same place -- something is corrupting memory that gets discovered later. (This explains why I can sometimes get the string to render for a split second before crashing)
isn't this also a bug :)
And probably want we should fix.. I think all browsers are susceptible, though last I tried in Firefox one had to not just allocate but also fill the memory with garbage.
For instance, this site will crash most Firefox browsers but not Chrome because they limit the modal dialog rates: fan-pages[dot]herokuapp[dot]com (be warned that it may crash your browser even with JS disabled).
Maybe, or maybe it would be better to exponentially decrease CPU time awarded to a tab over time. And ask the user to confirm that this tab should be allowed to do CPU / memory intensive computations, and otherwise reduce the frame rate, number of ticks, or crash early.
Maybe, big memory allocations and CPU intensive stuff should only be allowed on background workers. And even then, we should still reduce resource allocations to save battery, etc. and allow users to award more resources.
> He means crash the single web page doing the loop... how would you fix this?
It used to be that you could bring my entire system to halt by allocating too much memory in the browser (mostly chrome). Mostly due to swapping and just plain bad system configuration I suspect :)
To allocate memory in a loop, you need some control over the JS. Websites try hard not to serve untrusted JS.
But websites serve untrusted text without a second thought. For example, I could post a comment on a news article and cause the article to be unviewable by anyone with a vulnerable browser.
Why is this a crashable offense? Probably a bit naive as a C# application developer, not a system developer, but shouldn't all this code be in the equivalent of a try/catch? If it throws an out-of-bounds error, just log the error and return, rendering what you've got.
And even if you don't disable it, out of bounds array access will cause segfaults, not exceptions (unless you're using some abstraction over arrays).
Looks like a heap corruption, leading to a failure when allocating memory later on:
Application Specific Information:
abort() called
*** error for object 0x604000673e00: Invalid pointer dequeued from free list
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff73a30e3e __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff73b6f150 pthread_kill + 333
2 libsystem_c.dylib 0x00007fff7398d312 abort + 127
3 libsystem_malloc.dylib 0x00007fff73a9bdbf nanozone_error + 502
4 libsystem_malloc.dylib 0x00007fff73a8fdac _nano_malloc_check_clear + 410
5 libsystem_malloc.dylib 0x00007fff73a901d7 nano_calloc + 72
6 libsystem_malloc.dylib 0x00007fff73a8acc0 malloc_zone_calloc + 87
7 libsystem_malloc.dylib 0x00007fff73a8b5d6 calloc + 30
Should look something like this:
$ lldb /Applications/Safari.app/Contents/MacOS/Safari
(lldb) env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
(lldb) process launch
Perhaps this sounds a bit Anglocentric, but isn't it unfortunate that this string crashes the devices even of those people who have never heard of and likely won't ever need to use the language it's written in? The majority of people use a tiny fraction of Unicode --- the parts that cover the languages they use; everything else is useless to them --- or in cases like this, even a liability. It would greatly reduce the number of affected devices, especially with bugs like this having possible security implications, if the text rendering system were more modular and perhaps divided into separate optional components: Latin (maybe not optional), CJK, and other complex scripts. I know Windows has/had a similar feature:
https://msdnshared.blob.core.windows.net/media/TNBlogsFS/Blo...
This way, those who have no need for anything other than Latin scripts get the simple and hopefully much less buggy rendering algorithm, while those who do need the others can do so without unnecessarily burdening everyone else.
That Windows option is specifically for:
- linebreaking support for east and southeast asian languages: Many of them don't use spaces, so if you want to know where you can break lines it is best to know what the words are. For this you need to have a pretty large dictionary file stored.
- fonts
- probably various assets for RTL languages (changed backgrounds, changed layout files, etc)
This is Windows providing this option for a very specific thing (and also downloading some files), for which there is okayish fallback functionality. Not cfg'ing the entire text stack.
I don't think you can easily slice and dice a text stack so that you can only pull in the components needed for Latin scripts without making it even more prone to bugs. You could write entirely separate stacks specialized for each group of scripts. But you'd probably end up with one for Latin/Cyrillic/Greek, one for Chinese/Japanese (not Korean) and one for "all the rest". There's enough feature overlap between most of the complex languages that there's questionable benefit to separating that out.
For example, most of the underlying text functionality in Telugu or other Indic scripts is not overall different from things in Hangul or Arabic (Arabic is more complex, actually), it's just that Telugu has certain features that press the buttons in just the right way to cause this crash. Which means that if you want to prevent this crash for other language users, what you need to do is not attempt to render Telugu, not swap out the font stack.
Like, looking at the last iOS crash that happened -- with the Arabic text -- that was because chopping off the end of a string of Arabic text doesn't guarantee that the string will be shorter. Really, you can replicate this for most scripts, it's just subtler (Even English has support for this, if your kerning is extreme enough. As long as your stack supports kerning, it supports everything necessary to crash this bug). So, the root text stack functionality that lets this happen is necessary for all scripts, Arabic just ends up pushing the right buttons in the right order to cause a crash.
Many users in latin-alphabet locales use other character-sets all the time ¯\_(ツ)_/¯
[This post brought to you by KANNADA LETTER TTHA and KATAKANA LETTER TU.]
If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search.
But yeah, this is one of my theories about it. One of the previous crashes had to do with an Arabic string which got longer when you truncated it, which made snipping it to display in a notification have bugs.
It's interesting to see it's causing a segfault, i'd expect measuring bugs to cause clean assertions or shitty rendering. Which is why I'm also wondering if it's actually a disagreement on the number of "characters" in the rendered things.
> If measuring the sub-strings gives surprising results (sub-strings being visibly longer for example), this could cause the algorithm to fail in any number of interesting ways: for example if a binary search is used to locate the cursor position, it could break the invariants of the binary search.
Cursor positions are based off of grapheme clusters -- there's a defined algorithm for that. Though different parts of the system may disagree on the specifics of the algorithm causing such a crash.
However, that doesn't gel with the fact that it's only specific consonants causing this, all versions of UAX 29 do not consider any differences between Indic consonants for a single given script.
http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
> Grapheme clusters can be tailored to meet further requirements. Such tailoring is permitted, but the possible rules are outside of the scope of this document. One example of such a tailoring would be for the aksaras, or orthographic syllables, used in many Indic scripts. Aksaras usually consist of a consonant, sometimes with an inherent vowel and sometimes followed by an explicit, dependent vowel whose rendering may end up on any side of the consonant letter base. Extended grapheme clusters include such simple combinations.
> However, aksaras may also include one or more additional prefixed consonants, typically with a virama (halant) character between each pair of consonants in the sequence. Such consonant cluster aksaras are not incorporated into the default rules for extended grapheme clusters, in part because not all such sequences are considered to be single “characters” by users. Indic scripts vary considerably in how they handle the rendering of such aksaras—in some cases stacking them up into combined forms known as consonant conjuncts, and in other cases stringing them out horizontally, with visible renditions of the halant on each consonant in the sequence. There is even greater variability in how the typical liquid consonants (or “medials”), ya, ra, la, and wa, are handled for display in combinations in aksaras. So tailorings for aksaras may need to be script-, language-, font-, or context-specific to be useful.
For example, in Chrome, we added an extra rule to not allow grapheme clusters to be split after Indic virama characters, but later had to modify the rule to not apply to Tamil viramas:
https://chromium-review.googlesource.com/c/chromium/src/+/84...
I don't know the exact cause of this crash, but I can see why Apple might be running into trouble with their logic for these languages. I suspect their algorithm for computing grapheme clusters has a bug causing an inconsistency somewhere.
However, given that some Brahmic scripts prefer explicit viramas (Malyalam, also Thai I think), this will probably be restricted to Brahmic scripts where joining is always preferred (even if not possible).
I'd been testing UAX 29 stuff out before and Apple seems to follow the spec. For example, Chrome and Firefox seem to do special handling for e.g. flag emoji (distinguishing between regional indicator pairs that render as a flag vs those which don't -- i.e the ones which don't correspond to a country code). But Apple follows the spec rigidly. In particular it does not consider joined consonants to form a single EGC.
I could be wrong on that, though.
That suggests to me that Apple made changes to CoreText, and did not perform adequate regression testing.
All software has bugs. I understand that. But I suspect a large part of power users' and developers' growing frustration with Apple is that they keep introducing severe, kernel-panicking, root-exposing bugs in software that previously did not exhibit the problematic behavior.
Honestly, how do you not have a stringent regression testing requirement for changes to the "Core" of the operating system?
By not managing expectations and deadlines properly. Especially if schedules are overriden from a manager who is not aware of the ramifications of changing something in the core.
Irony aside: you are, perhaps unintentionally, omitting from your narative all and any damage that would be caused by such a deliberate bug - the vendor is usually the only one who can fix it, but not the only one who can exploit it. Also, what of unpatched devices, and of liability (you are introducing a backdoor, intentionally)? And realistically, your original change might introduce more holes than you bargained for, or the fix might. This is a horrible idea on so many levels, even discounting it's inherent evil.
I have never had the fortune of working for a manager who prioritized quality over (sometimes imagined even) random dead-lines.
Developers' complaints sound like gibberish to them ("Plus, they always complain, such perfectionists!") and when the shit hits the fan, sometimes years later, well, it might not even be their problem - they've been promoted by then, fucking up another unfortunate team.
I think perhaps Tim Cook doesn't realize that Apple didn't create all this emotional loyalty from artists just by having pretty boxes and wallpaper, but by cultivating a kindred spirit with them. And to do that, you have to actually care about the quality of your work on a fundamental level.
This particular bug shows the problem with multibyte characters (terminology may be wrong, perhaps multi glyph is better) where certain parts of the character become left or right associative based on context.
क्र is currently not a grapheme cluster, for example (there is a proposed change to UAX 29 to make it such, but that doesn't handle all such cases).
The strings in this situation are three grapheme clusters according to the spec even though they usually render as a single "thingy".
Also, do you learn the large number of graphemes separately? Or are they the "obvious" way to write the consonant and vowel? Or is there a set of rules you learn?
- You normally think of clusters as single "letters". The word for them in Marathi is "joined letter". This notion of "letter" may not be totally in line with what English speakers would expect.
- You still think of clusters as having component parts that make it up. However, क्ष/ज्ञ and sometimes त्र are thought of as their own "fundamental" consonants even though they're clusters. I think folks think of things like क्र as being both -- a letter of its own, that is built up from component letters.
- Most of the clusters in Devanagari are predictable. So for any given pair you usually know how to make a "half form" for the first one and join it on, or you know how to stack them. Usually whilst reading it's obvious because you see the components; and whilst writing it's usually fine to try what makes sense.
- For unpredictable clusters, you just eventually get used to them. No different from learning that some words are spelled weirdly in English, really. Though I very much dislike ह्म , ह्य, and द्य since they're kinda confusing.
I think Bengali has a lot more unpredictable conjuncts; I stumbled across this a lot when I tried to learn the script a year ago. I suspect folks just get used to those.
There seems more subtlety with the word structure, like tiny little flicks which seem to have significant meaning. While Latin-like script has characters like the comma/full-stop, and modifiers like the dot on the i, these are far rarer than Devanagari's modifiers from what I have seen.
Another related question is, because Devanagari's information density seems higher, does that mean shorter text for the equivalent information?
PS - Sorry if anything I said was offensive, that wasn't my intent. I am just an ignorant idiot wanting to learn someone's experience with a different language script, nothing more.
The consonants of Devanagari are easy to tell apart; they are pretty different. In most cases the "similar" ones are basically with an extra line (प/ष, ब/व) or with a little loop (य/थ, ट/ढ) . (The line/loop has no semantic meaning, these letters are just random letters that look similar) . The only super annoying one is घ vs ध -- the thing up top is a loop in the second one, but depending on the font/handwriting can be rather unclear.
The vowels are also easy to tell apart. There are 12 main ones, and they're made up from some really basic components which are quite distinguishable.
So you can easily distinguish consonant+vowel combos.
Consonant clusters can get tricky, like I said there are a bunch of ambiguous ones. Fortunately you end up realizing which is which and then it's nbd. But like, द्म and ह्म can be infuriatingly similar and you just get it from context.
Bear in mind, at one stage you start sight-reading words, so the actual details of the word matter less.
> Another related question is, because Devanagari's information density seems higher, does that mean shorter text for the equivalent information?
Visually? Yeah. Words can be really small. But it ends up roughly being the same number of code points (and probably more bytes in UTF-8).
It really annoys me when websites/apps keep the same font size for Chinese that they use for English. 8pt Chinese is awful to read.
It does not apply to text using absolute font-sizes however, just text that uses things like font-size:medium (or inherits the document default font size, which is also medium)
for example: letter "Ka" క becomes "Kaa" కా by adding "Ka" and "A" like A in apple.
అ becomes the line and circle that gets added to the consonant.
Consonants are not mixed with each other, cannot be.
this seems like it could help for those interested in more info
For learning, you just learn the sounds separately for each vowel and consonant. So you can read & pronounce anything written in Devanagari script but not understand it if its a different language :)
/peɪ/ is a syllable in English, but can be represented by these grapheme clusters: <pay>, <pei>, <pai>. The similar syllable is represented in Telugu or Devanagari this way: పే (Telugu script) or पे (Hindi/Devangari script); there are phonetic differences in the way this syllable represented: (a) /p/ is aspirated in English, but not in the telugu/devangari syllable; (b) the diphthon /eɪ/ is realized as a monophthong in Telugu/Devanagari scripts.
<ja, virama, nya> sequence plays a major role in Sanskrit sandhi. In Sanskrit, in assimilatory contexts, a voiced palatal affricate /dʒ/ + a palatal nasal /ñ/ + some open vowel, leads to this sequence <ja, virama, nya>. Since this assimilation occurs in many contexts, the resultant grapheme is taught as part of alphabet in this special set of clusters: kSa, tra, jna ( క్ష త్ర జ్ఞ) or (क्ष त्र ज्ञ). In fact, there was a poet in Telugu, whose name was ksEtrajna (combo of these three graphemes with ligatures). https://en.wikipedia.org/wiki/Kshetrajna (in general) https://en.wikipedia.org/wiki/Kshetrayya (Telugu poet)
The context of jna in Sanskrit: for instance, declension of the masuclar stem: rAjan http://littleredbook.vedicsociety.org.in/noun-tables/rajan--...
There's two types of letters. One type is used as an addendum to the other type. The first type are called vowels and the second consonants (I'd say that's mischaracterization but whatever). In this[1] picture, the top half of the alphabets are the type which are the addendum part, and the bottom part are the 'primary' ones.
So how it goes is that one of the bottom always comes with one on the top. Most often it tends to be the very first vowel, which is pronounced 'a:', and it is the default way letters are learnt(pronunciation chart[2] for each of the vowels).
So each of the consonants (the bottom part) can be used with any one on the top. Let's take the very first consonant. It is pronounced 'ka'(because by default, consonants are accompanied with the first 'vowel' which is 'a:'). Now, lets take a look at the the bottom part of picture [2]. That same consonant is pronounced differently when accompanied with each of the first one. For example: kaa, kee, kuu, etc. The first half the the pronunciation is the same, but the second half, is replaced which whatever vowel's symbol is attached as an addendum. Also notice the the center part of the alphabet hardly changes visually between changes in vowels.
Not to throw a lot at once, but there's another part. Each of the consonants can again be used as addendum to any other consonant (which always comes with a vowel). Take a look at this[3]. Notice the little tick at the top you saw in [2] is still there? That picture contains the letters 'ka', which each of the consonants being tacked on. The vowel is always still there. That's because, as I said consonants always come with a vowel attached. The consonant addendum is optional though.
Congratulations, you can now pronounce (almost all of) Telugu (and almost any Devanagari script using this same play book). It is because these scripts are deterministic, and there is a single pronunciation unlike English and many other languages.
[1]:https://www.telugucorner.com/images/aksharamulu.jpg
[2]: https://www.omniglot.com/images/writing/telugu_vwl.gif
[3]: https://www.omniglot.com/images/writing/telugu_conjuncts.gif
weeeellllll, not exactly. For example, spoken Marathi differs considerably from written Marathi, with vowels turning into other vowels, etc. My favorite example is the word लहान which is spelled "lahaan" but is often pronounced pronounced "lhaan" OR "laahaan".
Almost all Indic language experience schwa deletion -- you will say the word "kanpur" with no vowel between the n and p even though it is spelled "kanapur". Anusvars have a range of pronunciations that depend on the word being used -- they can nasalize a vowel, or add a nasal consonant like n or m.
However, these scripts are definitely way more phonetically consistent than English. These differences are minor.
Take for example Hyderabad. Looking at the Hindi spelling, it should be pronounced Hy-der-aa-bad. But it it pronounced more like Hai-dra-bad.
What I'm saying is that people made those changes which we got so used to in our day to day lives that most now regard them as the 'correct' pronunciation, but it isn't how they were supposed to be.
Pronunciation changes over time. Things which may have been highly phonetic may lose this property over time.
The bottom if used by itself would be called "na"
So basically the grapheme you see is made by joining to consonants Ja and na ..so it basically becomes "Jna" (spelt gna)
there are specific rules like english but like any language we have to learn these rules..eg I know that I can add k+ nowledge and that makes something in english..
But what if I add ra+no "rnoledge", yes you can add but that sounds weired, not sure if there is any rule which says you cannot add R+na, same with telugu.
My idea would be just to create a program that continually creates Unicode strings and display them on screen with a pause before creating and displaying the next. Record the screen and see if any thing exciting happens. Would never end, if there was a SETI @ Home style program for it with insentives for running it would be cool.
iOS emulator (part of XCode) might be enough.
The only thing that I found somewhat confusing and perhaps a bit un-Apple was the fact that the installer would not prompt me to partition and format the HDD first, but was perfectly happy to let me try to select and then fail to install onto the install media itself (with an odd "not enough space" message); I had to use the Disk Utility to do that before going back to the installer. I've always wondered why --- even Windows' installer includes partitioning and formatting as one of its steps.
After they did this my phone would not recognize its sim card. You see my phone originally had AT&T service. Then I switched to sprint some months back. I guess the dfu restore ended up relocking my phone to AT&T. At least that's what the Apple geniuses said. Then they said I had to go to a sprint store.
This is the first time I was disappointed with Apple's human help. I remember past experiences going in there with a frayed power supply cord - they gave me a new one. On other occasions they replaced my phone when they couldn't fix it. They never spent more than 30 mins trying to fix a problem and always fixed it. Their knowledge was deep. This time their opinions were confabulatory - for example one guy said 'your phone does not have the antennae to work with sprint' EVEN THOUGH I reminded him I walked into the store with a working iPhone using sprint. Another one said that sprint jailbroke my phone and that's how they got it to work. WRONG. Another one said that the phone was 'soft unlocked' and relocked itself after the DFU restore. I felt sad for Apple because I know that's not what they were about.
Then I suggested they replace my phone, the manager lady's eyes bugged out, her voice became stern and said ' There is no scenario where you walk out of here with a different phone'. This is the first time they wasted 5 hours of my time and actually caused me to have more trouble than what I walked in with.
Galaxy here I come. or maybe pixel2
Apparently not a very high bar.
م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future.
An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...
BOM or endianness don't seem to be relevant to this bug.
Don't get me wrong, I know fonts can be malicious, even... but given the history[0] here, with these unicode[1] issues... I think it's pretty safe to say the issue is not with a specific font, per-se, but with CoreText and unicode parsing. For example, when I ran OSX on my old macbook pro, I used open source fonts on my terminal, and this unicode parsing bug still happened.
In 2015 an Apple spokesperson had this to say: "We are aware of an iMessage issue caused by a specific series of unicode characters and we will make a fix available in a software update."
> BOM or endianness don't seem to be relevant to this bug.
Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem:
> I don’t really have one guess as to what’s going on here – I’d love to see what people think – but my current guess is that the “affinity” of the virama to the left instead of the right confuses the algorithm that handles ZWNJs after viramas into thinking the ZWNJ applies to the virama (it doesn’t, there’s a consonant in between), and this leads to some numbers not matching up and causing a buffer overflow or something.
This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that...
> Yes, I could attach a debugger to the crashing process and investigate that instead, but that’s no fun
Nah, you should do that... and you'll likely see that it's CoreText being the same old piece of shit as usual. If it was a font problem, then loading that font on a different system that uses a different rendering engine should reproduce the same problem. It doesn't, I tried that years ago.
Sure, I would posit, that potentially, this is a whole different bug, but... given the history, and the repeated failed attempts to fix this entire class of issues... it's safe to say that IOS and OSX do not handle unicode very well.
[0] https://www.theregister.co.uk/2013/09/04/unicode_of_death_cr...
[1] https://www.theregister.co.uk/2015/05/27/text_message_unicod...
You seem to think I'm blaming it on the font. I'm not. I'm blaming it on the font stack (CoreText)
I avoided using the term "unicode" to refer to a bug in the font stack because font stack bugs don't always have to be specific to unicode.
You keep saying "unicode parsing"; that's a meaningless term in this context.
> Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem:
That doesn't have to do with endianness or BOM, that's a totally different kind of ordering. It's ordering of the code points, not the code units.
> This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that...
It's a dissection of the string. Not a full debugging of the issue.
Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.
Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?
However I'm not aware of any such things in Telugu aside from explicit virama-showing which rarely exists in input methods (and doesn't end up with zwnj in the position shown here, but that could have happened after editing).
https://en.wiktionary.org/wiki/%E0%B0%9C%E0%B1%8D%E0%B0%9E%E...
Allows for 17 planes, but only small portion of those are actually used. According to Wikipedia[1], currently Unicode has 148944 codepoints + 128k private use ones (which might, or might not make sense to include in unit tests). So your time estimate is off by mere 5 orders of magnitude.
Doesn't really change the result, imho.
text display code is not so simple as it seems, and don't fall into the trap of "how hard could it really be"
East Asian ideographs bring up interesting questions about what constitutes a character, with Unicode "solving" the problem by saying "every distinct rendering is a distinct character," necessitating somewhere in the region of 80,000 characters or so once all of them get added. Even more difficult are scripts like Korean Hangul or Egyptian and Mayan glyphs, which are composed from a relatively small set of independent units but laid out in blocks and sub-blocks themselves composed in linear text. Unicode has both precomposed Hangul characters and the individual Jamo radicals, but it has currently punted on being able to accurately Egyptian or Mayan text in the first place (although they do appear to be revisiting that decision).
Scripts like Arabic differ from Latin in that glyphs changing shape according to their surrounding context is the norm rather than the exception. However, in Arabic, you largely break this down into an initial/medial/final form, with some characters inducing a shift from medial back to initial. Indic scripts go far beyond this by needing to treat entire consonant clusters as single rendered glyphs.
The end result is that it's very easy to find that invariants that one expects to exist if one is used to Latin text to be violated in other languages. Some concepts of text metrics might not even exist in the first place. As speculated elsewhere, it's probable that the text is crashing because someone insufficiently versed in scripts is asserting an invariant that doesn't actually exist. It does not appear to be a rendering issue, but rather a slightly higher level operation on top of that instead.
Finding these sorts of issues pretty much requires extensive fuzzing with known problematic scenarios. The fault is caused by "I didn't know this exists" which is both a very reasonable situation (few UI experts are well-versed in the complexities of foreign scripts) and very hard to solve from a process perspective.
FWIW, Arabic does this too, just that most default Arabic fonts don't. https://www.google.com/get/noto/#nastaliq-aran contains a bunch of specialized ligatures (and is overall a very complicated font).
I always say: There's a reason a lot of the folks working on font shaping are Persian/Arabic speakers :)
That said, there's a reason that I want to learn Arabic.
But for example خ/ح/چ at the end of a word often form cool ligatures, and the dots on ب/پ/ت often go to interesting places, and سے forms a cool ligature where it uses the other form of the bari yeh and the س forms little teeth marks up top. (Some of these are Urdu-specific; I can read regular Arabic but I have more experience trying to read Urdu calligraphy)
Arabic calligraphy can be pretty involved.
In Bengali and Oriya specifically, a ZWNJ can be used to force a different vowel form when used before a vowel (e.g. রু vs রু), however this bug seems to apply to vowels for which there is only one form
This seems to say that the ZWNJ has a meaning before vowels that have different forms, but the crash happens with vowels that only have one form, where the ZWNJ has no effect. Maybe I am misreading?
I'm saying that this crash _also_ applies to vowels with one form.
রু was the original Bengali crash, and that has two forms. I'm saying it's less likely to be related to the zwnj-vowel interaction because it also occurs for vowels where such interaction doesn't exist.
For example, you have a wrapper that only lets you iterate forwards. Once you hit languages where you need to "move back", you have to explicitly code in the functioanality. And in theory you can do it in a safe way.
Sure, some might argue you'll end back at arrays. But I believe that if you encode the traversals in a specific way, you'll at least end up at bounds-safe arrays.
For example, if you use an array as a queue, but some other part of the code doesn't, you're gonna have problems. But if you wrap your thing as a queue, no other part of the code will be able to pierce the veil.
Though I don't know how how well C abstractions let you do this.
OpenType basically requires you to operate on a glyph buffer.
What you can do, is try and make all glyph buffer operations throw an error that makes you hit a fallback case.
While working at Apple in Software QA, both of these cases in part inspired me to create the Big List of Naughty Strings years ago: https://github.com/minimaxir/big-list-of-naughty-strings
Thats a great explanation (though quite a bit of it was too much for me to understand).
The only correlation I can think of, is that the Kannada script keyboard and full support for the script rendering in the UI didn't exist in iOS until iOS 11.0. So it is crashing for Indic scripts that were supported pre iOS 11. Probably Apple tweaked something in iOS 11 for the languages already supported and broke something.
I tried replicating it on High Sierra out of curiosity (single text file with the broken character, opened in TextEdit to cause a crash) and noticed that the crash doesn't happen in consistently the same way each time (I got 8 crashes in a calloc call in CoreText's OTL::CoverageBitmap::Reset, 4 crashes in a malloc in Foundation with no CoreText in the stack trace, plus 10 KERN_INVALID_ADDRESS errors from various places, as well as one time it opened successfully with no immediate problems). Seems like a memory corruption error somewhere earlier, but I'm not sure how to figure out where–I tried rooting around in CoreText with Hopper for a bit but didn't get very far.
(it's just easier to trigger in text boxes because you paste and that's it)