Swift UTF-8 String | Hacker News Reader

Swift UTF-8 String

swift.org

123 pointsgok95 comments

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language.

Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?

Objective-C strings are UTF-16, so it’s likely this was the case. The String ABI has diverged since then, but there is a need to finalize those changes now due to ABI stability.

And worth noting, Objective-C’s strings (actually Foundation’s NSString) is UTF-16 because at the time it was implemented UTF-8 had become as universal as it is now. UTF-16 still seemed like a possible option. And changing the existing strings from 16-8 would have been a huge undertaking - Swift adds an interoperability layer and more flexible compiler/language that makes the transition much easier, not to mention a whole new standard library that has been designed to make this transition work well.

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness.

Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.

They are still partially right, and I'd go so far as to say were totally right if efficiency wasn't a concern. From the programmers point of view utf-32 is still by far the easiest memory representation to use.

Back in the day, when the Unicode people were pushing it as an alternative to whatever the ISO standard is that solved the same problem, they argued it was better than the ISO standard because the was only one encoding, UCS2 whereas ISO defined 8, 16 and 32 bit encodings. They said they could get away with that because 16 bits was enough for all modern languages (read: languages in use). Windows, Java and Javascript among others drank the kool-aid.

But they were wrong, 16 bits was not enough. USC2 morphed into UTF-16, which is actually worse than UTF-8 because if has the byte ordering problem.

To make matters worse, Unicode decided to abandon grapheme == code point. (This means there is ä and ä. They look identical, but one is '\xe4' and the other is a combining diaeresis, '\61\u0308', and the latter is now preferred.) I don't know why that did that as it makes string handling fragile, but I notice the encoding chosen for UTF-16 drastically reduced the number of code points available, so I wonder if they were worried about running out. The end result means that makes utf-32 actually harder to parse than utf-8 (which is quite an achievement), so you may as well use utf-8 for everything.

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them).

Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT.

The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is.

IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.

It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.

When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.

Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.

To be clear, the Windows filesystem isn't even UTF-16. It's UCS-2, which is simply a sequence of 16 bit values. The difference is that not all sequences are valid UTF-16 (see surrogate pairs).

Thus you have to be very careful when handling filesystem paths.

The problem is, the rest of the world decided to settle on UTF-8. We have some normalization disagreements but it's UTF-8 anyway.

Now we have surrogate pairs and the IVS, there is no advantage of using UTF-16 at all.

I will not surprise the Microsoft someday convert everything including kernel to filesystem to UTF-8 someday... or more likely they lost the market share.

Is anyone using Swift outside of Apple specific stuff professionally? When they first announced swift it felt like it might catch on as general purpose gced language. The language itself is quite beautiful in a lot of ways (although being OO was obviously an awful step backwards). But it feels like it's still just the crazy language Apple uses, just like Objective-C was.

Jeremy Howard and Chris Lattner are starting to push Swift for Tensorflow:

- https://www.github.com/tensorflow/swift

- https://www.fast.ai/2019/03/06/fastai-swift/

They must push it really well for it to ever be considered an alternative to Julia, ML.NET, Python/C++, Deeplearning4j on Windows.

Google are beginning to use it for Tensorflow: https://www.youtube.com/watch?v=s65BigoMV_I

There's a course currently running in University of San Francisco run by Fast.ai which is using Swift-Tensorflow taught by Chris Lattner for two lessons: https://www.usfca.edu/data-institute/certificates/deep-learn...

Not exactly what you asked, but pertinent.

I don't expect that will do very well. ML is already crowded, and Python is already solidly at the top. Having worked in ML myself, no one would switch if it meant dealing with a real statically typed language they could hack away at easily in a repl or jupyter notebooks, or access mypy, numpy, scikit, etc. ML is an ecosystem of tools that are all in Python currently.

Just a side-note. Swift comes with an repl and has support in Jupyter and Google Colab for whatever that's worth. As you mentioned the numerical computing ecosystem is just not anywhere close to Python or even Julia. There is a silver lining in that Swift has pretty decent Python interoperability.

In Swift you can just write

import Python

var np = Python.import(“numpy”)

And can then use numpy as you would normally. Same goes for plotting.

Unless you consider reference counting to be a form of GC¹, Swift is not a GC'ed language.

¹It technically is, but most people don't mean reference counting when they say GC.

It doesn't matter what people think, rather CS definition.

Software Engineering is not about what people think, rather what is technically correct.

Well.. what is technically correct depends on a definition. And there are customary/practical definitions too. And sometimes definitions are tainted by context, like in "is this gc or ref counting language".

Which should be "is this tracing GC or ref counting language".

Naturally when in some juriditions one is allowed to call themselves engineers after a 6 months bootcamp we land in such customary/practical definitions.

Even wikipedia admits that most people mean tracing GC when they say GC.

What exactly are you trying to accomplish here?

Advocating learning CS properly.

It certainly does matter because the question is what did the grandparent mean.

Some people want to improve their skills, others don't.

We've been eye-ing it for a while at remind.com. Our hangup has been that we're heavily aws focused, but https://github.com/amzn/smoke-aws is looking promising.

We're interested because it's one of a handful of statically typed functional programming languages. Haskell/Ocaml are great as ML type languages, but it's a big paradigm shift for a team to jump into. Rust still has a pretty decent learning curve. Swift seems to have landed nicely in the "anyone can pick it up" territory. Kotlin is also in that territory, but the JVM is a beast.

For now, we're heavily focused on Typescript as the statically typed functional language, and we leverage types pretty aggressively (conditional types, discriminant unions, etc). Our iOS devs bring a lot of the Swift style over to our backend Typescript systems and from what I've experienced I think Swift absolutely will have a place in backend development.

Swift is a nice language but I think the reliance on ref counting is going to limit its adoption. You have to work a lot harder to avoid reference cycles in Swift than you do in a garbage collected language. Unless GC latency is a deal breaker for your problem domain it’s not worth the extra effort.

My money is on Rust for really low level stuff and Kotlin for anything where JVM overhead is acceptable.

It really depends on what you're doing. A lot of the time, neither garbage collection pauses nor reference cycles are an issue; and when they are it's typical that one or the other will solve the issue.

If GC pauses aren't an issue for your app then you're better off with a full-blown GC because it's a lot easier to code for. Reference counting, as used in Swift, forces you to think a lot about cases where you might be creating reference cycles and memory leaks. GCs can figure this out for you and take that burden off the programmer.

Yes, but reference counting doesn't have unpredictable pauses, which can be important in certain situations. That's what I was trying to say above.

One of the major advantages of reference counting is that it's easier to integrate with other languages. For example you can manipulate Swift objects from C via CoreFoundation, which then allows them to be used from C++, Rust, whatever.

GC'd languages have some support (JNI, etc) but they require handles, pinning, write barriers, etc. which ends up being more complicated and less flexible.

Right, which is one of the reasons it makes sense for Apple to use it, since they have a lot of lower level C API to interact with.

For more general purpose backend work this is not usually a major consideration though, which is one of the reasons I predict most backend work will continue to be done in a GC language.

What do you see as the advantage of Rust or ref counting?

I've just read the Rust docs and not written any code so I'm probably the wrong person to ask but Rust's ownership rules seem to make it possible to eliminate a lot of dynamic allocations in the first place.

Ref counting can beat GC if you're in a domain where GC pauses can be an issue. Things like games, for example. Although I understand state-of-the-art GCs can also be fast enough to use for these kinds of applications in some cases now. Ref counting can also be more memory efficient than GCs.

Ref counting can also be slower than tracing GC when getting rid of complex data structures, heavy multi-threaded code and can lead to stack overflows if destructors are incorrectly written.

> You have to work a lot harder to avoid reference cycles in Swift

It depends on how you use it. As I understand it, if you use enums everywhere it all sort of works out. If you start mixing in structs, there are certainly a lot of minefields.

I would assume good debug-tooling should make spotting such cycles quite easy, why is this even considered a problem?

It’s not easy and iOS developers spend a lot of time explicitly breaking ref cycles in code and tracking down resource leaks. I’ve seen a lot of code from experienced iOS devs that gets it wrong.

What kinds of mistakes do you see? Forgetting to use capture lists entirely? Using them improperly? Not making delegates weak properties?

I don't really consider a language to be functional without a concise application and composition operator (also automatic currying and partial application). How can you be functional if you don't have the basic operators for combining and applying functions?

It looks like Swift has automatic currying though, so that's pretty close.

It doesn't have automatic currying. What code/docs are you getting that impression from?

Swift used to have a special syntax for defining automatically curried functions, but it was removed in Swift 3: https://github.com/apple/swift-evolution/blob/master/proposa...

Indeed, and the compiler still had the relicts of that functionality at least until Swift 4. I'm wondering how they got the impression Swift still has currying, given it sounded like a recent "discovery".

If you have automatic partial application you lose the ability to have multi-arity functions. Functional programming is about functions, not operators. In Clojure you can compose and partially apply functions with the normal functions "comp" and "partial".

https://github.com/clojure/clojure/blob/clojure-1.9.0/src/cl...

Automatic currying and partial application are really just nice to haves though. You can manually do both as long as the language supports higher order functions.

The JVM is indeed a beast in available tooling, libraries and production monitoring.

Kotlin has an LLVM backend these days and can compile to native binaries. It's more Swift-like, then.

Sibling comments mention Chris Lattner. For readers that might not know him, he's the main inventor of Swift. Him pushing Swift should not necessarily be taken as a sign of general community enthusiasm.

I would love to see more Swift on the backend. It's surprisingly hard to find a language that's concise, expressive, statically typed, and easily accessible to beginners. Swift checks all these boxes.

That being said, it's much easier to succeed on the front-end when the only competitor is Objective-C (see: JavaScript), than on the backend when the competitors are Go, Rust, Elixir, Python, Node, C++, Java and more, all with large ecosystems already in place.

Swift may succeed in a niche, and "Swift for TensorFlow" appears to be an attempt at that, but I don't think it will become ubiquitous (call it traditional HN negativity).

He started a podcast last year to promote Swift outside Apple platforms, it lasted three episodes.

Which podcast are you talking about?

This one https://www.swiftcommunitypodcast.org/

As far as I can tell, that podcast was started earlier this year, and is still active?

Right, I was wrong about the start date, on my mind it was started in December.

As for being active, no signs of life since 08th February, more than one month without updates.

There's still stuff going on on the GitHub repository: https://github.com/SwiftCommunityPodcast/podcast

So in one and half months they haven't found time nor a volunteer willing to do another podcast?

My favourite podcasts have weekly and bi-weekly updates.

Yeah, he can't even release a podcast episode whenever you want him to. Does he not know he won't make it to your list of favorite podcasts? Why does he even wake up? Lattner is such a loser.

If he cares about actually grooming a community outside Apple platforms, maybe.

Swift is a great language.

The main problem for me is that it's still very Apple centric. Other platforms are still very second class.

For Linux, there are only Ubuntu packages. Even on Arch which is usually front of the pack with these things, there are only brittle third party (AUR) packages.

Windows support is even worse.

For me, it's the best language for game development that I've seen ever since I wanted to make my own games as a kid.

They'll need to do an awful lot more on their IDE for that to happen.

That GUI drag'n'drop onto code... ugh.

That's just a convenience thing. I've worked in Xcode for nearly 10 years and I never use that feature. Good feature for teaching though. But if you hate it, just don't use it.

Great work by the Swift team on stabilizing this (especially around UTF-8) for Swift 5. One question: how do Objective-C's tagged NSString pointers fit into this? Are these imported as "opaque" strings? I see that _SmallString seems to store its characters inline; are there any plans to do tagged pointers in Swift as well?

It would have to be opaque as there is no backing allocation of any kind so the string can’t provide a pointer to the contiguous memory. (Plus tagged strings use an encoding scheme that makes the incompatible with UTF-8 anyway) It’s possible that a smallstring version of the tagged pointer is created in some situations.

SmallString actually was just a tagged cocoa string at one point but that was removed in favor of more powerful implementations. The latest implementation allocates no memory on the heap either but can store more characters, so using a tagged pointer would be a step back.

”An opaque string is capable of handling all string operations through resilient function calls. […] Currently, these are used for lazily-bridged NSStrings that do not provide access to contiguous UTF-8 (i.e. ASCII) bytes in memory.”

⇒ it seems tagged NSString pointers indeed map to opaque strings.

Because strings are structs in Swift, I don’t expect them to implement tagged pointers in Swift. Alternatively, you can see the SmallString struct as a tagged pointer that’s twice as large as a regular pointer.

Seems like the right move. Julia made a similar transition in 1.0 for similar reasons and it has turned out very well.

Wise move, in my opinion. I hope Python will do that too, eventually.

Python implemented multi-representation strings in 3.3. It picks the best encoding from ASCII, UTF8, UCS2, or UCS4. UTF8 is the preferred encoding for strings exposed to the C API.

https://www.python.org/dev/peps/pep-0393/

CPython provides but does not use UTF8 internally, it's a cache for PyUnicode_AsUTF8 (formerly _PyUnicode_AsString), which exists to convert Python strings to char* in order to interoperate with C libraries. It is not the canonical representation of the string.

pypy, on the other hand, made this change in the latest release (7.1.0): https://twitter.com/pypyproject/status/1095971192513708032

The "kind" field is one of [uninitialized, Latin-1, UCS-2, UCS-4]. I'm not sure how it would pick UTF-8.

(In memory usage, the Python way is strictly worse: adding one 4-byte character to an ASCII string forces the entire string to be upgraded to UCS-4. The advantage of the Python way is O(1) access to codepoints, but that's an operation that rarely comes up in practice, when dealing with Unicode strings.)

I'm not sure how that would be compatible with Python, either. Either you're giving up string[index], or you're causing this one subscript operation to take O(n) time, or you need a new index type just for strings (which is what Swift does). None of these seem very 'Pythonic'.

EDIT: Apparently recent versions of pypy solve this by computing their own index into your string, which seems like it would be terrible for memory usage, but I look forward to their blog post about it.

What do you hope Python will do?

UTF-16 allows to quickly index a char in a string covering the most languages that way.

UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII.

Now, how can a char be indexed then? Only by slow iteration from the beginning of a string?

UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the algorithmic point of view.

Or maybe there is a catch that makes UTF-8 strings well-suited for string processing as well?

> UTF-16 allows to quickly index a char in a string covering the most languages that way.

Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either.

There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).

Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages".

Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16.

P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.

All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index.

That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.

You definitely can't claim "most languages" today with the commonality of emoji and emoji-based sub-languages/communication forms/media. Surrogate characters are no longer rare in UTF-16 in average usage on the internet and that's impacting everything.

Even if you are restricting "most languages" to mean "most programming languages", there's an increasing rise of emoji in comments alone for languages that support UTF-8 and some languages are picking up increasing emoji usage in places like identifier names.

Platforms that treat UTF-16 like UCS-2 and allow raw char indexing rather than codepoint-oriented traversal are wrong in 2019. (It's been wrong since the very definition of UTF-16, such as RFC 2781 in 2000.) UTF-16 is a disappointing hack because it allowed UCS-2 platforms the luxury of pretending that the UCS-2 era never ended. UTF-8 at least leaves you very aware that the UCS-2 emperor has no clothes.

If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent.

It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.

> give us strings that make sense

This is really, really hard.

I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable.

Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)

Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.

The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.

To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.

Ah, so your solution is the same old canard: just ignore the complexity and call it pragmatic.

No. How about a string that returns a Unicode character length?

That's not 'ignoring complexity' and it's pragmatic.

That would help in English as it's really hard today just to deal emojis.

The point I tried to make: I suspect if you dug really deep into what is a character or glyph you might find "character length" is an imprecise and messy notion, with differing performance costs associated with differing definitions which may or may not apply to you.

So even if you pick one definition and decide to live with it always... You'll still have to know what tradeoffs you (or an API designer or implementer) made.

First - a combined unicode length is definitely universally useful. Second - a grapheme length is universally useful. Third - I didn't say 'pick one definition' - you did. A simple function which took as a parameter some kind of linguistic construct would be immensely useful in cases of choice/ambiguity.

'It's complicated' is the utterly the wrong answer as to why we 'can't have better strings'.

Strings in almost every programming language were designed before Emjojis and true internationalization. If we designed them today, they would have support for dealing with most of the problems we face.

And FYI - there are not infinite corner cases. They are limited, and we can develop constructs for dealing with all of them.

There are myriad problems today that 99% of developers couldn't be bothered to think about, but can be a source of bugs. eg. I'm going to guess every time you call tolower() have you haven't thought "I wonder what this does to Turkish I", but it's still there. An API won't fix us from having to think, and having ambiguities occasionally become bugs. (The fact that you've acknowledged the need for multiple string length functions -- I agree with it, but it also means you'll sometimes use the wrong one.)

As I said in another comment, I still applaud efforts to improve the status quo.

> No. How about a string that returns a Unicode character length?

That doesn't mean anything. Do you mean a code unit, a codepoint, a grapheme cluster, if the latter is it legacy, extended or tailored, how do you handle the locale dependency, … All of them can be useful in some context, all of them are length. And if anything, the truly useful one is the first one (because it tells you how much room you need to store something).

> That's not 'ignoring complexity' and it's pragmatic.

It's true that it's not ignoring complexity, because you're ignorant of it to start with. Which isn't really different.

> That would help in English as it's really hard today just to deal emojis.

It… really isn't.

Totally wrong. Your implied solution is to just have every single developer, everywhere, to be a linguistics expert and to handle the myriad of international cases for basically everything.

This is totally the wrong approach.

Language is shifty, but there are definitely a series of tools that should be integrated into every 'String' that can facilitate most problems.

"That doesn't mean anything" This is false. It absolutely means something, you can't brush it aside by inventing a separate series of specific measurements. Should we have those other measurements - yes, in many cases, we probably should have them as well -> that's the whole point.

Totally wrong. Your implied solution is to just have every single developer, everywhere, to be a linguistics expert and to handle the myriad of international cases for basically everything.

This is totally the wrong approach.

Language is complex and sometimes ambiguous, but there are definitely a series of tools that should be integrated into every 'String' that can facilitate most problems.

Not every company is trying to satisfy Hindi speakers.

"That doesn't mean anything" This is false. It absolutely means something, you can't brush it aside by inventing a separate series of related measurements. Should we have those other measurements? - yes, in many cases, we probably should have them as well - which the whole point. We desperately need a series of common tools built into String, some of them which provide more nuance.

"It's true that it's ignoring complexity, because you're ignorant to start with"

Well, since I have a background in NLP, I've developed writing recognition and word prediction algorithms, as well as language models for dozens of languages for products used by millions of people every day ... maybe I'm not as ignorant as your snide remark suggests?

"> That would help in English as it's really hard today just to deal emojis.

It… really isn't."

Uh, yes, it absolutely would help, and it's painfully obvious. In most languages, which cover the exceeding majority of markets for most apps made, simply knowing the character length (i.e. 'visible character' as to avoid confusion over messy nomenclature here) is very useful. In Latin languages, after Unicode normalization and other kinds of cleaning, Emojis present a very common case for confusion.

There are very small set of tools (or rather, a slightly better, more standardized String approach) which would solve most of the problems for most companies developing for most markets. There are even a few extra tools which would solve, pragmatically all of them.

Tell that to the Python community where switch to UTF8 was/is a huge battle!

Swift is meant to be retro compatible with Obj-C and all. As someone pointed out windows in the meantime have basically no announced battle plan to transition out of UTF16.

I this game latecomer were rewarded because they started with UTF8 or switched from ASCI. But innovators actually tried UTF16 in the mean time an now have to do the job twice in a retro-compatible manner.

In what language do you want them to make sense? A Hindi string would be very different to an Arabic strig, or a japanese string, or an emoji string.

Reply on news.ycombinator.com