Hacker News Reader

Comment by AndyKelley | Hacker News Reader

A lot of people reach for string handling when the actual correct thing to do is intentionally avoid string handling, and only handle strings as opaque encoded UTF-8 bytes, that cannot be reasoned about in terms of human language.

I would even argue that having string handling in a standard library (or language) has the potential to cause a net increase in bugs, because of people thinking they are handling strings when actually they are just screwing around with codepoints. Go's string handling is completely broken, for example. As a result of strings in the language, Go programs tend to be more broken than C programs in terms of string handling.

View on HN

> A lot of people reach for string handling when the actual correct thing to do is intentionally avoid string handling, and only handle strings as opaque encoded UTF-8 bytes, that cannot be reasoned about in terms of human language.

I have lost count of how many times I have wanted to find substrings, transform cases, catenate strings, find patterns, substitute patterns. I'd be happy to do that in a language that didn't permit me to index the underlyinc characters or the bytes. Keep 'em opaque, sure. But I think it would be a mistake for a language not to have an idiom with a favorite library to perform these operations on encoded text. If resolving library dependencies is easy enough, then it doesn't need to be "standard" but it should be "the defacto standard." And if it turns out the defacto standard stagnates and doesn't keep up with the needs of developers, a new one can come take its place.

Maybe Java or Python are better choices for your problems. The point of a systems programming language is to allow implementation of specific solutions to specific problems. There are many, many ways to do the things you mention here (and that starts with the strings' storage format and allocation strategy) , so it's the right choice to not include _anything_ like that in the core language.

IMO, Swift is a language that gets it right, by exposing an interface which feels to very closely match how "humans" understand text — the default String interface is a collection of grapheme clusters, while `.utf8View`, `.utf16View`, and `.unicodeScalarView` are optional views which expose data explicitly encoded as needed.

Swift is pretty pedantically strict about Unicode correctness, and avoids some pitfalls which lead to incorrect handling (e.g., integer-based indexing). This means that the traditional way many developers might be used to interacting with strings is cumbersome and annoying — but once you get past the initial hurdle*, most operations are actually (1) really easy, especially when expressed in terms of generic Sequence/Collection operations, and (2) much more difficult to get wrong.

*I think the largest part of that hurdle is overcoming what you may have gotten used to from other languages, i.e., treating strings as an array of "characters", for some language/library definition of "character" (whether bytes, code points, etc.). It's relatively rare that you actually care about indexing into an arbitrary spot in a string: instead, combinations of slicing operations (including `prefix(_:)`, `dropFirst(_:)`, `take(while:)`, etc.) and generic Collection operations will get you what you want. Things like `.reversed()`, `.sorted(), and `.shuffled()` all work trivially correctly too (since you're not operating on a bag of bytes), and it's exceedingly rare that user input will confound the operations you might need to perform. (Exception: operations like case folding and collection, which are locale-specific, need special handling through a framework like Foundation.)

To be clear: not everything is sunshine and roses, but an amazing amount of functionality "falls out" of basic protocol conformances on String, and its exposure as a Collection of grapheme clusters.

Given a specific string manipulation task, I'd be happy to provide an example of what it might look like in Swift!

> Given a specific string manipulation task, I'd be happy to provide an example of what it might look like in Swift!

How would you safely get the nth index of a string, clamped to the valid indexes? So if the nth index is out-of-bounds you get the first/last index instead?

Or you get an Optional and you have to check if there's an error or you go the character within the range? Sounds perfect safe to me...

I don't think Swift has a clamping function in the standard library, unfortunately, so we'll have to roll one ourselves. One question remains of what should be done for the empty string; I've chosen to return nil for this case.

  extension String {
      func character(atClampedIndex index: Int) -> Character? {
          guard !self.isEmpty else {
              return nil
          }
          let clamped = max(0, min(index, count - 1))
          return self[self.index(startIndex, offsetBy: clamped)]
      }
  }

> Things like `.reversed()`, `.sorted(), and `.shuffled()` all work trivially correctly too (since you're not operating on a bag of bytes), and it's exceedingly rare that user input will confound the operations you might need to perform.

One thing to note: aside from programming interviews, these operations are fairly rare. And that's a good thing, because none of these produce results that are very intuitive, because they are not very well defined on strings in general (I don't fault Swift for this, but it's just a general problem with text). Using any of these to create a new String may cause entirely new characters to show up, or the length of the text to change. So Swift actually doesn't expose these as "string" operations, but operations on the characters themselves; in each case returning a new collection of characters that is not a String. Now, you can reconstitute them into a String pretty easily, but you should keep the this in mind when doing so.

I use these operations on a daily basis, not in interviews. If you're ingesting third party data somehow (even if it's structured like JSON or XML) there are going to be cases where you need to hand roll a simple parser.

I’m genuinely very interested in the kind of data you’re dealing with. It’s common to need to reorder a list of strings, but I’ve personally never had any use case that requires me to reverse or sort _characters in a string_ to consume any kind of data (some kind of decoding, perhaps, but those tend to work better implemented at the byte array level, not string).

> the actual correct thing to do is intentionally avoid string handling,

That sounds nice and all, but wr have 50+ years of protocols and formats and APIs built up around strings. Unless you're just writing code to run on a small microcontroller, you need to be able to parse and generate strings. So its going to be pretty frustrating not to have good support for them, or to have every codebase use its own libraries and idioms for working with strings.

Protocols and formats absolutely should not require decoding strings. I think you are mistaken. Can you name any well-established protocol or format that does not treat strings as opaque encoded bytes?

Edit: so far these examples have been given:

* HTTP: wrong. the spec does not tell you to decode any strings

* CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode awareness in order to properly read and parse the data or deal with the delimiters.

Additional request: if you attempt to provide a counter-example, please also point to the place in the spec where it tells you to decode a string.

CSV?

HTTP for one. It does something like assume ASCII-like until it encounters odd looking bytes and/or a meta encoding tag.

I think HTML tags like 'meta' are merely payload to HTTP, right? Presumably a markup language doesn't fit Andrew's criteria for "protocol"?

Protobuf string fields are expected to be in UTF-8. Although I'd expect a sane implementation of a protobuf decoder to throw an exception or otherwise indicate an error if it receives a malformed protobuf encoding.

Protobuf decoders are expected to validate UTF-8 strings for syntax="proto3" files, but not syntax="proto2". The behavior diverges mostly for historical reasons.

This is a validation pass only and it doesn't make any meaning of the code points, except to validate that none of them are surrogate code points (disallowed in UTF-8).

JSON[1], for instance, specifies:

1. "JSON syntax describes a sequence of Unicode code points. JSON also depends on Unicode in the hex numbers used in the \u escapement notation."

2. "A JSON text is a sequence of tokens formed from Unicode code points that conforms to the JSON value grammar."

3. "A string is a sequence of Unicode code points wrapped with quotation marks (U+0022). All code points may be placed within the quotation marks except for the code points that must be escaped: quotation mark (U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F. There are two-character escape sequence representations of some characters."

4. "Any code point may be represented as a hexadecimal escape sequence. The meaning of such a hexadecimal number is determined by ISO/IEC 10646. If the code point is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the code point."

5. "Note that the JSON grammar permits code points for which Unicode does not currently provide character assignments."

JSON does require Unicode awareness, both for general parsing, and for correctly interpreting strings. Backslashes are allowed for special escape characters, which means that you must be aware of the format (and the encoding of the text) in order to be able to decode.

Note that JSON also doesn't specify a required encoding, only Unicode correctness, so a parser may need to be able to handle multiple Unicode encodings and differentiate between them.

The spec doesn't specify what to do with code points which are not understood as Unicode (given especially the allowance for unassigned characters), but explicitly-invalid Unicode should be rejected.

[1] https://www.ecma-international.org/wp-content/uploads/ECMA-4...

Parsing this out of utf-8 encoding requires no knowledge of unicode or even utf-8. All of the relevant characters (reverse solidus, quotation mark, and control characters) are single byte characters in the ascii subset. These characters cannot be found inside multi-byte characters in utf-8 due to the design of the encoding. Converting the unicode character escape codes to utf-8 would require knowledge of utf-8 encoding, but this unescaping is not a feature that would be provided by the language regardless.

> Parsing this out of utf-8 encoding requires no knowledge of unicode or even utf-8.

If you have valid UTF-8 already, then yes, the task is a lot easier. But depending on the level at which you're parsing, this might not be the case — i.e., if you're writing a JSON parser from the ground up, you do need to know what UTF-8 and Unicode are, and will need to validate the input data.

> Converting the unicode character escape codes to utf-8 would require knowledge of utf-8 encoding

Agreed. Even if you're not working at the "array-of-bytes" level, you will need to be able to parse and translate "\u..."-style strings into the appropriate output character encoding.

> but this unescaping is not a feature that would be provided by the language regardless.

I'm not sure we're talking about this being handled at the language level. This translation is something that would likely be offered at the parser level (working with the features offered by the standard library), but the parser does need to know about it — and does need to be able to work with strings at a granular level to be able to parse it out. By definition, it cannot leave the input data as an undecoded bag of bytes.

Note, too, that the JSON spec does not specifically require UTF-8. UTF-16 is a completely valid encoding for JSON (though much less common than UTF-8), in which case none of these characters are an ASCII subset, and greater awareness is needed to be able to handle this.

> it cannot leave the input data as an undecoded bag of bytes

But all it's doing here is taking a hex string (which is entirely ASCII) and converting it into the respective hex representation. Since ASCII translates unambiguously to bytes, it doesn't really matter if `str[0]` is operating on a byte stream, codepoint stream or grapheme stream, because in utf8, they're all the same thing as long as we're within the ASCII range.

Where things get hairy is stuff like `str.reverse()` over arbitrary strings that may or may not be in ASCII. This repo[0] talks about some of the challenges associated with conflating characters with either bytes or codepoints. The problem is that programming languages often approach strings from the wrong angle: you can't just tack on handling of multi-byte codepoints on top of ascii handling; you lose O(1) random access and you don't actually model the linguistic domain properly by doing so, because in the first place, humans think of characters not in terms of bytes or codepoints, but in terms of grapheme clusters. Clustering correctness falls deep in the realm of linguistics, and is therefore arguably more suitable to be handled by a library than a programming language.

[0] https://github.com/mathiasbynens/esrever

I agree entirely with your second paragraph, but regarding this:

> hex string (which is entirely ASCII)

My point is that JSON doesn't need to be UTF-8 or a superset of ASCII to be valid. It can be any representation of Unicode, including UTF-16, UTF-32, GB 18030, etc.; so long as the text is is comprised of Unicode code points in some Unicode transformation format, the JSON is valid.

As I said in the parent comment: if you are working within UTF-8 exclusively, and can assume valid UTF-8, then great! But this isn't necessarily true, and in some cases, you will still need to care about the encoding.

(Either way, this starts straying slightly from the more general discussion at hand: regardless of the encoding of the string, you will still need an ergonomic way of interacting with the contents of the data in order to meaningfully parse the contents — even past the hurdle of decoding from arbitrary bytes, you still need to manipulate the data reasonably. In some cases, this means working with a buffer of bytes; in others, it makes sense to manipulate the data as a string... In which case, you may run into some of the string manipulation ergonomic considerations being discussed around these comments.)

> JSON doesn't need to be UTF-8 or a superset of ASCII to be valid. It can be any representation of Unicode, including UTF-16, UTF-32, GB 18030, etc

Sure, it can also be gzipped, encrypted, etc but that goes back to the point that there's nothing inherently special about JSON as it relates to encoding to a byte stream. All there is to it is that somewhere in a program there's an encode/decode contract to extract meaning out of the byte stream, and in a protocol one most likely only looks at byte streams as sequences of bytes (because performance-wise, it doesn't make sense to look at payload size in terms of number of codepoints/graphemes at a protocol level)

By "decode a string" do you mean that you must apply some sort of Unicode transformation to it? If so, the HFS+ file system, which .dmg files often are, is an example: Unicode NFD normalization must be performed on all filenames because of case insensitivity requirements [1].

[1]: https://eclecticlight.co/2021/05/08/explainer-unicode-normal...

The problem is you've invented your own definition of the terms "string" and "string handling" which doesn't seem to agree with the generally accepted usage of these words. You're fixated on the word "decode" despite that not being the original focus of the discussion, not being present in my comment or the original comment you're replying to.

I could create a standard library called "arrayofbytes" that lets you, for example:

- Search an array of bytes for a smaller array of bytes

- Split up an array of bytes based on an array of delimiter bytes

- Selectively convert an array of bytes in range 41-5A to bytes in the range 61-7A

But wouldn't it be more appropriate to describe these as "string handling" functions using the generally accepted terminology of our profession?

In Zig, those functions are in std.mem (distinct from std.ascii and std.unicode)

Anything with case insensitivity, like domain names and email addresses, require knowledge of strings as strings.

Case sensitivity, ordering, and case changing rules, are reasons not only to have strings, but extensive culture support built into them. String types also let you compare utf16 bytes to utf8 bytes, for example.

To think that strings are opaque bytes is massively naive. It presumes one encoding exists, the input is always valid, in addition to what is said above.

An example of what you're setting yourself up for, is an exploit based around differences in handling invalid utf8.

What???

> the spec does not tell you to decode any strings

Which "spec" is that? My user ask for csv, see csv, upload csv, do SEARCH on CSV, transform them, etc. Even do some scripting on them.

Nowhere, EVER "opaque encoded bytes" is on the spec of the users!

The HTTP specification does tell you to behave differently depending on the presence or absence of specific values for case-insensitive headers. That requires decoding the header name strings.

>That requires decoding the header name strings.

Which should be ascii though no?

Maybe, but it's not "treating strings as opaque encoded bytes" which is what was asked.

ASCII is kinda special in that it's both a way of encoding strings and it has character-to-byte equivalence. You can specify an HTTP header as being any number of characters between `\n\r` characters or any number of bytes between `0x0A 0x0D`, and the parsing code is going to compile to pretty much the same thing regardless of whether you know what `0x0A` means or not.

Right, but to do case-insensitive comparisons of header names you need to decode them as strings, not just byte sequences.

> * CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode awareness in order to properly read and parse the data or deal with the delimiters.

CSV is defined in terms of characters, not bytes, so CSV parsing does require you to be encoding-aware: if your CSV file is encoded as UTF-16, bytewise parsing will destroy the data.

That's not really true at all. The single byte / single character comma separators are all that matters there. As long as you directly acknowledge / exactly replicate whatever blob of data happens to be in between each set of commas (even if it is nonsense garbage text) then you're correctly parsing the CSV.

Well let's try :grinningface:,:grinningface: which is d83d de00 002c d83d de00 in UTF16BE. If you extract first column by naively cutting the blob bytewise before the comma you end up with d83d de00 00 with extra NUL byte, which is a problem. With UTF16LE you'd prepend NUL which is even worse.

I think you somewhat misunderstood the manner I was suggesting you'd be approaching things there, though nothing you've said is incorrect.

Then I’m not sure what you were suggesting. The crux is that CSV operates upon strings, not bytes.

> cutting the blob bytewise

That doesn't make sense. If you're working with utf16, why would you slice bytewise? That's like slicing a zip file bytewise and wondering why it got corrupted.

The whole point of the argument for string support at a library level (rather than assuming some sort of equivalence between stringness and its underlying byte buffer at the language level) is that fixed width bytes fundamentally cannot model human language characters unambiguously because what a "character" is depends on the encoding/decoding contract of the program manipulating the byte buffer.

Assuming equivalence between 0x2C and `,` stems from the ancient history of ASCII, english and usage of C `char` as a mechanism to squeeze performance out of string operations by not properly supporting the full gamut of valid human language characters.

For a low level language that might be used to implement protocols, it totally makes sense that foo.len is length in bytes, because you're pretty much never going to want to know number of grapheme clusters at a protocol level. It doesn't make sense for a language level .len to be length in terms of codepoint count because that assumes encoding, which is fundamentally a business logic level concern.

The entire matter in question is whether CSV is encoding-independent, operating on bytes (we’re addressing AndyKelley’s comment). The answer clearly demonstrated here is: no, CSV is operating upon characters, not bytes, so you need to decode the Unicode first and let the CSV operate on Unicode data, so that it’s splitting on U+002C, rather than 0x2C in the byte stream before Unicode decoding which destroys the data.

The tar / star format requires decoding strings of octal numbers.

https://en.wikipedia.org/wiki/Tar_(computing)#Header

"Stringly typed" programming gave us SQL injections, shellshock, and the log4j debacle. Not to mention probably 99% of processor cycles being wasted on parsing and re-encoding again and again from/to text-based formats that are in no way actually human-readable without the use of specialized tools.

When have you last browsed the web using telnet? It's all plain text, so you should be able to, right? Press Ctrl-U right now and get a taste of how readable it is, and even that is with the benefit of syntax highlighting.

On the machine level, processing binary data is effortless. At most, you have to swap the bytes around to conform to wrong-endian network byte order. Check that the length of the data fits into your buffer. Not a problem with Zig, or any language that doesn't silently ignore integer overflow as C does!

Scripting languages make handling strings seem easy, because that's what they were built for. And of course in most languages there are "mature" libraries for parsing JSON or XML. But all the fundamental complexity is still there. Each layer of abstraction may introduce some bug that can be exploited. Compared to classical buffer overflows, it may take several more steps to gain arbitrary code execution, but every bit of extra code you depend on still increases the attack surface.

Ideally, only user interface code should be dealing with strings at all, but legacy protocols may unfortunately require it too. This should be isolated as much as possible from the rest of any application, and not dictate what features are "first class" in a programming language.

That's right, my suggestion was clearly that we should replace all datatyoes with strings. This comment brought to you by TCL/Tk.

I’m not aware of any significant breakage due to Go’s handling of strings. Where would I read more about this?

Go presents strings as slices of bytes (chars). As long as it's ASCII, it's fine. However, when dealing with multi-byte UTF-8 characters (codepoints), the proper unit is the rune. So, before attempting to measure a string's length, or read its n-th character, one must remember and access the string as a slice of runes.

  s := "naïve"
  
  // bad
  fmt.Println(len(s)) // 6
  fmt.Println(string(s[2])) // Ã
  
  // good
  r := []rune(s)
  fmt.Println(len(r)) // 5
  fmt.Println(string(r[2])) // ï

https://go.dev/play/p/YbMo49wU7vu

Yes, but I think most Go developers are aware of Go's quirks, so I'm wondering what bugs happen despite that awareness.

Yeah strings are ugly and programmatically impure -- they are an extension of human language after all -- but virtually all human facing apps use tons of them for obvious reasons. Same goes for regex.

IMO they are a great example of how Golang is a pragmatic rather than "clever" or "pure" programming language.

By not treating strings specially, you get people screwing around with individual bytes inside codepoints which leads to at least as many bugs.

The UTF-8 encoding is designed so that this is usually not a problem. If you do a search in a utf-8 encoded byte array for an ascii character, for example, you can never get a false positive. Compound UTF-8 characters always have the most significant bit set of each component byte, and ascii characters always have it unset. Additionally, treating the string as an array of unicode codepoints doesn't solve the problem -- now you have people screwing around with individual codepoints inside grapheme clusters :P

> Additionally, treating the string as an array of unicode codepoints

I suggested no such thing.

> individual codepoints inside grapheme clusters

That's less severe than invalid codepoints.

Perhaps the whole thing whichever way it is represented should not be mutable given that there's no way to make it mutable in a sensible way?

I don't think I've ever had a problem with Python3's string handling, which is very robust.

    >>> "ñ"[0]
    'n'
    >>> "ñ"[1]
    '̃'

Huh, which version is that? Python 3.9 on my system:

  >>> "ñ"[0]
  'ñ'
  >>> "ñ"[1]
  IndexError: string index out of range

Which is what I would expect.

Python 3.9.9 (main, Nov 20 2021, 21:30:06) [GCC 11.1.0] on linux

EDIT: I can't seem to replicate this on my Ubuntu system. Strange.

Ah, interesting, I suppose then there is a difference in the way Linux handles strings? Didn't realize that, very unfortunate if true. I am running MacOS.

Actually browser normalized string and it works for me also. Here is original byte sequence:

    >>> b'n\xcc\x83'.decode()
    'ñ'
    >>> b'n\xcc\x83'.decode()[0]
    'n'
    >>> b'n\xcc\x83'.decode()[1]
    '̃'

But I agree, it's rare case when you need to deal with non-normalized data.

this is normalization issue, not version issue

import unicodedata list(unicodedata.normalize('NFD', "ñ")) >> ['n', '̃']

list(unicodedata.normalize('NFC', "ñ")) >> ['ñ']

both are correct, the issue is that unicode allows accented letters to be written as _accented_letter_ or _letter_, _accent_. The idea of "character" in uncicode is not very useful, most of the time you will want graphemes, not codepoints. User-friendliness wise, this is what Python should use (another rant - strings should not have length method, they should have byte_length, codepoint_length and grapheme_length).

As a counterpoint from primarily experience with higher level languages, not being able to distinguish strings from just any blob of bytes is something that’s consistently put me off lower level languages. Sure you can just treat it as a blob of bytes but… you can’t do anything with it with certainty. That seems to be the class of bugs you’re describing? But this isn’t an issue in languages where a string is a distinct type (even if dynamic).

Can you elaborate on why you think Go's string handling is broken?

> Go's string handling is completely broken, for example.

Classic Andy, shitting on other language with no references or examples. Go has some of the best string handling I've used. Seamless byte, rune, string conversion. Simple iterating and slicing. Plus helpful tools like strings.Builder and strconv.AppendInt. while Zig has nothing.

It seems like every language must have some dumb decisions made because the authors don't care/think they know better/are stubborn/etc or because it painted itself into a hole from the start, without which it would be close to perfect for its use cases.

Python has the GIL.

Go itself has a few such cases (usually revolving around "NIH" and misguided simplicity).

Zig has the prejudice about proper string handling.

> Python has the GIL.

You are welcome to show fast single-core python interpreter without GIL. Coreteam will gladly accept your patches.

That's how I understand "painted itself into a hole from the start" from the GP - removing GIL at this point is very hard because virtually all of Python language and code has been created in a world with the GIL.

It's nice to have atomic guarantees of GILed implementation. For me current CPython semantic is a golden middle: you have pretty fast interpreter without race conditions (for huge part of user code). Due to GIL.

I always like to ask for what kind of task one needs GILless python?

CPU bound? If you are using native python code for CPU bound tasks you are already in a bad place. C extensions can release GIL. For example numpy.

What else?

> I always like to ask for what kind of task one needs GILless python?

> CPU bound? If you are using native python code for CPU bound tasks you are already in a bad place.

While I think you may have misread my comment for a value judgement on Python's GIL, I don't see as particularly useful dismissing major potential (multithreaded) performance gains for a slow language just because there's faster languages. Languages that are better in some way - like speed - should stand as a benchmark and a goal, not a reason to give up improvements.

I'm not against faster interpreter also. It's history now, but IMHO super slow GILless python with Java-esk data races in 199x would not gain such broad community and ecosystem.

Anything with a hard or soft real-time constraint (audio, video, industrial automation, etc). You'd be shocked how many algorithmic prototypes are written in numpy then hand converted to C++ for realization.

NumPy releases GIL. I love NumPy myself and created complex interactive realtime synths. It always was more convenient to slap Cython or C code to achieve desired 100x performance instead of getting potentially 10x (4x on my laptop) with multithreading.

Zig certainly has things like strings.Builder and strconv.AppendInt, which aren't really related to language-level encoding-aware string support.

No it doesnt

strings are called []u8, strings.Builder is called std.ArrayList(u8), strconv.AppendInt is called std.fmt.formatInt. Are you trolling?

those dont do the same thing at all. strings.Builder can append a byte, rune, string, or byte slice:

https://godocs.io/strings#Builder

ArrayList cant do that. And strconv.AppendInt can convert a number to byte slice, then it appends to an existing byte slice. formatInt cant do that.

ArrayList(u8) exposes a writer. You can write things to writers in various ways. Probably more than four ways!

"appends to an existing byte slice" is not a completely coherent thing to ask for. If I give you a byte slice, the byte after the end it might belong to something else. If you want to deallocate my byte slice and make a new, replacement byte slice, you'll need an allocator to do that. An ArrayList(u8) has a byte slice, and knows how much of it is unused, and has an allocator so it can make a new larger byte slice if needed, and exposes a writer so that you can call std.format.formatInt to write into the byte slice (or allocate a new byte slice, copy the entire contents, and write into the new one, if appropriate).

> strconv.AppendInt can convert a number to byte slice, then it appends to an existing byte slice. formatInt cant do that.

As we have learned just now, we actually can use formatInt to append a formatted int to an existing byte slice (to the extent that that's a meaningful thing to ask for), by passing it the writer exposed by an ArrayList(u8)!

Is your complaint that you're not aware of any function in Zig that takes a u32 representing a Unicode codepoint and encodes it to UTF-8 and writes it to a writer (which is what "appending a rune" seems to mean)?

Is your complaint that people do not, by convention, allocate new, larger byte slices without an explicit reference to an allocator?

Is your complaint is that Zig does not have a garbage collector?

Can you share a Go program that uses the Go stdlib functions and cannot be trivially ported to use the Zig stdlib functions instead, for some reason other than that Zig programs must decide where the bytes will live, and Go programs need not do that?

> Can you share a Go program that uses the Go stdlib functions and cannot be trivially ported to use the Zig stdlib functions

Heres an easy one:

https://go.dev/play/p/JFIvQYTF3R8

Alright, I guess your objection is 100% about the function that accepts a 32-bit integer and encodes it as UTF-8 then.

Please let me know the ways this does not do the right thing. Thanks.

https://www.godbolt.org/z/Y3z8rhzrq

Im not sure how a 50 line program is better than a 5 line program, but OK. Also it seems you had to write functions to get the same result.

My original comment, was that Andrew has a habit of shitting on other languages without proper references or examples. Nothing you can really say is going to change Andrews behavior, so maybe you should stop, unless you can justify Andrews comments.

Your original program which is claimed not-possible-to-port is 14 LOC. This comparable program is 18 LOC: https://www.godbolt.org/z/xf4Tdr1Ps

Most of the difference comes from the demand that we represent unicode codepoints as integers at some point in the program, which is a nonsensical thing to do, because unicode codepoints don't correspond to anything useful in the actual text being represented.

You seem to have a habit of making false claims about the standard libraries of languages you dislike, and when pressed on the matter ask other people to do your homework. I certainly should stop doing other people's homework.

You seem to have a habit of getting off topic. This thread was never about Zig, it was about Andrew shitting on other languages without justification. I think you can agree that for the basic example I gave, the Go language is easier, more streamline, more comfortable to work with. It allows to get the task done quickly.

So for Andrew to shit on Go string handling with not a single example is rude, and frankly just wrong as I have demonstrated.

Maybe Andrew is referring to this? I'm just guessing but that is the only thing I can think of which could be considered broken. Go mostly just treat strings as bag of bytes and you have to use one of the unicode packages to actually do any significant work with them.

https://play.golang.com/p/Dla3sXciYXC

I also think string handling in Go is pretty sane. But range vs indexing on strings is something you need to be aware of.

I don't see anything wrong in that example. As you said, golang has unicode/utf8, as well as the utf8string package. Is anyone really suggesting that the default slice should be on graphemes?

That's one of relatively few things that language-level utf-8 support could get you over utf-8 support in the standard library.

Reply on news.ycombinator.com