Hacker News Reader

Comment by mjevans | Hacker News Reader

I would _like_ to use Python (2) for that, but it's EOL/effectively EOL, and Python 3 made the _wrong_ call with respect to Unicode support.

Source code is Unicode? Great.

Forcing _everything_ to have an Encoding when the surrounding world isn't an academically clean environment? Nope, this means Unicode Enforcement and Error Handling in what should be simple programs.

I don't mind have __optional__ libraries for handling Encoding conversions, corrections, and errors... but if I'm working with a legacy filesystem, human input, or anything else, the only basic language support I want or need is transparency of bits/bytes. Let a programmer decide if or where things are validated and what to do about handling invalid input at that moment; do not force the issue upon them.

View on HN

(author here) Yes, that is exactly why I wanted to provide an upgrade path from Python 2 to Oil (reusing a portion of its CPython). But I mentioned in this post that I'm explicitly cutting it out for reasons of engineering effort.

But yeah I still think that idea has merit... Somewhere on my blog I made the observation that Python 3 and Perl 6 are both worse than their predecessors for the shell scripting domain. They're better languages in general, but worse for that domain -- which is still important and getting bigger along with the rest of computing, IMO.

I think Oil would be useful in any case, but both Python 3 and Perl 6 opened up that gap in the design space even wider. I mean the string is the CORE TYPE of such programs, so it's important that it be logically applicable to the problem.

I'm sure many people saw the Mercurial / Python 3 thread on HN last week: https://lobste.rs/s/pnuak4/mercurial_s_journey_reflections_o...

Basically the same story, although I questioned their decision of not embedding the Python 2 interpreter instead. I think the only reason not to do that is "memes".

Please note that Perl 6 has been renamed to Raku (https://raku.org using the #rakulang tag on social media).

With regards to being better / worse in the shell scripting domain, I would say that currently performance wise, Perl is better for shell scripting. But development of shell script-wise, I think Raku has superior handling of command line parameters built in: https://docs.raku.org/language/create-cli

I agree; at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler.

Python did eventually acquire a sensible (lossless, errorless) way to convert back and forth between binary and Unicode — UTF-8B, known as surrogateescape, described in PEP 383. Unfortunately, it isn't the default everywhere. PEP 540 makes it default in some cases.

This is the reason Oil is written in its own private fork of Python 2 instead of in Python 3.

Yes those kinds of errors are extremely common... In fact 2 days ago I was visiting a well-known company in SF, and someone mentioned the reason for an aborted Python 3 rollout: a database contained utf-16 surrogate pairs mis-encoded in utf-8.

To me this is the same problem I had with HTML and DOM mentioned on the other thread (which I will reply to!).

The analogy is

    HTML files : DOM :: string data :: Python strings.

I've long had a rant about "models/abstraction vs. reality", which is basically: Reality always wins :) The DOM is an overabstraction of HTML to the point of causing bugs and performance degradation; ditto for Python strings and "real world" textual data. I'd rather solve "honest problems" than self-inflicted problems caused by the wrong abstraction.

A lot of software is basically "undoing" problems caused by other software, not addressing something in the application domain. PEP 383 and 540 seem to be clear instances of that. (And note that I said Python is an amazing achievement in the appendix of this post, which is simultaneously true :) )

> at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler.

Woah. If you're using a dynamic programming language for mission-critical software and your tests don't properly cover "coercing binary data into a Unicode string", then your problems run way deeper than Python 2 vs 3.

This isn't restricted to on-orbit software. Your GSE ("ground support equipment", for those not in the industry) also shouldn't have that kind of quality problems.

The problem, as I said, happened inside an exception handler; if we had written tests for that error case, they might have passed if it had thrown a UnicodeDecodeError instead of an IOError or whatever it actually was throwing.

In defense of Satellogic, it never would have gotten off the ground using traditional risk-minimizing aerospace engineering practices. And it's arguable whether the software in question was "mission-critical" — the mission did continue, after all. My efforts to raise the bar on software, process, and documentation quality were not appreciated.

> The problem, as I said, happened inside an exception handler;

Ah, I missed that bit. Still, that sounds to me like a problem with Python's attitude in particular, and with the attitude of many (not all) dynamic programming languages in general, and not so much with a problem of Python 2 vs 3.

> In defense of Satellogic, it never would have gotten off the ground using traditional risk-minimizing aerospace engineering practices.

Having worked for several years in the traditional space industry: Yes, I totally get what you're saying. In fact, the paperwork-heavy quality assurance we have to satisfy, which is more concerned with formalities than with actual hardware/software quality, probably would have missed that case as well.

> In fact, the paperwork-heavy quality assurance we have to satisfy, which is more concerned with formalities than with actual hardware/software quality, probably would have missed that case as well.

Do you have testing coverage metrics for exception handling? In my experience outside Satellogic, that's one of the places testing usually misses, especially in languages with Python's attitude, which I agree is not well suited to on-orbit code.

> Do you have testing coverage metrics for exception handling?

I have not personally come into contact with software PA/QA requirements, only with those for FPGA designs (VHDL in particular). In my largest project, I had to meet 100% statement coverage, with documented justifications for any uncovered lines. Typically, in such an environment, exceptions are a no-go for flight software.

Yeah, that makes a lot of sense to me. Clearly Python is not an option for such an environment; adding two small integers can raise MemoryError if it doesn't have space to allocate for the resulting integer object.

There are several notes on "failure-free programming" in Dercuano; maybe you'd be interested. Essentially I came to the conclusion that although purely local restrictions (like not allocating memory or raising exceptions) can be helpful, I don't know how to structure a program so that all of the desired safety properties can be checked locally, except for very simple programs. Let me know what you think if you do check it out; I'd be interested in your opinion.

Use the "bytes" data type for raw bytes.

All the standard libraries need to use it too. golang gets this part right.

In golang strings are immutable sequences of bytes (which can be concatenated) or have 'slices' taken from within them (I believe addressed to a rune; which IIRC is a complete UTF-8 sequence). Standard libraries typically operate with bytes[] for low level operations (file I/O) but strings and bytes can be coerced between each other with ease, and notably many objects (such as a File https://golang.org/pkg/os/#File.Write ) have methods for interacting with bytes[] and strings.

There's a core library specifically for low level Unicode operations and external libraries for more complex tasks.

Go gets this very wrong, because you can still index strings as arrays - and you get bytes back, which 99% of the time is not what you want. But which also happens to "just work" on common ASCII inputs, so you get code floating around that is broken without its authors' knowledge.

(This, coincidentally, was also the problem in Python 2, and why Python 3 breaking the world was necessary, if inconvenient for some.)

At this point, the only sensible design for strings is as fully encapsulated, opaque values that are emphatically not arrays of anything. If you want to index them, you should be specifically saying what you want - nth byte, nth codepoint, nth grapheme etc. If you want content as an array of bytes in a particular encoding, then you should ask for that specifically. Internally, yeah, it should use UTF-8, because it makes the most sense - but that's just an implementation detail.

So far, the only language I know that got this right is Swift.

Rust also gets this right: Every string and every string slice is guaranteed to be a valid UTF-8 sequence. If you slice a string at invalid UTF-8 boundaries, then – depending on the operation – the thread either panics, or None is returned.

I agree with your stated goals, but I'm less convinced Go gets this totally wrong. It has the nice property that a string is a string is a string -- a slice of bytes. The only exception is raw strings which as part of source code are defined to be UTF-8.

So, while there's a tradeoff in elegance I do think it's logically very simple. I do think the Rust approach for example is more "correct", but it's also tedious to work with and I think the ergonomics matter (can't speak for swift).

How do you feel about how golang handles runes?

Example: https://play.golang.org/p/7XWC9Irui3N

x = abc

3 vs 3

y = é

3 vs 2

z = 世界

6 vs 2

s>>r 0 1 true e

s>>r 1 2 true ́

[]b>>b 0 true true 00000065

[]b>>b 1 false true 000000CC

[]b>>b 2 true false 00000081

(not putting in a code block given the above and mobile...)

package main

import ( "fmt" "unicode/utf8" )

func main() { var x, y, z string; x = "abc"; y = string([]byte{0x65, 0xCC, 0x81}) // 'e' + 0x0301 ( 0(011 00)(00 0001)) = "é"; z = "世界"; fmt.Printf("x = %s\n%d vs %d\n\ny = %s\n%d vs %d\n\nz = %s\n%d vs %d\n\n", x, len(x), utf8.RuneCountInString(x), y, len(y), utf8.RuneCountInString(y), z, len(z), utf8.RuneCountInString(y)); for k, v := range y { // iterates as runes, not string slices //fmt.Printf("s %d\t%s\t%v\t%s\n", k, utf8.RuneCountInString(v), utf8.ValidString(v), v); fmt.Printf("s>>r %d\t%d\t%v\t%c\n", k, utf8.RuneLen(v), utf8.ValidRune(v), v); } for k, v := range []byte(y) { // iterates as byte units, not runes fmt.Printf("[]b>>b %d\t%v\t%v\t%08X\n", k, utf8.FullRune([]byte{v}), utf8.RuneStart(v), v); } }

It seems like you want to do string-like standard library operations on byte arrays. Python 3 is clear here. You can do string stuff with strings only. If you want to do string stuff, you must decode the bytes.

The world is not only about English and ASCII.

People who want these shortcuts usually don't understand what problems Unicode and character encodings solve in general. And that's exactly what produces software that won't allow me to properly enter my Hungarian name as it is, with diacritics.

It's utterly inexcusable and lazy and stems from an annoying sense of Anglo superiority.

Reply on news.ycombinator.com