Chris_Newton

≤ ≥ ≠ × — – “ ” ’ ° … ¹ ² ³ ™ • ♣ ♢ ♡ ♠

If you work in languages other than English but have a standard English keyboard layout, a compose key is handy for typing accents and non-English letters/ligatures too.

While the modern world of mobile devices and near-permanent fast and reliable connectivity has brought some real advantages, it has also brought the ability for software developers to ruthlessly exploit their users in ways no-one would have dreamt of 20 or 30 years ago. Often these are pitched as if they are for the user’s benefit — a UI “enhancement” here, an “improved” feature there, a bit of casual spying “to help us improve our software and share only with carefully selected partners”, a subscription model that “avoids the big up-front cost everyone used to pay” (or some questionable logic about “CAPEX vs OPEX” for business software), our startup has been bought by a competitor but all the customers who chose our product specifically to avoid that competitor’s inferior alternative have nothing to worry about because they have no ulterior motive and will continue developing it just the way we have so far.

The truth we all know but don’t want to talk about is that many of the modern trends in software have been widely adopted because they make things easier and/or more profitable for software developers at the direct expense of the user’s experience and/or bank account.

“Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it’s the only thing that ever has.”

I think it’s very promising, if you believe in the potential of Linux on the desktop, that gaming used to be the standard “Linux doesn’t do what I need, so I stay on Windows” argument. Thanks to a lot of investment and hard work, particularly by Valve and others contributing to software like Wine/Proton, that is no longer the case. Many games work fine on Linux today, even among the big names. Some even have native versions. It mostly seems to be “anti-cheat” measures that are statistically indistinguishable from malware that still cause trouble.

Another potential sticking point for adoption by home users today is that few, if any, of the big streaming services work well on Linux. This also seems to come down at least partly to DRM. A cynic might suggest that this is because Linux will give a more appropriate response if a copy protection system tries to do invasive things that it has no business doing on someone else’s computer. In any case, it’s another significant barrier, but if we could get to the point where you could at least watch HD content like users of other platforms when you’re paying the same subscription fees, it’s another barrier that could fall.

This latter example is, of course, more than a little ironic given the subject of today’s discussion. But then the behaviour that the DRM system is being subverted to protect against by Signal probably wouldn’t fly for more than five minutes on Linux in the first place, so I don’t think Linux not enabling intrusive/abusive DRM is really the problem here…

An alternative, which in FP-friendly languages would have almost identical performance, would be to make the shift in objects more explicit:

    var favoriteFoodsOfFurryPetsOfFamousAuthorsOfLongChineseBooksAboutHistory =
      books
        .filter(book => isLongChineseBookAboutHistory(book))
        .map(book => book.author)
        .filter(author => isFamous(author))
        .flatMap(author => author.pets)
        .filter(pet => pet.isFurry)
        .map(pet => pet.favouriteFood)
        .distinct()

I did once write a moderately substantial application as a literate Haskell program. I found that the pros and cons of the style were quite different to a lot of more popular/conventional coding styles.

More recently, I see an interesting parallel between a literate program written by a developer and the output log of a developer working with one of the code generator AIs. In both cases, the style can work well to convey what the code is doing and why, like good code comments but scaled up to longer explanations of longer parts of the code.

In both cases, I also question how well the approach would continue to scale up to much larger code bases with more developers concurrently working on them. The problems you need to solve writing “good code” at the scale of hundreds or maybe a few thousand lines are often different to the problems you need to solve coordinating multiple development teams working on applications built from many thousands or millions of lines of code. Good solutions to the first set of problems are necessary at any scale but probably insufficient to solve the second set of problems at larger scales on their own.

This only works for versions of your package that do exist in nixpkgs but aren’t currently the default for your chosen channel, so it doesn’t help if your channel is out of date and you want to install a newer version that hasn’t been packaged yet. But then that’s the case in almost any Linux distro if you rely on installing your software from the distro’s native package repo, and much the same solutions exist in NixOS as in other distros. Although if you’re really determined, you can also start writing your own derivations to automate building the latest versions of your favourite applications before they’re available from nixpkgs…

¹ https://lazamar.co.uk/nix-versions/

² https://lazamar.github.io/download-specific-package-version-...

As a fellow daily driver of NixOS, you’ve just summed up my biggest problem with it. You can do almost anything, and once you’ve figured out how and why you do it a certain way, that way often makes a lot of sense and the NixOS design can offer significant benefits compared to more traditional distros without much downside. But NixOS is out of the mainstream and its documentation is often less than ideal, so there is a steep learning curve that is relatively difficult to climb compared to a more conventional distro like Ubuntu.

The shared object problem in particular comes up so often, particularly if you use applications or programming languages with their own ecosystem and package management, that I feel like having nix-ld installed and activated by default with a selection of the most useful SOs available out of the box would be a significant benefit to new users. Or if including it in a default installation is a step too far, many users would probably benefit from some HOWTO-style documentation so they can learn early on that nix-ld exists, how it helps with software that was built for “typical” Linux distros, why you don’t need or want it when running software that was already built for NixOS such as the contents of nixpkgs, and how to work out which shared objects an application needs and how to find and install them for use with nix-ld.

I haven’t yet felt confident enough in my own NixOS knowledge to contribute something like that, but one nice thing about the NixOS community is that there are some genuine experts around who often pop up in these discussions to help the rest of us. I wonder if there’s scope for sponsoring some kind of “Big NixOS Documentation Project™” to fund a few of those experts to close some of those documentation gaps for the benefit of the whole community…

When it works, it’s great. I like that I can install (and uninstall) much of the software I use declaratively, so I always have a “clean” base system that doesn’t accumulate numerous little packages at strange versions over time in the way that most workstations where software is installed more manually tend to do.

This is a trade-off, though. Much is made of the size of the NixOS package repository compared to other distros, but anecdotally I have run into more problems getting a recent version of a popular package installed on my NixOS workstation than I had in probably a decade of running Debian/Ubuntu flavoured distros.

If the version of the package you want isn’t available in the NixOS repo, it can be onerous to install it, because by its nature NixOS doesn’t follow some popular Linux conventions like FHS. Typically, you write and maintain your own Nix package, which often ends up similar to fetching a known version of a package from a trusted source and then following the low-level build-from-source process, but all wrapped up in Nix incantations that may or may not be very well documented, and sometimes with a fair bit of detective work to figure out all the versions and hashes of not just the package you want but also all its dependencies, which may in turn need packaging similarly themselves if you’re unlucky.

It’s also possible to run into this when you’re not installing whole software applications, including those that are available from the NixOS package repository, but rather things like plug-ins for an application or libraries for a programming language. You might end up needing a custom package for the main application so that its plug-in architecture or build system can find the required dependencies in the expected places when you try to install the extra things. Again, this is all complexity and hassle that just doesn’t happen on mainstream Linux distros. If I install Python and then `pip install somepackage` then 99.9% of the time that just works everywhere else but frequently it won’t work out of the box on NixOS.

It’s one of those things that is actually perfectly reasonable given the trade-offs that are explicitly being made, yet still makes NixOS time-consuming and frustrating in a way that other systems simply aren’t when you do run into the limitations.

This comment is already way too long, so I’ll just mention as a footnote that NixOS also tries to reconcile two worlds, and not all Linux software is particularly nicely arranged to be managed declaratively. So in practice, you still end up with some things being done more traditionally/imperatively anyway, and then you have a hybrid system that compromises some of the main benefits of the declarative/immutable pattern. There are tools like Flakes and Home Manager that help to overcome some of this as well, and as others have said, they are promising steps in good directions, but we’re not yet realising the full potential of this declarative style and it’s hard to see how we get from here to there quickly.

ORMs have entered the chat…

These sometimes use a lot of dynamic modification, such as adding implicit ID fields or adding properties to navigate a relationship with another type that is defined in code only from the other side.

It can also be awkward to deal with “not null” database fields if the way the ORM model classes are defined means fields are nullable as far as the Python type hints are concerned, yet the results of an actual database query should never have a null value there. Guarding against None every time you refer to one of them is tedious.

I’m not exactly the world’s loudest advocate for ORMs anyway, but on projects that also try to take type safety seriously, they do seem to be a bit of a dark corner within the Python ecosystem.

Matrix dimensions are certainly doable, for example, because templates representing mathematical types like matrices and vectors can be parametrised by integers defining their dimension(s) as well as the type of an individual element.

You can also use template wizardry to write libraries like mp-units¹ or units² that provide explicit representations for numerical values with units. You can even get fancy with user-defined literals so you can write things like 0.5_m and have a suitably-typed value created (though that particular trick does get less useful once you need arbitrary compound units like kg·m·s⁻²).

Both of those are fairly well-defined problems, and the available solutions do provide a good degree of static checking at compile time.

IMHO, the range question is the trickiest one of your three examples, because in real mathematical code there are so many different things you might want to constrain. You could define a parametrised type representing open or closed ranges of integers between X and Y easily enough, but how far down the rabbit hole do you go? Fractional values with attached precision/error metadata? The 572 specific varieties of matrix that get defined in a linear algebra textbook, and which variety you get back when you compute a product of any two of them?

¹ https://mpusz.github.io/mp-units/

² http://nholthaus.github.io/units/

Indeed. Relying on truthiness has always felt very un-Pythonic to me, not least because it contradicts several principles in the Zen of Python:

• Explicit is better than implicit.

• Special cases aren’t special enough to break the rules.

• There should be one — and preferably only one — obvious way to do it.

I wrote a more about this in a Reddit post a while back if anyone’s interested: https://www.reddit.com/r/Python/comments/16w97i6/flask_300_r...

    user.password = await hashPassword(user.password);

2. The mutation here apparently modifies an object passed into the function, which is a side effect that callers might not expect after the function returns.

3. The mutation here apparently changes whether user.password holds a safe hashed password or a dangerous plain text password, which are bad values to risk mixing up later.

4. It’s not immediately obvious why hashing a password should be an asynchronous operation, but there’s nothing here to tell the reader why we need to await its result.

At least three of those problems could trivially be avoided by naming the result hashedPassword and, ideally, using TypeScript to ensure that mixing up plain text and hashed passwords generates a type error at build time.

I do agree with many of the other comments here as well. However, I think the above is more serious, because it actually risks the program behaving incorrectly in various ways. Questions like whether to use guard clauses or extract the password check into its own function are more subjective, as long as the code is written clearly and correctly whichever choices are made.

https://news.ycombinator.com/item?id=12048693

I continued to work on that project for a while afterwards, but I don’t think my overall views on the pros and cons of the literate style changed much from what I wrote there.

If there’s anything you’re particularly curious about, I’m happy to discuss further (though my professional experience was limited to that one project, so I’m hardly an expert).

The sidebar to show the documentation is just a presentation detail. What I’d really like as a developer is (a) the ability to attach documentation with a specific scope ranging from small things like a function parameter or variable, though larger things like a function or a set of related functions, up to whole files or packages, and then (b) editors that recognise the scoped documentation and present it contextually.

I’d like all relevant comments to be easily visible when I might want to refer to them. Also, I’d like to know what documentation exists in the first place, which is not always obvious when we rely on things like big banner comments at the top of files/sections or having separate README files scattered through the tree.

I would like to use those same characters, and where applicable with the same sizes and alignments of related characters, everywhere — in my editor, my linters and formatters, my diff tools and version control system UIs, my greybeard CLI text file batch processing incantations, everywhere.

All that really needs is languages that use Unicode characters like ≤, fonts that include all the relevant glyphs, and tools that provide a simple way to enter those characters. But all of these are important, because right now we don’t have simple, standardised ways of doing these things, and so we mostly don’t do them at all.

It’s obviously not a big deal, just a small “quality of life” improvement, but it’s 2024 and I don’t see any good reason why we can’t have nice things. :-)

I agree, though there is a lot of subtlety in this problem. I tried, with very modest success, to write a tool that would overlay some graphics showing these kinds of information on top of a regular code listing.

The results were rather satisfying when the structure to be illustrated was relatively simple, as it often is in well-written code. However, trying to use those visuals to help analyse and understand less well-written code with complicated, tangled flows felt like looking at spaghetti code in a very literal sense.

Qualitatively it made it obvious when code could benefit from some cleaning up, but that isn’t very useful information unless it also provides useful insight into how to clean the code up, and I never really found a style/format for the visuals that conveyed enough information to provide actionable insights but didn’t become so busy in bad cases that it became was hard to follow.

I know some editors specialize in novels by having wiki like features of linking characters to storyline and research work. I'm not sure how much of those are used by real world authors though…

I’ve only had this discussion with one author recently, but FWIW, they swear by that kind of structured story planning software. I’m not sure how many of the concepts or presentation details in those applications would translate readily to tools for working with code, though.

What I’m advocating here is a little different, though. While it would certainly be useful to write documentation using more expressive formats than plain text, here I’m arguing for the ability to (a) attach that documentation to specific parts of the code at different scales and then (b) have all relevant information at all scales readily available depending on what I’m currently editing.

One of the reasons I would like to have this ability in my development environment is that I found the literate style to be very helpful for writing informative documentation but relatively unhelpful for finding it again later. By its nature, literate documentation is organised primarily for a linear reading flow, like the story in a book, but most of the time when I’m editing code, that’s not my typical searching behaviour.

Given the potential of modern graphical environments and typography, I’d like to see languages and editors adopt simple spacing improvements like elastic tab stops¹. I don’t want to chose between an aesthetically pleasing layout that draws attention to patterns and anomalies in my code and a layout with completely uniform spacing because anything more carefully aligned is a pain to maintain in later edits and adds noise to diffs. We could shrink blank lines to half-height as well, which would keep the useful visual separation but waste less space.

I’d like to use a (somewhat) wider range of basic symbols than whatever we had in ASCII last century. Even allowing comparison operators like ≤, <, ≠ and = that align consistently would be a useful start.

I’d like simpler syntax highlighting, concentrating on distinguishing major language elements like literals, variables and types, and emphasising problems I probably care about like unclosed strings and misspelt keywords. I’d also like to see colour used more effectively for matching related code like opening/closing pairs or the definition of a value and later references to it.

I’d like to see a sidebar for documentation, not unlike the example for comments in the article, but allowing documentation to be attached at different levels from an individual function or variable up to an entire file or beyond, then having the editor automatically offer all documentation relevant to wherever I am in the code and whatever I’m typing there.

If we were designing a UI for another application, we’d always consider aspects like spacing, symbols and iconography, colour schemes and how to show and navigate related information. It’s strange that we give so little consideration to them in the tools we use ourselves!

(Join me next week for part two of the epic rant, provisionally entitled Why aren’t we all using semantic diff tools yet?)

¹ https://nick-gravgaard.com/elastic-tabstops/

Given the UK’s privacy and data protection environment is still largely the same as the EU’s, I wonder whether this was an oversight.

If I’m investigating a new field or trying out a new language or library, relative to my own experience, then it’s quite common for an AI code generator to use idioms or libraries I hadn’t yet come across. That alone sometimes saves me a useful amount of time doing research.

However, it’s almost all breadth and very little depth. The quality of the generated code is rarely better than something a junior-to-mid-level developer might have written. It needs to be reviewed and corrected with similar diligence.

Similarly, the quality of a generated review of existing code or of generated supporting assets like test cases or documentation is often superficial and error-prone. I rarely find it an overall win to use current AI-based tools for these things instead of existing tools that can’t do as much but are consistent and reliable at what they do do.

So I wouldn’t necessarily expect current AI tools to help me code better, only sometimes a bit faster, and that mostly in new areas I’m exploring rather than areas where I’m doing professional work that is going to get shipped in the near future.

Offering hands-on software development and technical leadership services, I specialise in:

• Mathematical, scientific and engineering applications

• Custom user interfaces and data visualisation

• Software for hardware (IoT, lab equipment, embedded)

I have over 20 years’ experience developing software, mostly working with smaller organisations. If your situation looks anything like these, I could probably help you:

• A scale-up struggling to overcome early tech debt and transition to more sustainable development

• A hardware start-up looking to outsource software development to a multi-skilled freelancer

• A software start-up that can’t hire permanent senior+ level developers fast enough to hit targets

• A research group needing a professional to implement models and visualisations in good code

Every project is different and will be treated as such, but recently I’ve been favouring TypeScript or Python for web development (with React or Vue for UIs) and Python, C++ or Haskell for more mathematical or hardware-related work. I have experience with a wide range of target platforms from AWS to embedded ARM.

If you have a project that I may be able to help you with, you’re welcome to get in touch. My contact details are in my profile.

I find them valuable. For example, I have a workstation that is used for different projects with different clients, as well as administrative work for my own business. I want 100% separation between assets related to those different contexts.

It’s bad enough that we have package managers allowing package installation scripts to run arbitrary code, or software wanting you to install via:

    curl https://example.com/imnotmalwareipromise.sh | sh

At least if I have separate users for each client or each major project then the worst that is going to be compromised by a vulnerability introduced during the work for that client or project is that same work.

It’s not just about security though. It’s also about convenience and manageability. Those different clients and projects frequently require the use of specific security credentials and configurations, often for remote services that other clients/projects also use. In a perfect world, I’d like all of the software I use to be XDG-friendly, and I’d like each client/project to have its own home directory with its own independent XDG-style directories underneath, so each user has the configurations and credentials required for its own work and has no knowledge of or access to those of any other user. Finished a project? Archive/nuke that entire user and home directory as appropriate, and nothing is left lying around to break anything or leak anywhere later.

I’m currently playing with NixOS, which means I can also have a limited set of system-wide software installed and have specific additional packages installed per-user or even activated on demand when I change into a specific directory. Again, this means my system has only the software I actually need available at any given time, at the exact version I need for that specific work, and if something is no longer needed by anything I’m doing then it will automatically get cleaned up next time I do an update/rebuild.

None of this really works without the concept of separate users running different software in their own isolated little worlds, possibly concurrently on the same workstation and even sharing the same input/output devices (in a safe way where again they can’t unreasonably interfere with each other – something else that is not 100% there yet, but certainly a lot better than on de facto single-human-user operating systems). The only real alternative is to spin up something like a different virtual machine for each client/project where everything from the OS down is isolated, but I don’t really gain anything by doing that and it’s potentially more work to set up and more difficult to share input/output devices.

Offering hands-on software development and technical leadership services, I specialise in:

• Mathematical, scientific and engineering applications

• Custom user interfaces and data visualisation

• Software for hardware (IoT, lab equipment, embedded)

I have over 20 years’ experience developing software, mostly working with smaller organisations. If your situation looks anything like these, I could probably help you:

• A scale-up struggling to overcome early tech debt and transition to more sustainable development

• A hardware start-up looking to outsource software development to a multi-skilled freelancer

• A software start-up that can’t hire permanent senior+ level developers fast enough to hit targets

• A research group needing a professional to implement models and visualisations in good code

Every project is different and will be treated as such, but recently I’ve been favouring TypeScript or Python for web development (with React or Vue for UIs) and Python, C++ or Haskell for more mathematical or hardware-related work. I have experience with a wide range of target platforms from AWS to embedded ARM.

If you have a project that I may be able to help you with, you’re welcome to get in touch. My contact details are in my profile.