Simple Is Not Small
jyn.dev
jyn.dev
MIT/Stanford:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the interface to be simple than the implementation.
New Jersey:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.
TFA maps "simplicity" to "MIT/Stanford simplicity" (simplicity for the user) and "smallness" to "New Jersey simplicity" (simplicity for the developer).
I wonder if the root of the tension between the two schools comes down to the ambiguity of the user/developer distinction. Developers are also users. Simplicity of implementation is helpful to developers when they are working directly on implementation, while simplicity of interface is helpful to developers when they are using other developers' work.
Another interesting irony I'll note, Lisp is the "New Jersey approach" towards symbolic AI. Americans clinging to their Lisp systems were deeply entrenched in a "worse-is-better" mindset. Your interface, the computational model, didn't need to be designed for logic programming, that was wholly secondary. Do everything as much as possible in Lisp, and then offload the relational description to a small (not simple) library. American knowledge engineers were looked at as overpaid procedural hackers with zero mathematical elegance and very little credibility. More or less the same perception these self-same Lisp-machine users had for Unix and C programmers.
It's all about perspective, at the end of the day. Where we draw the line in the sand on these categories is free-to-choose, yet it also determines everything. We're always someone else's villain under different semantics.
Sorry, I meant Microsoft Copilot 365.
Now consider Microsoft Word.
Excel is great, but it's an arcane beast rivaled only by Emacs configs. People pass down Excel formulas by word of mouth like they're magic spells.
From the authors, Charles Simonyi went to Microsoft as one of the original Word implementers, while Excel was born on Mac OS and only later ported to Windows.
One of the core features of the Unix command-line is that it is user-extensible. If there's no "native" command equivalent to frequencies, you can write your own, and it will be given the same first-class treatment as any other binary in your PATH. This is entirely in keeping with the Unix philosophy of simple implementations.
OK but how would it look like if you had such a program? Shells are not known for having the extensive set of functions that real programming languages have.
That said having key/value semantics and not just stream of bytes would make the shell much more versatile, at the cost of making it bigger.
All of this is completely correct, but it has nothing to do with bash vs. Clojure. The same decoupling can easily be achieved in either language.
But Unix pipelines are not simple too. They have a couple of nitty-gritty details that often come out and bite you.
1. They can only stream raw bytes, so all the programs that deal with lists like sort and uniq have to separate items using a delimiter (usually newline). If you want to process data with that delimiter in it, you're in for a ride. And if you want to write a custom tool, you have to do all the splitting yourselves (luckily it's so common most programming language will provide a ready-made facility for you to do that). This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".
2. Error are hidden by default in shell. Nowadays you can explicitly change this behavior, but you have to remember to do `set -o pipefail` and I don't think it was always there.
3. There is no data typing at all. Everything is binary or text. Nowadays a lot of programs just output JSON, and the users (if they even stay inside the shell) almost always reach for jq to parse it. But jq is not a Unix philosophy program: it's an entire streaming functional programing that can do quite a lot. But even jq gets hairy when you have to do a bigger query or transformation. In that case users often reach out to Python or another language and just move the business logic there.
I think this fits well with what the article is trying to say: Unix pipes are pretty easy (small) to implement on your own (compare that to something like Nushell's pipes). But the moment you to do something that's a little different than the happy path it was built for, you need to go for another tool (jq) that has its own built-in pipe and small programming language, because Unix pipes won't cut it. And for more complex (hehe) things, you'll have to reach for a larger (and simpler) tool: a full-fledged programming language.
I'm not sure if you're actually trying to argue for a specific position here, but you highlight several negative results of one "philosophy" of development, with the implication that the alternative wouldn't have those negatives.
This is a tricky point to refute because you're right, these are flaws and there could be a system that doesn't have them.
So why do these flawed systems exist and proliferate?
Because the real life result isn't actually a choice between "sloppy but quick to develop" and "elegant well engineered but slow".
The choice is actually between "sloppy but exists" and, well, nothing, because the other version never actually materializes.
(And of course, I feel compelled to point out that bash is a user interface not a programming language. Any attempt to replace it or improve it without focusing on that main point is doomed to failure, which is why you see so many people who apparently think that what bash really needs is strict type checking or something and end up creating a completely awful user experience)
awk '{ freq[$0] += 1 } END { for (n in freq ){ print freq[n] " " n } }' < file.txt
Honestly, so much of our conceptions of "what's wrong" is more to do with lack of familiarity with history than any actually unsolved problem.
I am currently building a piece of very modular software and it has been the hardest-to-design project of my entire career. I would never be allotted this amount of time-effort at any job I have held to make something this robust and clearly defined. Many aspects of this project have taken 3-5 rounds trying-and-trashing to get an abstraction that is uncomplicated.
This is precisely why vibe coding is so successful for building tiny isolated scripts, and so disastrous for anything else. It's just really dang hard to build something large and simple.
Bottom line is probably true, but if you are an open-source maintainer mentioned, and you have only so few hours to spend, you just cannot create those gigantic softwares either. You need to choose from the cards on your hand.
Brooks, in “the Mythical Man-Month”:
“One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams.
Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.
In the upper left of Fig. 1.1 is a program. It is complete in itself, ready to be run by the author on the system on which it was developed. That is the thing commonly produced in garages, and that is the object the individual programmer uses in estimating productivity.
There are two ways a program can be converted into a more useful, but more costly, object. These two ways are represented by the boundaries in the diagram.
Moving down across the horizontal boundary, a program becomes a programming product. This is a program that can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data. […] As a rule of thumb, I estimate that a programming product costs at least three times as much as a debugged program with the same function.
Moving across the vertical boundary, a program becomes a component in a programming system. This is a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks. To become a programming system component, a program must be written so that every input and output conforms in syntax and semantics with precisely defined interfaces. The program must also be designed so that it uses only a prescribed budget of resources — memory space, input-output devices, computer time. Finally, the program must be tested with other system components, in all expected combinations. […] A programming system component costs at least three times as much as a stand-alone program of the same function. The cost may be greater if the system has many components.
In the lower right-hand corner of Fig. 1.1 stands the programming systems product. This differs from the simple program in all of the above ways. It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts.”
> Clojure decouples data representations from type checking.
This is funny to me because seen from the other side, (this) Clojure couples runtime type information to data structures: you're no longer allowed to define a data structure that doesn't have some runtime type information attached. A fixed static structure is just the consequence of not adding dynamic type information.
Meanwhile in Rust you can get type-checking ‘without’ a fixed structure by using trait objects.
In this sense, most statically-typed languages conflate how data is structured with how it is restricted. Some overlap is unavoidable, as anything represented by a single byte is always going to be restricted to at most 256 values, but Clojure tends to take the view that the more decoupling (or decomplecting) you can achieve the better.
This can be useful when dealing with data that is in some sense invalid. You might receive data that's outside expected bounds or even of a different type, and it might make sense to handle it in some fashion. This is a common necessity in pharmaceutical trials, for example.
> In this sense, most statically-typed languages conflate how data is structured with how it is restricted.
Most statically-typed languages are actually very loose about how values are structured at runtime, leaving it mostly up to the implementation (e.g. see C++ padding and field reordering, or Haskell's autoboxing, which makes approximately no guarantees about what's behind the pointer — usually some graph-rewriting metadata). Where the conflation does exist is that a lot of systems languages allow you to write and typecheck code that assumes something about the language's representation of the type's values (e.g. that you can take the address of a field of a struct and later dereference it), though you can usually opt out of that with PIMPL or a trait object or something. But guaranteeing (and allowing the programmer to rely on the guarantee) that every value's representation also carries a bunch of additional runtime information is a much stronger version of that coupling.
> This can be useful when dealing with data that is in some sense invalid. You might receive data that's outside expected bounds or even of a different type, and it might make sense to handle it in some fashion.
The very fact that you can handle that data at all means that the value carries additional type information that allows you to do so. It's only ‘decoupled’ from the type in the sense that you didn't have to write it there, because it's automatically coupled to every value representable in the language regardless of what type you give it.
> This is a common necessity in pharmaceutical trials, for example.
I'll have to do some guesswork here, but I imagine when people make arguments like this they are significantly imagining a situation in which, say, all the values are expected to be in the range [5, 100] and some befuddled experimenter or piece of machinery gives you the value 2. A-ha, you say: I know sometimes the equipment undermeasures near the bottom of its range, so I'll clamp this value to 5!
This isn't really a type error. The fact you know you can safely do that means that the data is really typed in a different range than you said — but it's still typed. The conceptual type (even if you never write it down) is inherent in the very fact that you can somehow handle it: you know what to do with values down to 2 so the real type of supported inputs is at least [2, 100] (with some special semantics for the low values beyond that of being numbers).
A real type error looks like: you're expecting values in [5, 100] and then one of the values actually turns out to be the concept of intellectual honesty. That type doesn't support ~any of the same operations as the numbers you were expecting, even with the extended domain that more accurately reflects the set of values you can really accept. Even discarding it might have disastrous results for your experiment! In fact, the concept of intellectual honesty doesn't even have a good discriminator: while I know that's what you got because I'm the rascal who snuck it in there, you have no idea what it is, and no way of finding out. Most likely you're going to try to compare it to 5 to see if it needs to be clamped, with unpleasant consequences for us all.
I think you're using 'value' to mean something slightly different to the way I meant it. So to get us onto the same page, by 'value', I mean something independent of how its stored; that is, the number 1 is a value, whether it's stored as as 00000001 or 0000000000000001.
A type limits both which values it is possible to represent (i.e. a u8 limits us to representing the integers 0 to 255), and also determines how it is encoded in memory (in this case 8 bits).
> Most statically-typed languages are actually very loose about how values are structured at runtime
Yes, but ultimately the compiler needs to be able to map a sequence of bits to the value it represents, even if there's not a strict one-to-one mapping.
> I'll have to do some guesswork here, but I imagine when people make arguments like this they are significantly imagining a situation in which, say, all the values are expected to be in the range [5, 100] and some befuddled experimenter or piece of machinery gives you the value 2
That's a common assumption, but the reality can be much more messy. It might be that we expect an integer between 5 and 100, but receive a string of UTF-8 characters instead.
For example, suppose you want to record a patient's date of birth. In Clojure, we might represent that as a map that connects a :patient/birthdate key with an encoded date object:
{:patient/birthdate #date "1972-04-08"}
In this case, the doctor might record the data as:
{:patient/birthdate "before 1980"}
I'm using it in its most general sense: as an object of discourse in a programming language, independent of representation or semantics. 1 is a value in most languages, to be sure, but it's also specific type of value, viz. a number: you can do number things to it, like add it or divide it, or check it for equality with 2.
> A type limits both which values it is possible to represent (i.e. a u8 limits us to representing the integers 0 to 255), and also determines how it is encoded in memory (in this case 8 bits).
To reiterate: a type doesn't limit the values but specifies the values (or rather, more generally, the meaningful operations on a value of that type). Values are not numbers by default; only by being typed as a number does a value take on number semantics. Without the knowledge that a value is a number it is meaningless to treat it as a number.
> Yes, but ultimately the compiler needs to be able to map a sequence of bits to the value it represents
Sure; in any language implementation you have to represent the values somehow, nobody could disagree. My point is that the type doesn't (necessarily) specify that representation in any language I can think of.
> That's a common assumption, but the reality can be much more messy. It might be that we expect an integer between 5 and 100, but receive a string of UTF-8 characters instead.
Sure: that's not fundamentally different from the first example I gave. The real type of `:patient/birthdate` there is just the (discriminated) union of the date type and the string type. It still has a type, and if it didn't you wouldn't be able to process it (definitionally, because a type tells you what kind of processing makes sense for the value). And the string values aren't ‘outside’ the type: even if you choose to write the wrong type down in your Clojure, the fact that you also process strings means that you know the real type (and you embed that knowledge into the code).
But a value can have more than one possible type. The number 1 could come from an unsigned byte, or a signed long, for example. These are different types that support the same numerical operations, but differ in cardinality. So we can't say that a type's only purpose is to specify the meaningful operations on a value, as we might have two types that are identical in that regard.
> The real type of `:patient/birthdate` there is just the (discriminated) union of the date type and the string type.
Yes, in this particular instance that would be the case, but that's not necessarily something you know ahead of time. The point is that you may not have anticipated that not everyone would know their date of birth, and the data you receive is invalid according to your earlier assumptions.
In Clojure this results in a more graceful failure condition. Functions that don't require the date of birth will continue to work with no change required. If I want the average white blood cell count of a patient, I don't care what the date of birth is, and therefore the output for that particular operation isn't affected.
Here we disagree. Unsigned 8-bit integers† and signed 64-bit integers, while both conveniently notated with Arabic numerals, are actually different values that support different operations, for example negation. They share quite a few similarities in how their operations interact with one another, for example each (assuming wrapping) is a monoid with 0 and +, but the semantics of the actual operations differs if looked at more closely. Mathematics agrees: the element ‘1’ of N/2⁸ and the element ‘1’ of Z/2⁶⁴ are not the same thing (their encodings coincide sometimes, but it's poor form to make assumptions about it).
† Bytes are data but not numbers, and so support only data operations like duplication and discarding, not number operations like adding and multiplication: as an artefact of representation you can usually ask the hardware or programming language to manipulate them as if they were numbers, but the result remains meaningless.
> The point is that you may not have anticipated that not everyone would know their date of birth, and the data you receive is invalid according to your earlier assumptions.
But that's exactly what I'm saying: you must have made some assumptions about that missing data, otherwise there is no safe thing you can do to it (including discarding it, which is a popular choice). This works in Clojure only because Clojure couples some semantics (data semantics plus operations on runtime type information) into every value, i.e. it restricts what values are even representable in the language in order to ensure that this function will always be safe to write.
In more strictly typed languages you can still talk about values that support these behaviours, but you are required to be explicit about it, because there are some values that can be represented that don't support these operations.
> Functions that don't require the date of birth will continue to work with no change required.
Any function that takes the date of birth ’requires’ the date of birth (or more generally a possibly-empty set of ‘leftover’ values). The only thing that differs is what's required from it: some functions might require that it be a date while other functions only require that it be data, for example. The universally imposed limitation that all values must be coupled to data semantics and runtime type information is convenient for ergonomics if you write a lot of these functions (since you don't have to remember to write that assumption down), but it's important to remember that it is a coupling — the resulting values are more complex than values without those things bundled on, and the trade-off is that you can no longer talk about values for which they don't hold.
What about a 32 bit unsigned integer and a 64 bit unsigned integer? Are they still separate values?
You appear to be saying that the values of a type cannot be a subset of another type; that is, there is no '1', only a '1' that is an integer, a '1' that is a short, and so forth, and every '1' is distinct.
Fine, that's a possible way of looking at it, but why is that more valid or consistent than a model that allows subtypes? That, for example, the value '1' could be both a Number, an Integer, and a NaturalNumber?
Further, what's the practical difference between a type defined as consisting of the numbers [1 2 3], and an integer that's restricted to those values?
> This works in Clojure only because Clojure couples some semantics (data semantics plus operations on runtime type information) into every value, i.e. it restricts what values are even representable in the language in order to ensure that this function will always be safe to write.
Even if we view a Clojure value as a coupling between type and data, that's only a coupling between two things, and it ensures we can avoid further coupling caused by large record types. On net, we reduce the amount of coupling a statically typed language that uses closed record types would require.
For instance:
(defn average-wbc-count [{:patient/keys [wbc-counts]}]
(/ (apply + wbc-counts) (count wbc-counts)))
Conversely:
fn average_wbc_count(patient: &Patient) -> f64 {
patient.wbc_counts.iter().sum::<f64>() / patient.wbc_counts.len() as f64
}
That's true, and you can prove it simply by observing that they react differently to ‘the same’ operations, e.g. (2¹⁶ - 1) + 1 has a different value (that responds differently to tests like `≥ 0`) depending on which type we're talking about. There's an injection into the larger type (a type coercion) that is very well-behaved, but it's not an identity map — it changes the operations on the value.
> Further, what's the practical difference between a type defined as consisting of the numbers [1 2 3], and an integer that's restricted to those values?
Nothing (ish: you have to be careful to describe what happens to all the integer operations when restricted to your type) — that's a totally valid way to define a type. But it's not the only way to define a type, because the world of types is much bigger than the world of restricted sets of integers.
> I need to ensure that a patient has some birthdate that's an anticipated type
No, you just have to be explicit about the possibility of a birthdate (or other fields) being of a wide type, e.g.
struct Patient {
wbc_count: Vec<u64>,
birth_date: Box<dyn Any>,
other_fields: HashMap<String, Box<dyn Any>>,
}
Then at which point does a type become equivalent to a restriction that could be decoupled from the underlying type by choosing a broader type?
> No, you just have to be explicit about the possibility of a birthdate (or other fields) being of a wide type
Yes, you could recreate Clojure's semantics in Rust. More accurately, I'd say you'd be looking at defining it like so:
struct WbcCount {
wbc_count: Vec<u64>
}
struct BirthDate {
birth_date: Box<dyn Any>
}
We'd then be faced with the challenge of creating a type that could contain an arbitrary number of unique structs, and to be able to pull a struct out of that set by its type. Realistically we'd probably just use a HashMap at that point and discard all static typing.
Alternatively, we could create a mega struct that contains every possible field we could want to use, whether or not we know they are related, and use this data structure to represent all structured data in the application. The "if everything is coupled, nothing is" approach.
But both of those options are difficult or unidiomatic to write.
Languages like Rust, Java, etc. encourage coupling of data because it's more convenient and space efficient to group data together in records/structs. In Clojure, there's no need to do so; we can couple only when necessary. If we don't need to know the birthdate to calculate the white blood cell count, then we can exclude that field from the input type checking. Most statically typed languages find this difficult, with TypeScript being one of the few exceptions in this regard.
I'm not totally sure how to interpret this question. If I already have a type that includes all the values I want as a subset, I can restrict that type by limiting the values it can take and restricting or removing the operations on it to guarantee they never produce any of the forbidden values. That's the basis of refinement type systems like Liquid Haskell etc. For any type I can describe this way I can also build it ‘from the ground up’ by starting with the empty type and adding operations, though it might not be as convenient. But the converse isn't true: not every type I can build additively can be refined subtractively from another type. For a start, you need to have a broader type to begin with, so that has to already be built somehow: you can't refine an 8-bit integer into an HTTP server.
> You might have a structure that has a WbcCount but not a BirthDate, or one with a BirthDate but not a WbcCount, or one with neither.
Remember that a `Box<dyn Any>` could also be a ‘no value’ type like `()` or a ‘maybe no value’ type like `Option<Date>`. But yes, there are many equivalent ways to write it depending on how likely the values are to exist; the precise formulation is just a question of ergonomics.
> We'd then be faced with the challenge of creating a type that could contain an arbitrary number of unique structs, and to be able to pull a struct out of that set by its type.
By its name, but yes, that's exactly what you get with the `other_fields` field in my example above.
> and discard all static typing
Well, that's not quite true: we discard (syntactically, but not conceptually!) the type information about the fields of this struct, but we can regain it later (from the conceptual information we retain) by trying to downcast the `Any`. Then you get your static type information back and the compiler can help you again.
> But both of those options are difficult or unidiomatic to write.
Sorry, I hope I haven't come across too harsh to Clojure here. As I thought I was clear about above, Clojure's ergonomics for writing dynamically-typed code are vastly superior to Rust's. By coupling runtime type information to values and restricting itself to only being able to deal with values that are data and have runtime type data attached, Clojure gets to make a whole bunch of simplifying assumptions that reduce the work the programmer has to do when dealing with such values. That's the trade-off: Rust chooses to be able to express a much wider range of types by not requiring that coupling, but in exchange you have to be much more explicit when you do want to couple runtime type information to your values. Some languages like C# aim to be able to do both, i.e. start strongly typed and drop down ergonomically into dynamic typing as desired, but Rust is not one of those languages.
> If we don't need to know the birthdate to calculate the white blood cell count, then we can exclude that field from the input type checking.
But that very exclusion requires that you know enough about the value to know that it can be safely ‘excluded’ without having any additional information about it. Specifically you need to know:
- how to access a subfield of the type regardless of the other fields on it — this requires that it contain RTTI
- how to discard the value — this requires that it have a destructor with a known calling convention
- how to move/copy the value into the function (and maybe even move out of the function if you return it or imperatively add it to some global state) — this requires that you know a ‘moving constructor’ for the value that can be used to safely move it to another location
In the Rust case you need to write down these things explicitly so the compiler can make sure you don't pass a value that doesn't satisfy them. In Clojure, the reason you don't need to write them down is that every value in the language is restricted to only be able to represent such values, so those constraints are implicitly on every function that you write.
Sorry, perhaps I'm being a little obtuse. What I'm ultimately trying to get at is that sometimes its useful to have different types for the same data under different circumstances. A subset of data might benefit from a narrower type.
For example, suppose we have some CSV file that contains a bunch of patient information, including average white blood cell count and birthdate. This CSV file might contain bad data! So perhaps we type it as:
struct Patient {
avg_wbc_count: Either<f64, String>,
birthdate: Either<Date, String>,
}
struct PatientWithKnownWbcCount {
avg_wbc_count: <f64>,
}
Obviously which we use depends on the nature of our program, but what I'm trying to get at is that a type isn't set in stone, but something we choose. Ideally we choose the most restrictive and decoupled type for the circumstances, but that might result in having many hundreds of different variations of the same type, so there's a tension between the number of distinct types and how specific or narrow they are.
> In the Rust case you need to write down these things explicitly so the compiler can make sure you don't pass a value that doesn't satisfy them. In Clojure, the reason you don't need to write them down is that every value in the language is restricted to only be able to represent such values, so those constraints are implicitly on every function that you write.
Granted, but Clojure's approach can result in greater decoupling with less effort, particularly when dealing with imperfect data.
In the previous examples I presented a scenario where we might want to think carefully about how exactly we store data that might contain invalid fields. But in Clojure we don't care - we can quite easily have a data structure that's partially invalid or unparsed - the entire problem of how we represent this data in memory is sidestepped.
We can still define what constitutes valid fields:
(s/def :patient/birthdate inst?)
(s/def :patient/avg-wbc-count float?)
Could you do the same in Rust or other similar languages? Sure, it's ultimately just maps and predicates. But many languages aren't geared up to make that pleasant to use.
These two types aren't typing the same data, though: in the latter case (in Rust) you've actually thrown away some of the data, and the type reflects that. In some languages like Clojure or TypeScript you can't distinguish that from the case where you haven't actually thrown away the data and are secretly carrying it around as well as whatever data is explicitly typed there, but if you want to get it back you have to know that it was once there, i.e. have additional (conceptual) type information that you chose not to write down.
> Granted, but Clojure's approach can result in greater decoupling with less effort, particularly when dealing with imperfect data.
I think this isn't a great comparison: the usual vehicle for this thing in Rust is the trait, which manages to abstract over the representation statically without introducing a bunch of runtime machinery for it (though you can opt in to the machinery by using a trait object!). That's very idiomatic in Rust, although the trait syntax is a bit noisier (because it's more general).
> the entire problem of how we represent this data in memory is sidestepped.
It's not sidestepped — the language just makes an opinionated choice for you, which you can't avoid.
> these schema definitions are independent of the type system (i.e. decoupled), so we can apply them selectively or not at all
These are just predicates; I don't think they have much to do with this discussion? They don't help you to type your data: you (the programmer) still have to carry that type information around in your head in order to work with the values even after they are validated.
> Could you do the same in Rust or other similar languages? Sure, it's ultimately just maps and predicates. But many languages aren't geared up to make that pleasant to use.
There are good reasons for that, though. Nominal typing allows you to express and enforce constraints that aren't necessarily enforced by the structure of the type, at the cost of some extra ceremony. And systems languages often want to be able to express types that cannot be treated this way: types in which there is no sensible way to ‘ignore’ a value. Clojure makes some things easier to write by forcing you to carry around a bunch of extra stuff (both runtime data and static semantics) with all your values; that's a perfectly valid choice that makes an important subset of programs nicer to write, but it also excludes values that don't fit into that category or can't reasonably be coupled to their RTTI.
As we get better at writing compilers we're increasingly seeing a move towards mechanisms like Rust's traits or C++'s concepts that allow the programmer to actually abstract over the representation of a type (as opposed to forcing a uniform representation as is common in dynamically typed languages). If you do that well you can get the best of both worlds semantically, but these kinds of systems languages will always be a bit heavier syntactically because the syntax needs to support a wider range of types.
In Clojure the data is thrown away too, just at a slightly later point. The data is parsed then destructured then processed, and it is at the destructuring stage that irrelevant data is discarded and marked for GC.
Just as the Rust function avoids coupling by using a more narrow type, the Clojure function avoids coupling by using a more narrow binding.
> I think this isn't a great comparison: the usual vehicle for this thing in Rust is the trait, which manages to abstract over the representation statically without introducing a bunch of runtime machinery for it
But you have to create and implement those traits. I'm not saying this is impossible in Rust; just a lot more onerous because you don't get all the machinery Clojure has that makes it trivial.
> These are just predicates; I don't think they have much to do with this discussion?
This might be why we're talking past each other. When I say "type", I mean a set of possible values. I don't think this is an uncommon definition; the first sentence of the "data type" Wikipedia page pretty much defines it the same way.
Given this definition, you perhaps see why we can narrow a runtime type by composing it with a predicate.
My understanding is that you view types differently, as more of an interpretation of some sequence of bits, rather than a set of data values. Is that correct?
> As we get better at writing compilers we're increasingly seeing a move towards mechanisms like Rust's traits or C++'s concepts that allow the programmer to actually abstract over the representation of a type (as opposed to forcing a uniform representation as is common in dynamically typed languages).
I agree that's there's no reason in principle that you couldn't statically type it with a sufficiently advanced compiler.
Right; the difference is that by the destructing time you have lost the type information about that ‘discarded’ data, meaning you no longer know how to discard it. In Clojure this is not a problem, because the only values you're allowed to have are those that can be garbage collected, and (furthermore) garbage collected by calling a known function with nothing more than a universal set of GC metadata that is attached at a standard location on every value. But this is only possible because of a large restriction Clojure places on allowable values: the base ‘unitype’ of all Clojure values already contains all of this extra data, and any values for which it can't be attached or it doesn't make sense are simply forbidden.
> But you have to create and implement those traits. I'm not saying this is impossible in Rust; just a lot more onerous because you don't get all the machinery Clojure has that makes it trivial.
On this I absolutely agree. The only thing I'd be careful to distinguish is not simply that Rust doesn't have the machinery but that the machinery makes sense only given a fairly onerous set of additional assumptions about your values that Rust doesn't make (and that are usually held to be incompatible with a systems programming language, but IMO are also helpful for many — though not all — ‘userland’ programs).
> When I say "type", I mean a set of possible values. I don't think this is an uncommon definition; the first sentence of the "data type" Wikipedia page pretty much defines it the same way.
It's not an uncommon one, indeed, and while there are more sophisticated definitions (e.g. including equality) I don't think they need to be invoked for this discussion.
> Given this definition, you perhaps see why we can narrow a runtime type by composing it with a predicate.
No, you're right: I think I was thrown off by your assertion that these are uncoupled from the type system when we were talking originally of ‘conceptual’ types.
> My understanding is that you view types differently, as more of an interpretation of some sequence of bits, rather than a set of data values. Is that correct?
Quite the opposite. These words aren't entirely standardized so I set out my definitions above:
- I am using ‘value’ to refer to any object of discussion in a programming language that nominally exists at runtime; effectively, a value can be defined by the set of operations available on it - I am using ‘data’ to refer to values that have ‘data-like’ semantics, i.e. has ‘default’ move and discard mechanism like bitwise copy/overwrite or GC
Yes, this is very true. Languages like Rust can certainly manage memory and data structures more precisely. Not only is Clojure garbage collected, by virtue of being based on the JVM and being dynamically typed, but in idiomatic usage it encourages data to be built from a small number of basic types.
> On this I absolutely agree. The only thing I'd be careful to distinguish is not simply that Rust doesn't have the machinery but that the machinery makes sense only given a fairly onerous set of additional assumptions about your values that Rust doesn't make
I guess only I'd disagree here that the additional assumptions are "fairly onerous" - but that's just an opinion. You may be doing work that butts into those assumptions far more often than I do (which is almost never).
> No, you're right: I think I was thrown off by your assertion that these are uncoupled from the type system when we were talking originally of ‘conceptual’ types.
I don't think I expressed myself very clearly, and indeed Clojure's type system isn't very clear either, since it's bolted onto the JVM and so can't entirely do its own thing. So we have "types" which refer to Java classes/interfaces, and "schema" or "specs" which refer to Clojure's own runtime validation ideas.
I originally meant this mostly in the sense that it forbids a lot of values. But I do also think this is a bit of a Blub situation. There's a good class of pure data algorithms where basically everything you want to interact with is data, and if you're working in that space it's pretty great. But having spent some time in resource-typed languages, for any situation where I have to work with external resources or effects that can't be undone working in a language that doesn't have resource types and deterministic constructors really feels like a big step backwards. There are so many invariants you can't capture if you can't reason about value invalidation.
But they are.
UNIX Pipes do not mandate having to use tons of different programs with stupid commandline options. I simulate them in ruby itself; method chaining works in a very similar way, but I built a pseudo pipe around it. The idea was more to have an object oriented shell, e. g. combine good ideas from UNIX pipes and the MS powershell.
They are simple if you design them well and have them be flexible too. The reason UNIX pipes were awkward is because they delegated onto many different programs such as awk or sed with their own strange rules. Nowhere does it say you HAVE to use such awkward tools. Use better tools and the idea of piping becomes simple, similar to (a more flexible variant of) method chaining. Just without being tied down to a specific object per se (I do use the pipe-handler master object to handle the pipe instructions; each pipe instruct I call cmdlet, e. g. shorter for commandlet, as this is how I like to think about this in terms. This also combines e. g. virtualdub + avisynth ideas. I loved them when I used windows. The idea behind avisynth is great - not necessarily all of the syntax, but the idea that all multimedia audio can be operated on at all times in flexible ways.)
Did you remember to:
- check for the other spawned process exit code?
- waitpid for all process in the other process chain?
- propagate/handle signals, like for example SIGINT/SIGTSTP/SIGPIPE/SIGHUP forward and back signals?
- change stdio buffering mode?
- remember to count how many bytes actually were written by write, and blocking if not, before clobbing the 64kb of the pipe buffer size with another write?
- flush, then close all file descriptors left behind by the pipes when it ends?
It's for reasons like that, that I don't trust anything non-trivial, not-shell to use pipelines correctly.
Expectedly, Rich Hickey explains it best, watch the "Simple Made Easy" talk if you haven't before, one of the few talks I probably watch bi-yearly: https://www.youtube.com/watch?v=SxdOUGdseq4
Few things, concepts and ideas have changed as much of my programming mind as Hickey's talk and ultimately Clojure have done over the years.
Wish we had new amazing Hickey talks to link to, maybe it seems he's about the hang up the hammock perhaps?
Copy pasted functions with subtle changes mean you cannot reuse the proof (DRY). Giant functions with lots of if/else statements however might cause a branch explosion in the proof. The right abstraction removes lots of assumptions that a proof could depend on, limiting the search space and often forcing elegance (this also applies to math, eg. when reasoning with abstract groups instead of integers). The wrong abstraction might force case distinctions on consumers of the abstraction.
It sounds like a reasonable concept, but then Principia Mathematica takes 300 pages to prove that 1+1 is 2.
This specific usage appears to come from this linked talk, Simple Made Easy:
https://www.youtube.com/watch?v=SxdOUGdseq4
My reaction to the Unix pipeline was that, the reason it exploded in complexity is because the pieces were too simple. They were insufficiently expressive.
But the word is used in a different way here, and I'll have to watch the talk to understand what exactly is meant. (Something like orthogonality?)
And it is important not to just make snappy quips by equivocating.
the reason it exploded in complexity is because the complexity is in the interaction of the parts, not the properties of the parts. It's not so much about expressiveness but about the fact that the Unix philosophy has a willful disregard for systems thinking.
Russ Ackhoff has the analogy of the house. How do you design rooms in a house? Only ever with the house in mind. You can't just hope the rooms work, you always need to check whether a new room improves the house, if somethings wrong you fix the room, not the house.
Unix tools have a disregard for the house. Clojure works well because every part of Clojure is designed not just to improve itself but to improve Clojure. Emacs is similar in that regard, you can see that when people make Emacs packages. The good ones always have Emacs in mind, not just their own functionality.
It is especially dangerous when something is "good" and people try to appropriate the term to appropriate the goodness of the term, as if goodness flows from a term to the thing it is attached to rather than the other way around. "Simple" is good so my good thing must be "simple" to be "good". But it doesn't. Simple can even be bad, in the wrong place or in the wrong sort of "simple" for a given job.
Because familiarity is a confound for intuition about complexity, even this is not always true.
Maxwell's equations will look complex to the uninitiated, and can represent the pinnacle of simplicity to those who already understand them.
"Few tokens" - perhaps the most literal simplicity, literally, it doesn't use many tokens to do the job. But as the article points out, that doesn't necessarily fit with...
"Easy to reuse" - This is that simplicity that functional programming aspires to, where you craft some precise abstraction that somehow captures something like "monad". Haskell is full of this sort of simplicity, oozing out of every pore, but people generally think of it as a very complex and hard langauge, contrasting...
"Easy to understand" - As in, not cognitively complex. It is amazing how quickly things that I would otherwise describe as very simple still blow out our little minds. Consider the first time you saw quicksort... or even how it feels now. It's not a lot of tokens, but it's twisty and recursive and especially if you're not mathematically trained and in practice it's easy to call it more "complicated" than a CRUD form that takes in and validates 10 parameters in a straightforward way, even though in terms of what is actually happening the CRUD form may be doing vastly more than the little quicksort algorithm. It just isn't being twisty, recursive, and subtle in how it does it.
And I'm just filling out the first three that come to mind. Note these are not always in conflict by any means... but they certainly aren't always in harmony with each other either.
(One might argue "easy to reuse" is more about the complexity of the code doing the reusing, but I feel like this is definitely something people mean when they talk about the simplicity of code.)
These concepts simplify for the user these questions: “I know what I want - how do I make the program do it?” (Execution) and “The program did something — what state is it actually in?” (Evaluation)
I believe the author was getting at these concepts, especially in their Google Drive example - how the large program has a “small” UX. Understanding the domain model provides a much stronger basis for designing user interfaces, and understanding the Gulf of Evaluation/Execution allows you to build incredibly complex-looking, large UX’s without confusing or overwhelming the user.
This is why I think the formalists studying complexity back in the 50s had the right approach. You can only give "simple" precise meaning within some kind of formal system with a shared set of initial axioms or assumptions. From that point you can define it quantitatively over some set of objects (relations, programs).
Funnily enough, this approach also touches on Hickey's etymological derivation. The root words also fundamentally have to do with the quantity of relationships.
(For what it’s worth, the point of both this essay and the aforementioned talk is that programs do not have to get complex, even when they get large, even when it gets hard because there are no more easy / close-at-hand / familiar helpers in the language to tackle the domain specifics. In support of this point I will note that Rich Hickey is the creator of Clojure, a language in which “building domain-specific helpers yourself” is very nearly idiomatic.)
https://news.ycombinator.com/item?id=49600275 is a great illustration of how to use Bash's "means of combination" more effectively than what OP thought of doing. But notice that it still relies on decorate-sort-undecorate for keyed sorting, and not only that but the undecoration has to be deferred until after `uniq` has made use of it, in a way that magically collapses the first decoration into something that can be used for a second sort, while adding a second decoration (the count). And then it still needs postprocessing because you can't express control over the structure of the decoration (there isn't really any, it's just string concatenation, but I mean you don't even control the order in which the pieces are concatenated).
A single regex line could be far more complex than a 100-line java program.
tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \
| tr A-Z a-z \
| nl \
| sort --key=2,2 --key=1,1n \
| uniq --skip-fields=1 --count \
| sort --key=2,2n \
| awk '{ print $3, $1 }'
(I originally posted the above as a comment on lobste.rs, on this same article.)
It is nevertheless "complecting": the uniq assumes the data is sorted and the columns of your data structure move together. Maybe this algorithm is already complex regardless of the implementation.
btw, this paradigm reminds me of APL
In contrast to the "we need a frequency table" idea in the article, this solution trades off memory by transferring all the line numbers in the stream. This is very much in the spirit of the infamous McIlroy/Knuth "bakeoff"[1] -- tradeoff some efficiency (via extra book-keeping or sorts) in return for composability.
Agreed, very neat.
[1] https://homepages.cwi.nl/~storm/teaching/reader/BentleyEtAl8...
In detail, for input "foo bar FOO qux FOO foo", we convert to
[1 foo, 2 bar, 3 foo, 4 qux, 5 foo, 6 foo]
(with newlines instead of commas, obviously), then sort by word (then line number) to
[2 bar, 1 foo, 3 foo, 5 foo, 6 foo, 4 qux]
at which point the uniq invocation gives <count> <first_line> <word>, i.e.
[1 2 bar, 4 1 foo, 1 4 qux]
albeit with an ugly mix of tabs and spaces. One final sort by <first_line> gives us
[4 1 foo, 1 2 bar, 1 4 qux]
and then it's just a matter of formatting the output:
[foo 4, bar 1, qux 1]
The generally-useful point is that the classic shell utilities really do work pretty well if you're operating within their paradigm, which isn't "throw everything in a hash table". (That's the paradigm of later scripting languages.)
grep -Eo '[A-Za-z]+' README.md \
| tr A-Z a-z \
| {
distinct=()
declare -A freqs
while read word; do
[[ -v freqs["$word"] ]] || distinct+=("$word")
((freqs["$word"]++))
done
for word in "${distinct[@]}"; do
echo "${freqs["$word"]}" "$word"
done
}
FWIW, I set hard limits to 200 LOC for every single source code file in any AI-related projects, also with restrictions on "formatting hacks" and other golf-like stuff.
I think beyond 5000 LOC in a single file and all available models already get lost frequently, even if the "context limit" theoretically is way above that. Maybe aim for like 1K LOC at max unless you want to have lots of misunderstandings.
I already preferred to keep source code files under about 200 LOC, so.
The clojure expression reads the entire file into memory first, and then operates on that memory representation.
The pipeline processes the input in chunks. The two `sort`s might read the entire contents into memory, but an implementation like GNU sort will instead, for large inputs, create temporary files for sections of the input and then merge sort them to the output.
You could make clojure do the same thing, but it might not be as "simple" as the pipeline.
> Now, let's say we want to make a small change: show the output in the original file order. In Clojure, this is fairly straightforward: store an ordered sequence of the words in word_seq, store a map from each word to its frequency in freq_map, iterate over the sequence, and look up each word in the map:
Emphasis on just "storing" something. The author seems to not understand that this change makes the solutions categorically different.
It's a little like critiquing the efficiency of moving a piano up a stairwell by saying why didn't they just use a crane via the window? Using a crane isn't a better way of getting a piano up the stairwell.
> That's because our original program was small but not simple.
I would say no - because it was inflexible.
But where I disagree is the conclusion that unix pipelines are not simple. IMHO unix pipelines as a platform are incredibly simple and powerful, allowing for solving a massive range of small problems much more elegantly than any general purpose programming language. Obviously the constraints that enable this simplicity at the low-end, are real tradeoffs that prevent simplicity at the high-end. But one of the core principles of effective engineering is do the minimum to solve the problem at hand, no more, no less.
I like this question. Some projects will sacrifice usefulness in the name of simplicity.
Logic that the blog presents is sound; but sometimes small works as a good-enough proxy to get to the simplicity that we find elegant.
< README.md \
tr -c '[:alpha:]' '\n' \
| tr '[:upper:]' '[:lower:]' \
| awk '
NF {
if (!($0 in count)) order[++n] = $0
count[$0]++
}
END {
for (i = 1; i <= n; i++) {
print count[order[i]], order[i]
}
}
'
Yes, associative arrays work well. I think it should even be possible to use bash associative arrays. But at that point you're no longer doing classic sh - awk is basically halfway to Perl. (And pretty awesome.)
Without processes shell is just a weird sad little language, with them it turns into this epic data flow language. With some real design stinkers, Most of these are due to it's interactive first focus, The features desirable for interactive use, often start to stink for stored program use, I will note that having the same language for interactive and scripting is pretty kick ass.
On the subject of dataflow languages has there been any research in this area? Something that can stitch together processes as well or better than shell? Perl may work in this role but I have to admit I really dislike it's syntax, and as such I never learned Perl enough to love it. and while most other scripting languages can technicaly create pipelines, it is very awkward compared to shell.
I've been working on a design for a Python library to facilitate this sort of thing.
{
$0 = tolower($0)
gsub(/[^[:alpha:]]/, "\n")
for (i = 1; i <= NF; i++) {
if (!($i in freq)) order[++n] = $i
freq[$i]++
}
} END {
for (i = 1; i <= n; i++)
printf "%d %s\n", freq[order[i]], order[i]
}I don't see how the example would benefit at all from having that information at run-time when it exists at compile-time. The problem with Rust specifically I think is that it's a relative PITA to get to the information that exists at compile-time in this case. In Zig, you'd just iterate over the fields using the same language you use for run-time code.
const fields = std.meta.fields(@TypeOf(instance));
inline for (fields) |field| {
const name = field.name;
const value = @field(instance, name);
std.debug.print("{s}: {any}", .{name, value};
}
Somehow when things get complex, I could never find fully functional style to be more understandable than imperative
1. "You need to have taste (so: experience) to do DRY right". Same chunk of code more than once, so natural/expected behavior is to export to a helper... Aaaand the now introduced coupling is (too) often ignored, as in "no thought given whatsoever".
2. "Sometimes it's better to just leave it as is". 95% to 98% of "same chunk of code" in a number of places. The temptation to DRY is strong, yet the "numerically mere 2 to 5 pp" make the extracted helper an exercise in all manners of gymnastics. The only correct answer is: do not start.
(Own experience; your mileage may vary - if it does, feel free to comment back!)
The Unix pipeline and the similar-looking Clojure expression are entirely different in how they do what they do. Pipes are process abstractions. The (perniciously improperly understood) "pipeline" macro is not pipelining anything in any way (not CPU nor process nor memory). It is merely syntax sugar to write a deeply nested call chain as a "flat" list of operations.
And it goes on to compare "ugly" code, again making the category error that "more" code is "worse". Because, one is swapping / splicing entire programs within a pipeline. Also, there are plenty of ways to slice that mango; you don't need temp files.
That said, having intermediate files in one's pipeline is a big help because one can use those to make pipelines idempotent. Plus, one doesn't need flat files, one could swap in a SQLite cache too, at will, without modifying anything else in the pipeline. This kind of design change is not possible with the equivalent Clojure function call chain, as-is. In fact, having such a requirement (one always finds need to restart processes after crashes, and have them pick up from where they died) causes us to write some custom (and therefore design-wise brittle) conditional restart loop on top of the computation to manage its failures inside the running program. An idempotent Unix pipeline simply needs to be... restarted from outside the process.
In this particular case, a valid complexity complaint would be about the lacunae of the Bash / shell programming language (and interpreter model) versus the Clojure language.
Also, again from a program design point of view, I feel there's a bit of "holding it wrong" going on there... Bash / Shell-fu, yes, but not enough to be dangerous (for example, sort | uniq | sort is a standard idiom of pipeline programming). Which claim is personal, and so I'm open to being corrected at the same level. Sources: my code and writing:
https://www.evalapply.org/tags/bash/
https://www.evalapply.org/tags/clojure/
https://github.com/adityaathalye (the pinned repos are Bash and Clojure)
And specifically, this log processing code, for a more apples-to-apples comparison with OP's post.
https://github.com/adityaathalye/bash-toolkit/blob/master/lo...
deduplicate() {}
frequencies() {}
drop_first_n() {}
drop_last_n() {}
drop_header_footer() {
drop_first_n "${1}" |
drop_last_n "${2}"
}
window_from_to_lines() {}
Taking 9 months to fix a bug sounds alarming to me.