A sketch of the biggest idea in software architecture | Hacker News Reader

A sketch of the biggest idea in software architecture

oilshell.org

113 pointsharperlee31 comments

The text tries to introduce the idea of a “narrow waist” as an architecture pattern. But I don’t fully understand how the author distinguishes this concept from similar ones, and some of the examples even seem to contradict the principles.

It supposedly solves the N*M problem (so it’s an interface?), but it can’t be statically typed (so the interface has to be implicit?), and success stories are HTML5 and the x86 ABI (so it does require explicit specification after all, and huge amounts of it? Just look at the size of that Intel ABI book…), but also, it must be plaintext (so not like x86 at all?).

I’m not trying to be mean because maybe there’s something there that I miss, but frankly, I don’t get it, at least not yet.

He's not saying it can't be statically typed. He's saying it's type specificity is very coarse grained. A stream of bytes has a well defined static type. But it's a very coarse grained one. As a method of communication a stream of bytes can express or contain almost anything. It's narrow in that it's type does not prevent any specific compositions or expressions of a wide range of things.

Similarly plaintext is an example not a requirement. Like any analogy it breaks down at the extremes but that doesn't mean the analogy is a bad one.

Yeah I think it's a bit of a stretch to try to combine all these ideas under a single name.

One guideline I would follow is:

Statically type everything that a service needs to inspect to do its job, and no more, but without introducing a layering violation.

So, for example if a component accepts this input:

    {
        "timestamp": "2022-06-17 00:00:00",
        "data": {
            ...
        }
    }

If it only uses "timestamp" to decide whether to discard the input, and then passes "data" to some other service, you would definitely want to make timestamp be a "datetime" type rather than a string. "data" should be an arbitrary JSON value, but it should still be parsed as JSON, just no additional structure enforced.

I wish real projects were so simple!

Guidelines like this are operating on something like the Platonic ideal shape of problems. In any real world project, you need to consider the cost of handing a malformed `data` blob downstream, as it's never free.

Architectural model or pattern might be a better word than guideline. You benefit by collecting a catalog of patterns more than you do by trying to internalize guidelines.

What is the N*M problem? I can't find anything on google (except a scientific figure behind a paywall).

If you have N things producing data, and M things consuming data, you have a problem where each of the M consumers needs to handle all N types of input data, meaning you have to implement N*M things. But if you have a "narrow waste", meaning a common interface like JSON that all N things are capable of producing and all M things are capable of consuming, then you only need to implement N+M things (N things that produce JSON, and M things that consume JSON).

But you still have to do some validation and presumably some type checking.

So this redistributes the problem instead of solving it.

JSON is a file and interchange format - more technically a protocol.

It is not a data type.

> JSON is a file and interchange format - more technically a protocol.

> It is not a data type.

Although me, and I'm sure many others, have abused JSON and used it as a data type, I totally agree with you. However, GP was using JSON as an example of a more general idea. I needed to aggregate similar data across many different tables holding information collected from web scraping - typically one table corresponds to a particular service, and each service has it's own spider. My approach was to define a data model that contains all the information that I care about then write classes per service to convert its data to the format of the aforementioned data model. While the model can be serialized as JSON, it's composed of standard Python data types/classes and I don't constrain its design to be more amicable to serialization. Point being, while JSON is an intuitive example and can be a good go-to if your data matches it well, there's many ways to solve the N*M problem in this particular context.

Heh it’s narrow waist (think thin point) but I like your term better.

An example of a narrow waist would be CSV - everything can export it, most things can import it, and you can usually fix whatever errors were introduced.

It also sucks bigtime.

Best example I know of:

You have M languages, and N computer architectures. You want a compiler from every language to every architecture, but don't want to write M*N compilers.

One solution is to make a virtual machine. Now, you write M language-to-VM compilers, and N VM-to-cpu compilers, so you only have to write M+N compilers.

For example - if you wrote a new language that targeted the JVM it’d “out of the box” work on almost all systems.

The reality is of course more complicated but the idea is there.

thats like webassembly these days :D

Before Unix, there were multiple types of files in an OS, Text files (sequential access), Random access, Random Access - Indexed, Databases, etc... and the OS required you to use a different library to access each type of file.

So if you wanted to be able to do N things (grep, word count, sort) across these M file types... you had to write N*M functions, each a bit different.

Unix simplified this to having everything as a text file (a special case of a file of bytes). This made it possible to have a set of N tools that could work on any part of the system, instead of 1/Mth of it.

The pipe operation allowed tools to be composed, you could grep, then word count, etc. This greatly increased the amount of computation that could be expressed as a task straight from the command shell, without having to write new tools.

The post works hard to normalize the term "Narrow Waist" to the point that's it's burdensome. Using nondescriptive terms to define a wide range of services, languages, and protocols only alienates clients and serves to differentiate academics from the rest in the development talent pool. Development over time should become more relatable over time, not more academic unless we're talking about the machine level perhaps... It's why employers are having a really hard time trying to find angular, python, C, and rust devs despite PHP and .NET still dominating in most web based apps and sites.

It would be interesting to know exactly how many no/low-code solutions are deployed in comparison to custom code solutions, and what type of business cases they serve currently... I think the results of that study would really make a lot of devs switch fields of expertise. Almost every employer is calling me about DevSecOps positions, while app design and development are an afterthought due to prevalence of low/no-code solutions because langs have become so convoluted and specialized in hopes of creating a monopolistic work dependency... Sorry to vent, but solutions simply are overcomplicated too often now, and most of the people making technical decisions are non technical, and not equipped to keep up with the ever-changing terminology and jargon used in the dev world, much less knowing the differences in languages and terminology.

Over complexity and tech jargon are some of the most painfully alienating detriments to development in my experience. I think we would do a lot of good if we really got back to breaking down use cases and their individual (right-fit) solutions rather than sticking to redefining entire disciplines and forcing new terms that aren't properly descriptive frameworks, tools, and methods onto the industry as if one great new shoe fits all feet... You know, simpler jargon is better.

I don't really know what I'm supposed to derive from this style of "blog post". It feels like the author wrote down a bunch of bullet points over the last 6 months of software development on Oil and just decided to publish them. There are lots of good links and cool trinkets (like projecting json onto a filesystem), and I'm now, having read between the lines, familiar with the term "narrow waist", but I'm not convinced of anything in particular because there's not a strongly argued thesis. There are a bunch of small assertions, some contradictory. I really struggled to get a "sketch" of.. anything.

Describing Win32 as a "narrow waist" (hundreds of interfaces, thousands of calls, no idea how many structures) seems off the mark.

I don't get it. It seems like the target got painted where the arrow landed.

> target got painted where the arrow landed.

Commenting mainly to say that I like this phrase very very much. Creative, Witty and at the same time so descriptive.

On the win32 point, it is adressed in the article:

> I also realized that there are two distinct senses of the word "narrow":

> - In terms of architectural connection (topology). For example, applications and physical networks are decoupled by the the Internet's narrow waist. They don't interface directly with each other.

> - In terms of the size of the concept. <snip> But the web is a large set of concepts (HTTP, HTML, SVG, etc.).

Somewhat related is the Lispy approach of using generic data structures (e.g. cons lists); compared to the OOP approach of defining new (hierachies of) datastructures/types/interfaces (usually classes) for each use-case.

With the former, there's an ever-growing set of functions we call on our data; whilst the latter is incompatible with all existing code.

having a common syntax gets you a crude kind of type-generic behavior. you can can write functions that just deal with 'stuff', including functions to merge data and to serialize/unseriaze. maybe you also get a base type system. maybe include some transport functions like idempotency...write a little data viewer.

you can build stuff on top of this - including additional semantics, like adding a schema with the same underlying representation in order to do validation. or transactions - nothing to say say that the body can't be another one of these generic data items wrapped up in a transaction envelope - or signed like a cert.

I dislike 'narrow waist', although I understand the image. it seems to me to really be about what everything always has been - separation of concerns and low level functions that support reuse.

This is an excellent overview of the narrow waist concept which is a highly zoomed out. I think it's hard to see sometimes because when you are working within a narrow waist you are typically working on something that looks more like a series of filters in one or both ends of the hour glass. The fundamental compromise that the narrow waist is much more general than the specific domain you are working in means it fades to the background and instead you are working to handle all the ways that too general a space needs to be specialized.

It seems like a restatement of the end-to-end principle. He does call it out as being different but… still seems pretty similar to me. Build apps on simple reusable abstractions like byte streams and message passing.

Small pieces, loosely coupled.

Like a glass jar that has been dropped onto a stone floor.

Or like the coins that were in the jar, whose lid was stuck. Before, they were a large mass of tightly coupled metallurgy. Now, they're actually useful.

Note for laypeople like me the glossary accompanying the article is really super.

ELI5? The ideas are interesting but I need help understanding.

  https://news.ycombinator.com/item?id=31778505

I was hoping for a ... sketch

Me too. This is a jumbled stream of conscious style mess of information, even if the topic is interesting.

Another obvious narrow waist is money. Language might be one, especially English today.

language is one! Thinking of that though, makes me lose the barrier between narrow waist and any interface whatsoever. All of them become a narrow waist all of a sudden.

Reply on news.ycombinator.com