Glue: The Dark Matter of Software
blog.metaobject.com
blog.metaobject.com
* contact cement
* superglue
* mastic
* wood glue
* gorilla glue
* liquid nails
* various caulks
As with construction and art, so with software. The Unix pipe is powerful because it enforced "the only thing you can connect together is line-oriented byte streams". But when you are building something where that's not the substrates in question, it's the wrong tool, and you need a different kind of glue.
Expand that to all the possible different things you may wish/need to "glue" together in software, and it becomes fairly apparent why there's a huge diversity in the glue being used, and why dreaming of a world in which you get by with just the software analog of gorilla glue is the wrong thing to do.
And yes, in order for this to work, we need lots of different kinds of glue: https://blog.metaobject.com/2019/02/why-architecture-oriente...
However, for most programming tasks, we have one kind of glue (with minor sub-variants): the procedure call. And so instead of having something that snaps together with a bit of glue, we have to write enormous amounts of boiler-plate in order to make the pieces fit. (The analogy only goes so far...)
Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed by design, language or other constraints. So, sure, stuffing a bunch of stuff on the stack, jumping to a new address and doing something with whatever is on the stack is, in some sense, a sort of glue.
But your TFA seems to be talking about a much higher level of glue than this, I think. The sort of glue that's represented by things like (random list): protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models. Questions like "I've got this customer data, I need to give it to some code that will munge it in some way... how do I do that?"
In that sense, procedure calls are low level mechanism that will support a higher level solution. Again, if it's not all just inline execution of machine code, then at some point, there's going to be a procedure call no matter what abstraction level you're working at.
> I'm been a compiled-language programmer for 35 or so years.
40 or so for me. :-)
> I perceive [procedure calls] as a fundamental mechanism for organizing code.
Yes, I've been arguing this for some time: that the call/return architectural style has been so incredible dominant for so long that it is not really seen as a specific architectural style, but instead as simply all there is. It is the mechanism for organising code.
A paradigm in the true sense.
I call this The Gentle Tyranny of Call/Return[1]. "Gentle" because as architectural style go, it's probably one of the better ones to have, particularly if you only get one.
Anyway, if you study software architecture, you will find that procedure calls[2] are just one of many ways to organise software. However, they are given special status, because they have special language support. So if you use procedure calls to organise your code, to glue your components together, you get to write code naturally. If not, it starts to look messy. And it turns out that procedure calls not always ideal, because...
> Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed
And a lot of what we do today is moving data around. Computers don't really compute all that much[3].
>But your TFA seems to be talking about a much higher level of glue than this, I think.. protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models
Yes, these are also glue. But so are procedure calls.
> In that sense, procedure calls are low level mechanism that will support a higher level solution.
One might even say they are the "Assembly Language" for connecting: [2]
[1] https://2020.programming-conference.org/details/salon-2020-p...
[2] https://resources.sei.cmu.edu/library/asset-view.cfm?assetid...
[3] https://www.youtube.com/watch?t=278&v=EKWGGDXe5MA&feature=yo...
On that topic, I am obligated to link to a wonderful no-nonsense guide choosing on the best glue for your particular materials:
To continue your analogy, if 95% of your house were made of glue, and 85% of the cost of your house was to buy glue, then that would be some cause for concern. That is pretty close to the truth for software.
Is this a general problem related to how we build interfaces for our implementations? Or is this more a problem with particular kinds of module systems? (Or both)
If every interface is bespoke while also being explicitly described top down, then we end up with this quadratic complexity.
Uniform, extensible interfaces and generic data structures are the solutions to this, as well as shifting complexity to data and away from code. Several modern languages, techniques and protocols address this in varying degrees and from different angles. An old language that does this very well is SQL.
Examples? (Asking as a curious beginner who doesn't know where to start.)
These are all attempts to enable generic, simple code over complex data structures, while respecting and incorporating extensibility and change.
It’s an old idea but I think more importance is given to it in more recent years.
So for example, if you want a build a simple REST api for your data model, you have glue code to read/write the data and transfer it to/from DAO's (Data access objects). Then you have glue code to transfer the data into to/from object model you expose via the API. This feels like 90% of server side CRUD app programming these days.
It's not quite boilerplate because it's a mix of ORM optimizations for your data model and data massaging to make it more digestable to end users of your API.
a) App frontend (UI logic) is compartmentalized from the app backend. Communication is handled by an async messaging system, requiring bundling up messages in both directions.
b) The app backend talks to a few different servers, each with their on expected message set. glue glue glue.
c) The app backend talks to the app database, inserting/altering info, and extracting database rows into objects which can be passed around.
d) (We also had peer-to-peer communication over local wifi, which required its own massive pile of communication code...)
e) Even different objects/methods in the backend code end up requiring small amounts of glue code to arrange+prepare argument sets to call one another.
So, every time we create a division between two pieces of code, we introduce boundaries, which introduces glue.
---
One way to mitigate the complexity is by making the passed messages first class. In our app, there was some 'AppThing' object which represented an instance of the fundamental thing the app was about. You then stuff that object with every piece of info about AppThings you could ever want, and pass it around freely between components. Invariably it gets kinda bloated and fractures, when someone doesn't want to bother building up the WHOLE thing with the necessary DB and server calls to ensure the data is all accurately filled in, when you just want to pass one or two fields to a neighboring UI element. And then you find yourself with lots of logic checking that certain fields are present amongst the million fields in your universal object...
Another approach is to aggressively prune the graph of possible component interactions. Functionality should be well-encapsulated, with absolutely minimal APIs for external interactions. Actually draw the 'cell membranes' between different parts of the app, map out the interactions between cells, and find ways to cut it back.
It feels very applicable to the functional vs imperative programming debate, and I say that as someone who's mostly on the imperative side. The top row of the "perimeter" is very "functional".
It's interesting to note that people from the same school of thought as those who built Unix (or even the same actual people) ended up creating Golang, which is anything but a functional language.
The glue code turns out inelegant, spaghetti-like, hard to work with. So you spend a little bit of time cleaning it up, abstracting reusable things out, and generally systematizing things.
Now you have three systems, so you need to write two layers of glue code to make them work together…
This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]).
A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things:
1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane).
2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that.
Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature.
So mathematically you could say (where each term is the amount of work required for one new feature + integrating it with the operating system): IBM/Multics: (0 features_before + 1 new_feature) + (1 features_before + 1 new_feature) + (2 features_before + 1 new_feature) + (3 features_before + 1 new_feature) + (4 features_before + 1 new_feature) + (5 features_before + 1 new_feature) + (6 features_before + 1 new_feature) ... + (n_features_before + 1 new_feature) --> simplifies to my favorite formula which is: n(n−1)/2 --> you can also see it as --> 0.5n^2 - 0.5n --> So, roughly this is n^2.
Just count the amount of numbers, it comes down to: 28 units at the 7th term/timestep.
Unix/2 garage guys: 1 new_feature + 1 new_feature + 1 new_feature + 1 new_feature --> simplifies to n_new_features
At term/timestep, you'd get 7.
At the 100th feature, you'd literally 100^2 amount of coding work at IBM whereas at the Unix place you'd simply need 1 unit ...
This isn't 80/20 or 99/1. This is n^2/n (meaning: what one needs n^2 time for, the other could do in n time).
[0] https://en.wikipedia.org/wiki/Pareto_principle
[1] https://www.youtube.com/watch?v=3Ea3pkTCYx4&ab_channel=FOAM
but frankly i don't see how its _ever_ a good idea to structure yourself around closed worlds unless the seller really has something that novel and indispensable.
I think this the most important problem in software development to be given the least amount of attention, or even awareness. Another new MVC framework, build tool, or even programming language, isn't going to solve this problem. First we need to acknowledge the problem.