Hacker News Reader

> tackling more complex problems

This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.

However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.

> small team of engineers

It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.

As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.

> Frameworks

One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.

> As teams get bigger, communication sales factorially,

That sounds a bit drastic. Surely at worst the scaling is quadratic?

You are correct, but quadratic is sufficiently bad.

Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...

The number of channels is exponential, that's just Metcalfe's law.

The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.

In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.

Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.

And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.

The bigger the team, the bigger the software, the more inevitable the collapse.

> The number of channels is exponential, that's just Metcalfe's law.

Metcalfe's law is quadratic rather than exponential -- O(n^2), not O(2^n). And if n people all need to be aware of communication happening between every pair of them (which is a worst-case), then that should just add a factor of n, bringing it up to cubic.

(On an unrelated and less pedantically nitpicky note, one of the most valuable professional skills I've developed is an ability and willingness to dive into black boxes. It's remarkable how many bugs arise from the interaction of two or more pieces of software with no overlap between their developers. Debugging those can involve a long, agonizing back-and-forth between people who know how X works and people who know how Y works -- or it can be over in a jiffy if you can quickly familiarize yourself with the basic internal workings of both. Past a certain point nobody can know everything, it's true, but this doesn't need to be as crippling for productivity as a lot of people allow it to be.)

One of the surprises I got when doing that was realizing that Y's code was terrible because of how X was interacting with it, and a rewrite of X allowed a rewrite of Y, and I was able to delete half the code in each. This has occurred countless times.

But that doesn't stop black-boxes from being created faster than I can fix them, unfortunately.

This seems like a fantasy considering how big the development teams for these projects are.

Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.

It was also already built. The clone is easier to make

The cumulative system design contributing to MULTICS, however, probably was a order of magnitude greater work than the implementation effort. On the shoulders of Giants may be clichèd , but it also seems increasingly as if the contemporary world has decided to deprecate inherited wisdom. Blame awful policy decisions starting under Regan..

Oh, you're right. I had in my head "A combination of MIT, Bell Labs and GE" and manually translated that to thousands but it was actually only about a hundred dedicated people.

Thanks.

Although it may have only taken a week to create the initial version of Git the amount of polish and documentation required to get it to something someone like me who's not a kernel developer was a lot more than a week's worth of work. You may be able to get 80% of the functionality in 20% of the time but it's that last 20% of functionality that gives software enough polish to actually be used by someone who doesn't have an incredibly specific reason to figure out how to use the unpolished software.

I thought Linus based Linux off of Minix, but used a monolithic kernel to keep things practical.

I think Tudor Girba has a similar vision. E.g. check out https://gtoolkit.com/ and his talks on software environmentalism, such as https://m.youtube.com/watch?v=N3l3eB62oSw

This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc.

All still manageable by humans but at continually higher and higher levels of complexity.

I sometimes wonder if we need to go back to the drawing board or model our problems differently lol.

yeah I would bet money there's better ways of doing things.

you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?

> If history is a guide, then you are correct. But I believe complexity is fractal

Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.

"Fractal" is used to describe the property of not resolving the complexity detail until you get up close ("zoom into" the complexity). The amount of use cases does expand alarmingly, but even combinatorial and exponential use cases imply they are inventoried and are thus amenable to prioritization.

In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.

The complexities at different levels of the hierarchy are so qualitatively different that I doubt whether a unified set of abstractions could ever be practically useful. We're going all the way from CPU microcode to loosely-coupled distributed systems with multiple dependencies on third-party services. If you want to make any progress you'll need to narrow down your focus.

You seem like a fun person to work with