Hacker News Reader

Comment by daenz | Hacker News Reader

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coprocessor" assist in seeing the solutions. I believe there are well-known strategies (taken from the real world) for managing hierarchies of complexity that ease the cognitive load of exploring any problem. We just need to get these approaches into a programming "langugage"!

View on HN

> I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers.

Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle.

So in that view, you can't succeed at making software engineering always simple. Instead, you can make previously intractable problems tractable.

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation.

Frameworks approximately do this, but for specific domains: they let you organize complexity into well-defined areas and put a pin in them, easing the cognitive load. Then you can handle those abstractions more easily. But yes, to your point, because frameworks typically only tackle a few levels of complexity, you still get that complexity back when you use the framework to advance the problems to the edge of what your framework is designed to address.

A recursive/fractal management of complexity will allow all levels of the hierarchy to feel similar, so you are never increasing complexity, only looking at a different resolution. I think the key to this is mapping out the fundamental organizational problems and how they relate to each other at different resolutions.

> tackling more complex problems

This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.

However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.

> small team of engineers

It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.

As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.

> Frameworks

One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.

> As teams get bigger, communication sales factorially,

That sounds a bit drastic. Surely at worst the scaling is quadratic?

You are correct, but quadratic is sufficiently bad.

Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...

The number of channels is exponential, that's just Metcalfe's law.

The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.

In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.

Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.

And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.

The bigger the team, the bigger the software, the more inevitable the collapse.

> The number of channels is exponential, that's just Metcalfe's law.

Metcalfe's law is quadratic rather than exponential -- O(n^2), not O(2^n). And if n people all need to be aware of communication happening between every pair of them (which is a worst-case), then that should just add a factor of n, bringing it up to cubic.

(On an unrelated and less pedantically nitpicky note, one of the most valuable professional skills I've developed is an ability and willingness to dive into black boxes. It's remarkable how many bugs arise from the interaction of two or more pieces of software with no overlap between their developers. Debugging those can involve a long, agonizing back-and-forth between people who know how X works and people who know how Y works -- or it can be over in a jiffy if you can quickly familiarize yourself with the basic internal workings of both. Past a certain point nobody can know everything, it's true, but this doesn't need to be as crippling for productivity as a lot of people allow it to be.)

One of the surprises I got when doing that was realizing that Y's code was terrible because of how X was interacting with it, and a rewrite of X allowed a rewrite of Y, and I was able to delete half the code in each. This has occurred countless times.

But that doesn't stop black-boxes from being created faster than I can fix them, unfortunately.

This seems like a fantasy considering how big the development teams for these projects are.

Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.

It was also already built. The clone is easier to make

The cumulative system design contributing to MULTICS, however, probably was a order of magnitude greater work than the implementation effort. On the shoulders of Giants may be clichèd , but it also seems increasingly as if the contemporary world has decided to deprecate inherited wisdom. Blame awful policy decisions starting under Regan..

Oh, you're right. I had in my head "A combination of MIT, Bell Labs and GE" and manually translated that to thousands but it was actually only about a hundred dedicated people.

Thanks.

Although it may have only taken a week to create the initial version of Git the amount of polish and documentation required to get it to something someone like me who's not a kernel developer was a lot more than a week's worth of work. You may be able to get 80% of the functionality in 20% of the time but it's that last 20% of functionality that gives software enough polish to actually be used by someone who doesn't have an incredibly specific reason to figure out how to use the unpolished software.

I thought Linus based Linux off of Minix, but used a monolithic kernel to keep things practical.

The complexities at different levels of the hierarchy are so qualitatively different that I doubt whether a unified set of abstractions could ever be practically useful. We're going all the way from CPU microcode to loosely-coupled distributed systems with multiple dependencies on third-party services. If you want to make any progress you'll need to narrow down your focus.

> If history is a guide, then you are correct. But I believe complexity is fractal

Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.

"Fractal" is used to describe the property of not resolving the complexity detail until you get up close ("zoom into" the complexity). The amount of use cases does expand alarmingly, but even combinatorial and exponential use cases imply they are inventoried and are thus amenable to prioritization.

In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.

I think Tudor Girba has a similar vision. E.g. check out https://gtoolkit.com/ and his talks on software environmentalism, such as https://m.youtube.com/watch?v=N3l3eB62oSw

This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc.

All still manageable by humans but at continually higher and higher levels of complexity.

I sometimes wonder if we need to go back to the drawing board or model our problems differently lol.

yeah I would bet money there's better ways of doing things.

you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?

You seem like a fun person to work with

This is true of almost everything. The inputs / resources dedicated are relatively fixed while the outputs are dependant.

I think you’re both right.

Well no to be honest Spring Boot/Java really does make managing threads and web requests easier than doing it manually.

Similarly using an operating system/assembly is much easier than trying to control pulses of electricity yourself!

I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down.

Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.

Another example: we model physical cables (both for power grid and for data transmission) in our CMDB. All works fine, until you suddenly have a Y-shaped cable with three connectors that doesn't fit into your data model. The real world always has these 1% of cases that don't fit the general pattern; if you focus on the 99%, the 1% make trouble. If you focus on modeling every case, you have 10x the complexity, even for the simple case.

And then there are things that are moderately complex and security critical, like password recovery workflows. We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.

If somebody has a good idea for how to tackle these problems, please let me know!

As you implied, I don't think the answer is to make the perfect abstractions that can handle any scenario.

But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario.

The amount of duplication solving the same problems is insane. But this is not an easy problem to solve and I don't intend to trivialize it.

I think we need to come up with better tools. code sharing through libraries/repositories (like npm) is great, but it can't be the final solution.

Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. You could import modules, functions, data structures, anything. But that vision is still a long ways off.

The problem is "pulling in a tried and tested solution" means pulling in a slice of the whole stack: CSS, Javascript, client/server communication, API endpoints, data-flow logic, and database schema. People are very very bad at this because they believe that software engineering principles apply only to little tiny portions of this stack. If you actually apply your software engineering principles to the whole stack, you end up with something that looks very different than what everyone else is doing (so there's not a lot of tooling to help with). I have a better way to do this than anything else I've ever seen, but even with that I struggly very strongly with the JS/API side.

Kinda reminds me of what they do in some areas of the finance world with Kx System's Kdb+. That entire install is tiny tiny and you get a high performance array language for querying your SSD database with either the main language "k" or a syntactical sugar dialect sitting on top called "q" that is more like SQL.

I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.

KDB is unique for translating "in house" tech to marketable CV entry status. Having Haskell and CS(FB) experience requires educating the recruitment agencies, for comparable recognition. Personally I consider every ill in computing as symptoms of the intellectual property system, from making reading potential prior art (aka learning) verboten to non disclosures making the Peter Principle look like the common cold to covid.

I had trouble following your message. You've stated that Kdb+ looks good on a resume, Haskell requires explanation, and IP is bad? I'm not a big fan of closed source myself (and Kx Systems sounds like it would have a lot of red tape), but the technology is supposedly very fast and I find the language to be very simple and charming. I'd love for our analytics database to migrate , but I think the technology would really confuse our IT folks and it would be difficult to describe why it would make our jobs easier.

Get in touch! I work within KX as a Solutions Architect and would love to see how we could help you out. [redacted]

Will do at some point. Thanks for the invite!

Anytime! FYI we've released a new product KX Insights which is a cloud native version of kdb+ that supports ANSI-SQL (PostgreSQL) and has a bunch of ease-of-use and interoperability improvements to open the technology to a much wider user base :)

> Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service.

The type signature of GPT-3 is `string -> string`

Well I'd argue it's more Model => String => String if you are talking about using a fully pre-trained model. Even then you could probably expand on the type signature of the model to make it more useful.

But if you look at like co-pilot for example, if that was given the ability to have type signatures serve as input you might get a lot more powerful results than what it does with raw text (which is very impressive).

But this comes down to type signature design. You can encode any function using simple types like int -> int which aren't very useful. Where Haskell shines is when using types to limit the scope of inputs & outputs. What I am getting at is that you can still write uninformative type signatures in Haskell, but it also gives you the power to write more informative ones.

I don't think Haskell is the answer, so please don't take that as what I am saying. I do think however using richer type systems could be a stepping stone towards a solution to this problem.

Type signatures (or ASTs) are structured data, and Copilot mainly works because GPT accepts any text and doesn't have to deal with your logical fully recursive concepts like ASTs.

I feel like it wouldn't work if you could actually mathematically constrain the outputs to be syntactically correct either. That's probably one of those Gödel things.

> Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service.

Type signatures don’t tell you which should be the “then” clause vs the “else” clause in any conditional.

> Names

Don't try to force schemas onto schema-less data. Store the "name" as a JSON string/blob representing the various possible attributes (given, middle, family, title, etc) and provide a variety of functions for representing that data. IF you really need to do this at all (for an internal app, you probably don't).

> Physical links

Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.

> Password recovery

What is so complicated about this, specifically? Python and Ruby are completely different languages with no guarantees for interop.

Your solution assumes atleast a dozen incorrect things from the classic https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

That's strange, because my one (1) assumption is that the name can be represented as a schema-less dictionary of zero or more Unicode strings. Heck, one of those strings could be an ID for a bitmap or SVG of their non-textual "name".

If you can't do that, you have more pressing issues.

Where are you getting "a dozen assumptions" from?

> Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.

Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example).

So if you model a Y cable as two physical cables, you have to both allow duplicate cable IDs AND multiple cables per port. Both of these have significant downsides in preventing data entry errors.

Can't you just have 2 tables? One for 2 end and one for 3 ends?

Or just have a cable type column (which could be a pointer/foreign key back to a table of cable types, so you can add the cable with 4 ends that comes out next year).

Yeah that sounds even better and what I typically see in production databases

> So if you model a Y cable as two physical cables

No. Model it as two abstract "Physical Links" or "Connections" which FK to the same physical cable Asset, which is uniquely determined by its inventory ID and/or manufacturer model+serial.

If a Link needs to know whether its part of a Y cable Asset, it can join the Asset table.

If a Y cable Asset needs to know what it's connected to, it can join the Link table.

Additionally, you have a nice clean Link table for doing graph traversal (get me everything connected to this impacted Asset by degree <= 2), and a nice clean Asset table for inventory tracking.

I could be wrong but using JSON as format for name is not a solution, its abstraction over possible solution. Suppose we store all names as JSON structure. Question - how people will create such structure from their names? Using form with dozens of fields and checkboxes? Second question: how we will display this structure? We have to find a way to convert this JSON into string to show name. And as result we will surely need function to convert this shown strings into JSON for sure. And how we can solve issue when some names could be presented as a bit different JSON structures? And how we can validate that these slightly different JSON structures are equal?

> We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.

C and C++ libraries with light language-specific wrappers largely serve this purpose, in practice. It's plausible that, say, a PHP postgres client lib and a Node postgres client lib will share nearly all their code—probably as C or C++.

This doesn't get leveraged much aside from interfacing with daemons and sometimes for extremely complex e.g. media or crypto libraries, though. No-one's doing this for high-level software workflow building-blocks, like a password reset flow.

[EDIT] as for the "why", I suspect it's because the things it's used for are far simpler than the things it isn't. A password reset flow can potentially need to interface with lots of different things, some of which may be custom to the project, some of which may vary with run-time input, et c. If you cut it down to only the parts that could truly be re-used anywhere, with all kinds of points where you can hook in as needed... you've only made about 5% of the work re-usable, so it's just about pointless.

I'd say this is more true for C vs C++. It has been my experience that the interop between C and languages like Ruby is much easier, the tooling is usually much better (just use ffi!), and the complexity is less (since that is what we're aiming for right?)

>> We haven't really found a way to reuse these among different technologies.

The Common Language Environment on VMS, or its lesser(but infinitely more available hence virtuous by effect) imitation, dot net CLR . FFIs, if you bought a good environment. Sounds like if open source implementation of good foreign interfaces existed, the OP's problem wouldn't exist, let alone system level interoperability.

> What's the "right" way to store person names?

Store both a unified name field and separated given name / surname fields and let the user manage both. Yeah, this risks going out of sync, but that's the user's problem, not yours. Yeah, three fields are technically more complex than one or two, but it produces less complexity down the line.

So what happens when someone doesn’t have a surname?

Falsehoods programmers believe about name: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

> So what happens when someone doesn’t have a surname?

Then they leave that field empty.

Falsehoods programmers believe about names:

https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

As a practical matter, the HL7 FHIR data model for names can work well enough for the vast majority of applications in almost any country.

https://www.hl7.org/fhir/datatypes.html#HumanName

> What's the "right" way to store person names?

  struct Screenname {
    Matches<String,regex"\w+"> _;
    // TODO: ensure it is illegal to ask for users' real names
    // for now we'll just do the right thing unilaterally
    }

Works for email, phone numbers and addresses, too.

Actually, it doesn't. All of those are, well, addresses, and have some mechanical structure necessary for routing (admittedly, for phone numbers, the structure amounts to "any sequence of 0-9, #, *, and possibly those four wierd extra symbols with the fourth DTMF column tone"). You probably don't (or least shouldn't) care about that structure as long as it routes correctly, but it does exist. (Ie, you'd still use a string for that data, but it's not inherently incorrect not to, just pointless and error-prone.)

I think the point is that someone else cares about the structure and sets its rules - which may change without warning or have complexity you are unaware of - and therefore if you are not the telephone company or the post office you are likely overstepping if you try to enforce structure rather than merely passing along what you were given.

The rabbit hole on names goes deep [1]. Lots of other fun rabbit holes. [2]

For how to model physical objects like those cables, go see how Grainger does it as a working practical model.

To really handle names properly, you need more context than the name in the presentation layer that many schemas take their modeling from can obtain. Government health care or similar widely-adopted encoding is sometimes Good Enough. If you want non-lossy exactitude however, then that's a much bigger scope (I'd be investigating a first-pass classifier with contextual hints taken from various geolocations, age, etc., that implies soliciting the name comes after what you normally solicit for input, and refining from there).

Password reset; this is why vendors like Okta exist to abstract authN away for us, and auth0 for authZ. Then there is the rabbit hole of what this abstraction leads to...

[1] https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

[2] https://github.com/kdeldycke/awesome-falsehood

> Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.

The way past bureaucracies that didn't have the luxury of unlimited cheap complexity did: you set some rules and anyone who doesn't like them can either suck it up and follow them, or deal with the consequences themselves. Allowing users to specify arbitrarily complex requirements and never saying "no" is what gets us into this problem.

I attended a Common Lisp convention at MIT around the year they did away with the SICP course and replaced it with a course using Python to program robots. It came up in the Q&A of one of the talks and the question got kicked to Sussman who was in the audience and gave an explanation.

He said that it used to possible, practically speaking, to treat software like discrete components so that you can reason about how to combine the units and predict the result, etc. By contrast, much programming today is bodging together a bunch of libraries whose behaviors are often poorly defined or otherwise opaque so that one often enough has to treat them like black boxes against which you have to apply scientific reasoning to understand how to use and integrate it. Hence Python and robots.

Presumably the new course teaches skills around managing this complexity, but the shift in model advocated -- from the metaphor of well-behaved discrete components in composition to opaque units whose inputs and outputs (possibly depending on hidden state) must be discovered was interesting to me, and ultimately adds to the accidental complexity of doing software now.

Do you think this sort of problem may have a distinct visual representation that would fundamentally reduce the complexity of library-dominated development? Or are you imagining a whole programming ecosystem built around your idea, so that the fundamental problem Sussmen described is somehow obviated? Or are you really trying to address essential, not accidental complexity?

This is an admirable and worthy pursuit, but keep in mind that any process you make streamlined:

1. becomes a post from which clever and ambitious people can build complex things. You invent docker to simplify application deployments and then somebody builds a n-dimensional microservice cloud on one side and starts commercializing new hardware architectures on the other. You don't remove the complexity, you just let it move around into new domains. (not a bad thing!)

2. is more temporal than you expect. The "finite and fundamental" problems of yesterday, today, and tomorrow are of different sets -- partly because (1) opens up new problem classes and partly because most problems are inescapably cultural and therefore subject to fashion cycles.

This assumes that complex software problems are decomposable into discrete components defined and analyzable solely by their public interfaces. Complex software frequently cannot be modeled in this way -- it is what makes the software "complex". In reality, components have complex interactions far outside of what is captured in the component interfaces (see also: Hyrum's Law). Surfacing these implicit interactions at the component interface level just creates insanely complex interface definitions, moving the complexity somewhere else.

To tie it to something concrete, just about everyone thinks C++ template metaprogramming is unreasonably difficult to reason about even aside from the syntax, but it exists to be extremely powerful at precisely expressing behavioral contracts of a software component that are difficult to do in other languages. Even then, it barely scratches the surface of what would be useful to express for the interaction of those components to be "simple". The number of design parameters of a component that really matter in various contexts are astronomical. No one can deal with reasoning about that many component traits in practice, so the software engineers simply hide most of them -- the complexity is still there and will manifest in unexpected ways because it is not visible at the interface.

All systems engineering is complex for this reason. Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity, which creates an unavoidable cognitive load. This isn't a software engineering problem, it is a systems-thinking problem. In chemical engineering, for example, there are often complex system design problems that cannot be adequately addressed by decomposing them into a sequence of sub-problems, all that does is hide major complexity at the interface of the sub-problems.

There are many embellished, hand wavy statements in this comment.

Could you give real world examples of:

  * "In reality, components have complex interactions far outside of what is captured in the component interfaces"
  * How template metaprogramming "precisely express[es] behavioral contracts... that are difficult to do in other languages"

I think you contradict your poetry with "Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity". If _any_ non-trivial software system _can_ be reasoned about as a monolith then it is true there exists some model with sufficient simplicity that there are well-defined discrete components.

That's a nice belief, but you've given no rational argument for it. Consider a monolith implemented as a single, unstructured function: 200,000 lines of unstructured code, written by a goto-loving maniac. Demonstrate why some equivalent model with well-defined, discrete components must exist.

You are misunderstanding the problem. Any systems problem can be decomposed into a discrete problems with sufficient fidelity to satisfy a quality requirement. The problem is that the number of discrete components in the model required to provide sufficient fidelity to minimize complexity can frequently number in the millions. No human can reason about that. All you've done is move the abstractions around. Many physical engineering systems have this property; it is in fact a major use of high-end compute resources.

Traditional engineering disciplines that routinely have this problem of non-decomposability actually treat it as a systems problem (or a supercomputing problem), they don't shy away from it simply because the cognitive complexity is difficult. They convert the entire system into a set of simultaneous equations that needs to be solved for. In software, we would call this designing a monolith, but the reason it is done in traditional engineering disciplines is that you can't wish away the fundamental nature of the problem. In software, because it is not a physical engineering discipline, you can pretend that this issue doesn't exist, for a while at least.

Systems complexity is intrinsic, there is no trivial reduction. If there was engineers of all types would serve little purpose. Not coincidentally, the engineers that command the highest salaries are those that have the cognitive capacity to reason about the most complex systems dynamics, the dynamics that can't be decomposed into independent sub-problems before understanding the behavior of the entire system.

Welcome to the club!

The fundamental problem of simplifying software is humans. Just consider date formats, time zones, and tax codes. Humans love to make complicated things.

My philosophy on this has been to take the complex human stuff and stick it in a black box. A professional feather in cap with this approach is called BladeRunner ( https://dl.acm.org/doi/10.1145/3477132.3483572 ) which radically simplified the distributed system aspect by putting all the gnarly glue and business logic in a V8 VM (JavaScript).

My next thing is a focus on board games where I have invented a programming language and have started to evolve a platform. It's called Adama ( https://www.adama-platform.com/ ), and I think it is pretty cool. The interesting thing is that the complexity of board games is exceptional.

I have a few clues to share. The first is that reactivity, which is found in excel, is a key to simplifying software as this makes the glue more automatic.

Another clue is figuring out bidirectional communication which relates to reactivity as a two-way street. However, this is primarily hard because we don't have great things off the shelf to deal with this beyond TCP. For more of a deep dive, check out https://www.adama-platform.com/2021/12/22/woe.html which talks about WebSocket.

My final clue is that you can't run away from state. So many people offload state because state is hard, and you have to contend with it. I'm building yet another database.

If you look from perspective of software creator that wants to handle all kind of timezones or post codes - yes humans complicate things.

If you look from a person or user perspective - time zones or date formats are used in one location or in some context. People living in that context have it a lot easier because most of the time they don't care about other time zones.

I would say people simplify things but on local scale. If you want to go global that is your problem not humans that live in one place and use single time zone all their life.

I have been developing software for over 20 years and leading teams of developers for the last 10. Please allow me to give you some insight from experience.

To your first point, in my experience, the best software, including complex platforms that handle massive amounts of traffic and data, can be built and maintained by small teams. I mean less than 20 developers. The larger the number of developers working on an application or platform has an almost inverse correlation to the speed in which new features are built or bugs fixed.

Software architecture has been done visually since perhaps its inception (tools like UML). At most places I've worked, every new project or large feature is diagrammed visually to use as a guide in breaking down the project into component parts.

In my experience complexity arises, not from lack of tools or industry knowledge, but from 3 main causes:

1. Inexperienced developer asked to create project, who just starts building without planning beforehand.

2. The main one - business demands features built that were never expected or planned for, and built as fast as possible. This causes developers to take shortcuts, make inelegant and difficult to maintain design choices. And leads to often inscrutable code that becomes technical debt especially after the original developer leaves the company. This will always happen as long as software is used to make a business money.

3. High team turnover - I've seen places where developers came and went so often that there was a myriad of things half started and never finished.

How I've solved or helped alleviate these issues: Make the business case to company owners or management that technical debt will be an ever increasing impediment to development velocity and the dev team will need a percentage of work in any given sprint to tackle tech debt issues (as opposed to having everybody work 100% on new features and bug fixes all the time).

I have successfully taken a platform that was bug-ridden, difficult to maintain, and where new feature development had slowed to a crawl due to the over-complexity of the software, to a place of stability and ease of development, simply by allowing our team to chip away at tech debt over the course of several years. Tech debt issues were rated by level of complexity, risk to the business in change (regression bugs), and impact on team velocity. We worked on the highest impact, lowest risk items first and kept going until there wasn't much left on that list.

His examples are getting a little dated, but reading this, I can’t help but think of Spolskey’s “Architecture Astronauts” essay:

“When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. Blah.“

https://www.joelonsoftware.com/2001/04/21/dont-let-architect...

Came to say the same thing, Joel nailed this 21 years ago. Show me the code, and show me what it can do that other stuff can't.

While I am a big critic of current software development tools and believe they are nowhere near what they could be, I also believe that there will never be anything that makes software development easy. The reason is that software is a tri-component system: how to do something via computation, how to communicate between people, and how to think about something. We are actually somewhat decent at the first. We are absolutely terrible at the last two. Again, I think there's a lot we can do to improve things, but we will always be pushing against the ceiling of humans to communicate, collaborate, and accurately encode a domain of knowledge.

If one thinks of software programs as stored knowledge, then it makes sense why they are so buggy, error prone, etc. It's that the knowledge was never complete to begin with. Throw in the difficulties of computation, and you have the mess that is the software landscape.

I think that's an honourable but unattainable persuit.

A huge part of the problem is the messiness of the real world and costs.

While there may be a finite set of fundamental problems, the set of possible hardware and software platforms is ever increasing and each has it's own constraints and strengths.

A "universal programming environment" would need something like a "universal hardware interface". That's why things like Java and the Web became so popular despite their poor design.

Also, visual programming is a hard thing to make work and human language skills seem to be far greater than human visual skills.

Perhaps the best that can be done is "starting from zero" and making sure everything, from the ICs in your hardware to the memory models behind your software is thoroughly tested and formally verified.

This is insanely expensive with current technology. Perhaps some fancy math and AI advancements can make formal verification powerful enough for ubiquitous use. Until then, I see little hope for a "universal programming environment".

Good luck but your motivations will probably not line up with the masters who dictate things that become complexity.

For example, I look at our software running on .net MVC and think as an Engineer, it simple and knowable but the Front-End Team are less worried about simplicity and more on flashy front-end stuff since that is their job. We end up bolting on a Front-end JS framework and complexity immediately ramps up by like 300%.

Are they wrong for wanting a better and more flexible front-end? Not necessarily, I mean all the other companies have cool stuff and if we don't maybe our company dies.

Ditto for lots of other examples...

I agree with you. I have other things to do, but good luck to you.

There’s an awful lot of cultural baggage in coding. Many of the concepts that seem essential - powerful text editors, devils tooling - can be completely removed. It requires rethinking from first principles and being willing to upset the Apple cart, but I believe it is possible.

I don't think you can solve it. (Not a comment on you - I don't think anyone can solve it.) The problem is that software at least to some degree reflects the external environment.

Let's say your software is in a financial company. Their software has to enable them to follow all the government financial regulations. Well, the government is following the larger societal "complexify to the point of collapse", and the financial regulations are certainly doing so. That means that the external behavior (the "business logic") of the software is insanely complex. You can't make that go away just by visual programming.

But maybe you're not in the financial world. Maybe you're just writing programs for internal corporate processes at some generic company. Well, your software is still subject to the complexity that builds up in the company processes. Again, the programmers can't eliminate that complexity.

Or maybe you're writing a customer-facing app - an external-facing web app, or an application that people actually install on their machines. Here you're at the mercy of the product or project manager trying to find new things for the app to do, and they still complexify the app to the point of collapse.

The problem isn't that programming is too complicated. The problem is that what we want programs to do is too complicated. Visual programming can't save us from that.

It's a solved problem, if you are solving the right one. Most companies out there are solving problems they don't have. The FAANG disciples descended upon the world to preach the gospel of Complexity, and everyone bought it - because everyone wants to be like Google.

What, you think "microservices" is a new invention? We called it distributed systems, and we really knew not to go there unless we wanted to decimate our productivity and sleep.

A very worthy, laudable goal. In addition to some of the other good responses here, my reason for thinking it can’t be solved with better tools alone is that, contrary to what the article suggests, engineering & tools alone are not the primary cause of our complexity, customers and consumers and management are. Business and the need to keep making money is what causes software to need to evolve and change, to support new features that weren’t planned into the architecture, and to develop software in shorter timeframes than is necessary for clean solutions. This is what I’ve seen in a few decades of professional programming, that programmers are perfectly capable of solving hierarchical and cognitively complex problems. Customers or salespeople ask engineering to implement strange inconsistent features, people aren’t happy with your software because your competitor does X, and we’re always walking a balance between time and resources to meet those demands, and there’s never enough of either. A lot of software business, for better or worse, is also driven by fads, so need complexity inducing maintenance just to not feel old and cruddy.

The other reason complexity happens is dependencies and DRY thinking, which is often good, but dependencies, centralization of code, and sharing of code are also a risk. Avoiding repetition is what causes abstractions to grow. Using other people’s libraries means that there are complex interactions you don’t understand. Most of the time it’s all fine, but occasionally sometimes it’s not. This trade is made consciously, for the reason that it’s much faster to develop your app using existing libraries, and not reinvent all wheels, and nearly everyone is doing it. The reason that tooling is unlikely to solve this part of the problem is that failures are so often caused by incorrect expectations - someone using a library hoping that it will do X when it only does Y. Even in theory, better tools could only tell us low-level information about what a library does, but I don’t think it’s possible for tools to tell you at a high level what a library can’t do.

> there are finite and fundamental classes of problems, that underpin the vast majority of problems

We already have had, for a long time, a 'universal' model for a specific slice of information processing that consists of a very small set of components: Graphic User Interfaces.

One problem is the diversity of domain semantics and all the associated nuances. In GUI programming, this is the tedious bit of naming the components and mapping them to processing elements.

Another problem is engineering culture (du jour). A reductionist approach that attempts to leverage structural commonalities requires what is poo poo'd as "boiler plate" and all the associated warts of component oriented programming (including factory-factories). A revisionist shift in mindset is required.

> I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers.

I think one thing missing from this line of thinking is that these are people. They have their own motivations and egos. They get sick, have families, take vacations, quit.

I think part of the way large teams and large companies are structured is a hedge against changing teams with highly variable capacity. It's hard to overstate how much it can hurt a project to lose a someone who has a ton of experience locked up inside their head.

The alternative is that I have worked for places that try to treat thier engineers like fungible tokens that can be shuffled and replaced at will. That environment feels extremely dehumanizing and demotivating.

How many clever computer scientist (including me ) had this idea and vision at least once? And how many attempts by languages/DSLs/visual languages have already been tried to create?

I believe you are wrong, I'd wish you are right ;)

The problem with software engineering is not technical complexity. The problem with software engineering is the complexity of humans and the real world. Have you ever worked on software that needs to encode 30+ years of government and employee/union agreements into rules that can be used to automatically decide how much a specific employee needs to be paid overtime in a given month? If you had you would know that no amount of tooling or software will make that problem simple.

I definitely share this belief. When I started delving into functional programming I was immediately struck by how whole classes of problems just disappeared once people found some useful abstractions. Programmes that took thousands of lines of code to express could be written in hundreds or even tens.

But FP seems a bit stuck nowadays (to my untrained eye) with the breakthroughs from the greybeards of the last decade.

Do you have any progress thus far? Anything people can contribute to?

Me too, and I definitely agree that it is time to move past thinking in "languages" to a more holistic view of programming systems. The fact that as of now, only ~1% of people can access programming and wield the magic of the computer is a tragedy akin to the comparable levels of literacy in the medieval era. At the present, there seems to be a minor renaissance of thought along these lines stemming back to the pioneers of the field (McCarthy, Engelbart, Kay, etc). Everyone has a different take as to what the answer is, but I think "let 1000 flowers bloom" is the best way to find a solution to a complex and largely unknown problem such as software. Personally, I'm afraid that if we fail, the means of programming will become gated behind AIs and society will continue in its trance-like state of passivity instead of creative expression.

We will have to contend with the last bit regardless. Programming requires attention to detail and tolerance for tedium, the erosion of which will only continue as AI grows.

> The fact that as of now, only ~1% of people can access programming and wield the magic of the computer is a tragedy akin to the comparable levels of literacy in the medieval era.

I don't know why programmers think this way. We don't expect everyone to be a doctor, a botanist, a novelist, or a musician but for some reason we think anyone can be a programmer. That's just not the case -- programming is a skill like any other -- it takes some natural inclination, some training, and a lot of practice. Just like any other skill. A programming language is to a programmer what an instrument is to a musician.

No, but we expect everybody to know reading, writing, and arithmetic. Programming may be more intrinsically difficult than these, but most of the reason it's so hard today is because of solvable problems that beginners run into, to the point of half of expertise is just knowing to avoid all these pitfalls. The downside of the open source developer ecosystem today is that putting it all together is typically a painful process, which is what Replit is aiming to solve. There is a world of difference between a good IDE and a bad one for a beginner, and a lot of that comes down to integration with the language.

Anyways, there's a reason programming should be made more widely accessible - it develops thinking skills and rationality when you learn it, as Mindstorms pointed out. Between the anecdotes in the book and things like [0], I think it's a travesty that we aren't pursuing this to its fullest extent, and instead try to teach children Python or JavaScript, two decent languages for software development but not exactly forgiving with beginners.

[0] https://medium.com/@stevekrouse/goodbye-seymour-cb712757264f

When I was a kid we also computer class were programming was taught. And music class. And drama. And woodworking.

And like with these other things, you can make getting into them easier. With music, children start with simple instruments that no professional ever uses. And it's the same with programming, there are plenty of easier environments for children. But if you want to make programmers and you want to make musicians, eventually they have to use the real thing. My own son jumped straight into the deep end of Unity development knowing nothing because he wants to build something real. I neither encouraged or discouraged that environment and it's pretty unforgiving.

I don't think the solvable problems you speak of as are as solvable as you think they are. Also making software development out to be special both in terms of it's benefit to thinking skills and rationality and how it's merely some tools away from being professionally approachable to the masses is totally unfounded.

I think they're saying we should make the tools to program more accessible and allow people to play music easier. Sorta like an electric keyboard vs a grand piano.

We already have those tools for just about every age range. But like with music, just because you can play twinkle twinkle little star doesn't mean you're going to be able to join the orchestra. And if you do, you're going to need to able to play that grand piano.

I think the same, but that requiere resources.

I try to work on the side in a spiritual successor of Foxpro/DBase (https://tablam.org).

I consider a mix of relational/array model fill a lot of ergonomics for this (and it was proved to be right by the family of DBase langs).

But what makes this much more complex today is the explosion on OS targets (Windows, Linux, MacOs, Android, iOS, Web), and the requirement to integrate with many other stuff, dealing with many formats (json, xml, ...), is harder to do UIs now and the base support is more inconsistent and mixed as ever...

So, is possible to make a simpler tool, I certain of it, but then the developer/user will say "ah, ok, so how this connect to Redis, GraphQL and Amazon Web Services, run this on Android, Windows, parse CSV, ..."

and that is what make this very hard at the end...

You're describing Mathworks Simulink, more or less. And, I also think it's an incredible shame that it's not a more commonly used tool for forms of software development outside of basically controls engineering.

May be you are looking at a much fine grained composition/decomposition tier to arrive at breaking down "all" software engineering towards simplicity, but anything real world usecase would end up being an aggregate so big, your composition/decomposition tier itself would add to complexity rather than reduce it.

BPMN and all of the UI pallet, drag drop coding applications try to do this very thing, this is super successful at smaller scales but breaks the first real world application.

I am skeptical of visual programming in general. Some of us just aren't visual thinkers. I find it much easier to conceptualize complex concepts in terms of text and narrative.

Don't worry - I am a visual thinker and I'm skeptical of visual programming. While it's cute for toy examples, I haven't seen it handle complexity well.

I am sure every SE agrees that somehow things are always the same. How about making code pieces algebraic in order to reason and write with them much easier? Like using Haskell?

Isn’t this the dream since year dot? The problem is the real world is full of leaky abstractions, with different layers evolving at different rates and massive adoption inertia.

Out of curiosity, have you watched/read Alan Kay's team work at VPRI ? They managed to make a kind of desktop OS in less than 1MLoC, tangling DSLs to generate the boilerplate from very succint rules (or parsing RFC specs for the network stack). They wanted a 100kLoC but couldn't do it, but I believe it's still doable.

I think I understand what you're getting at, but do you have any examples? Like you said, seeing helps understanding! :-)

Can you elaborate and/or give examples? I read your comment and I can only see vague pseudo-profound statements.

This is like a philosophers stone of swe or what fred brooks referred to as “silver bullet”, good luck ;)

This is essentially Alan Kay/VPRI's old research program. I'm not sure why, but when people get into this they always think dataflow programming will solve everything - we already have Excel and it does do a lot, but it doesn't do everything.

Software complexity in most cases is a direct reflection of business complexity. Changing requirements, overly complex processes, “approval” steps, etc., etc. if you can build it faster and cheaper - great! But you will end up in the same place eventually.

What you're describing is one approach, the other is that you pay a bunch of SaaS/PaaS/IaaS companies to handle a lot of the workload for you so you and a small team can just focus on the business logic of your particular niche.

Agree that this helps in some ways, but it also adds complexity. You can e.g. subscribe to a billing SaaS, but then you have to link your system to the billing system somehow

Have you looked at Clojure and REPL/data driven development?

In addition to the comments below, complexity in software is just as often driven by politics/organizational structure. There is no solution in software to that problem.

Yes. And that's the root of this problem:

> Nobody can subtract from the system; everyone just adds.

Congratulations, you just (re-)invented UML. Or perhaps the "no no, its not UML" semantically anemic C4 thing with lots of blue squares.

Nothing on the language side fixes the problem of people just not knowing/understanding the project requirements.

this reminds me of the Head First series of books. I can still recall specific pages from the Head First Statistics book not because of the content but because every topic (and probably page) had a creative visualization.

No Silver Bullet. Frederick P Brooks 1986

Sounds like the work of Bret Victor.

Don't want to discourage you...but... LOL

Reply on news.ycombinator.com