> I am here to tell you that the reason so much of the world seems incomprehensible is that it is incomprehensible.
This statement rubs me the wrong way, in a similar fashion to what you've pointed out. All of the world's systems and technologies haven't been able to be understood by one human for (most likely) a very long time now.
The world is incredibly complex, but at this point in history I can go and learn about pretty much all of it whenever I want. By definition that means it is very comprehensible. It just isn't practical for me to do so.
Modern scale as you've pointed out, and speed at which things are expected to happen to keep that modern scale going, are the bigger factors I think.
Instead, I found that the more I learned, the more I realized I didn't know. Finding answers inevitably lead to more questions that I had answered in the first place.
I don't know that we will get to the point that will stop being true, or how many lifetimes it would take a single person to understand it all. It certainly feels like we are past 1 already.
[Edit-] My comment originally focused more on natural sciences in general, but then I changed the opening quip to the personal anecdote about programming. The last paragraph, however, was geared more towards science, and knowledge in general, not programming languages per se.
What previously was an issue of "I have no idea how to even approach this problem, not to mention write code that solves it" it is now a question of "I have at least 5 ideas how to solve this problem, I need to analyze which one is best solution for this particular context". It's much less about learning the tools and a lot more about how to correctly apply those tools for the right situation. I feel that this latter part could take my entire life but I can definitely tell I'm getting better at it every year, even if I significantly change the problem domain I work in.
As a matter of fact, every time I change the problem domain now I noticed less and less time necessary to become proficient in the new domain as I am able to pull from the experience I gathered previously. Computer science seems like an "infinite" problem domain but actually what I found in practice is that the more problem domains I move through the more I notice similarities and solving the same problem over and over again (albeit, with some small differences which may matter if you work in a performance or resource constrained environment).
I would guess poster above doesn't yet have 30 years experience. Perhaps 10 or less.
My experience echoes both yours and the above poster's - following a sort of bell curve - after initial ramp up, there was a significant period where every question led to several more. At some point that peaked, and by the time I'd been writing code for 20 years the rate of questions was dropping quickly.
Also the character of problems changed a lot. There's a point where QA sites like StackOverflow become largely useless for your problems, as they're too broad, complex, and dependent on specifics to fit the QA format. Even relevant blogs and academic papers become thin after 20 years.
And I want to introduce my young children to programming, and also find that there is no obvious good modern starting point. Scratch isn't it. Python isn't it. JavaScript most definitely isn't it.
I'm considering Logo and BASIC -- possibly even a BASIC with line numbers and no multiline procedures/functions. Everything modern has a huge barrier to entry for just-learned-to-read children (and the otherwise uninitiated).
Assembly is easy to grok, but hard to use. Modern stuff is easy to use, but takes a while to grok. BASIC seems to hit the right spot for Beginners (All Symbolic Instruction Code).
Information retrieval is its own domain with its own body of knowledge that you don't understand if you haven't done it.
Ditto for dozens of other subfields. (And subfields of subfields like AI.)
Distributed programming is its own thing now. That's why there are so many HN articles on say Raft, eventual consistency, or CRDTs -- those weren't topics of interest when I went to school, etc. People are learning them on the job.
In other words, there's really no such thing as a "programming" expert.
But I would also say, while it's probably impossible to really master more than a handful of domains in one lifetime, an awful lot of problems really can be reduced to inputs and outputs, and the more experience you get with solving diverse programming problems, the easier it is to penetrate some of these "advanced" sub-fields, at least at a superficial level, and solve a problem or two using tools from that domain.
It makes me want to go back and get a Master's in CS, focusing on those topics. But I am unsure if I would just be paying for a credential for something I could learn myself. On the other hand, it would be hard to beat the structure of good program.
I recently started working through Network Programming in Rust, but it wasn't as low-level as I wanted.
I am also planning to take a hack at the open source Illinois CS 241 book & course
For instance, at various stages in my career, I've done professional work in a lot of different languages and ecosystems: c/c++, objC, Java, Node.js, Swift, React, Rust - and each one of these has it's unique features, strengths, and weaknesses, but I think in general the more different tools you use, the more you see how much they have in common.
For instance, when I pick up a new framework or language, I'm looking for the same basic things: like how does the build system work, and how do I make an http request, how do I optimize for performance, and then I'm generally plugging the new tool into the same design patterns I've been using for years.
If you do algorithms, nothing changed. If you do frontend, everything changed. If you do backend, change is slower - you are now expected to master (the misleadingly named) serverless; and you are once more not considered uncool for preferring an RDBMS to a NoSQL.
The bleeding edge changes rapidly, but that’s what it’s there for.
I agree but more due to the blog author combining interesting topics like complex systems with value judgements asserting that a) the net effects are a major problem that needs to be worried about/addressed more than before, b) it's reached a tipping point, c) it could now get much 'worse' much faster than before and in new, unexpected ways.
It's common these days for authors to try to elevate the relevance or urgency of their topic this way but how we should feel about the 'way the world is' is uninteresting. The world has always been complex and it's always been growing more complex at increasing rates. Such transparent present-day exceptionalism just evokes the "but this time it's going to be different (and much worse)" echoed by every concern troll and prophet of doom for hundreds of years.
In reality, change is constant, net effects at large scale (ie global and decadal) are, on average, generally both positive and negative in roughly similar proportions to broad past trends. While there are certainly significant disruptions and calamities, the long-term, global meta-balance of pos/neg tends to eventually regress to the historical mean. In any given localized context there are always winners and losers and determining whether the net impact is "good" or "bad" is a subjective value judgement which varies based on perspective, context and how one chooses to count.
For example, it is understood how to safely run a nuclear plant. But human nature being what it is it might not be possible for that large a group to correctly and consistently do so. Someone is going to be lazy or greedy. More processes and controls are added, making it more complicated, etc.
Another example is voting for president. We've made a political system that puts so much authority and influence in the hands of one person, over such a complex society and economy, and we ignore all of the important issues of the day instead squabbling over irrelevant details.
In practicality things keep becoming more complex and the risk of major incidents increase because of this, incidents that nobody sees coming and nobody knows how to fix right away. A serious problem.
You could learn any of it, but most certainly not all of it. And that’s a subtle but crucial difference. As the number of pieces/interactions in even a single end-to-end pipeline multiply, it would be extremely difficult for you (or anyone) to stay on top.
Those things change from place to place, but lets put them in Jamaica, for instance. They have no knowledge of the hurricane coming in a month, or the yellow fever next door, or the goings with tax laws back in the metropole, etc. The grand effects of nature and other people outside of your little plantation were totally unknown to our Jamaican.
This has been true for eons, and I'd say it's mostly still true today.
But, strangely, it is a bit less true today. We do have pretty good hurricane predictions now, and pretty good practicing epidemiologists (though with covid it's debatable), and pretty good information on the goings on in the capitol, etc.
I feel that the complexity of the world is finally being laid a bit more bare, whereas it was more covered before.
None of this is based in an increased understanding of nature, these are all problems we've created for ourselves.
And nobody can see the problems coming (like the great recession), or convince all the disparate special interests to fix them (like climate change), until things blow up.
As a programmer I've learned to be scared of too much complexity. I have the same feeling every time I have to deal with the government or big corporations, where they are complex that they are often utterly retarded.
And it seems like everyone's first solution is more of the same :-(
Well this is just academic knowledge. Which is a tremendous advantage, but when you include everything else known about western civilization at the time, there's a much lesser percentage difference. And when you bring in the whole world, we're in the same state of confusion as 300 years ago.
You have to go back a lot closer to the cave man for us to be much more fully baffled.
Even though there sure are a lot more people now spending lifetimes making things un-necessarily complex, it's been pretty gonzo the whole time.
I would also say we may now just be more aware of our complete misunderstanding.
I think it applies to emerging complexity not much differently than it did in antiquity.
Regardless of what you are saturated with, anything more is an overload.
Note: it's just the greatest recession this century. So far. Well, probably until now. The century's still young. Plenty of people saw 2008 coming, even more knew it was inevitable, and way way more than that FELT a downturn like that was going to happen sooner rather than later.
Mitigating that we have better information than ever, but it scares me that these enormous decisions are often made based on greedy influences.
Instead, as the sphere of human knowledge grows, our awareness of the area just outside that sphere becomes both larger and more obvious.
The general complexity of the universe exists regardless of how much humans understand it. (probably)
Basically, I'm saying that the world of 50 years ago is probably much more comprehensible to a highly skilled 2020 human than it was to someone actually living in it. It wouldn't be totally comprehensible to either, of course, but certainly the added context brings a lot of advantages.
It's not really incomprehensible either, you just have to adjust your expectations toward not knowing everything well enough to predict its behavior.
A thousand years ago there were numerous independent but relatively identical farms and towns, all facing a circumscribed set of similar problems. Life was simplier.
Fast forward and now we have the a limited set of large powerful nation states, and groups of nation states, centrally controlling finance/trade/health/education systems that are invented (by politicians and lobbyists!) as they go, and affect everyone.
Instead of many simple small similar systems we have fewer larger more complicated systems. More eggs in less baskets.
Some politicians in the U.S. decided in the 1990s to allow banks to sell securities, which resulted in an almost collapse of the worldwide financial system not twenty years later, millions of people drastically affected.
That was not possible one hundred years ago, because the world was not so complicated and interconnected.
Wouldn't you agree that the complexity has scaled up enormously? Or were you thinking of something else?