Hacker News Reader

Comment by imgabe | Hacker News Reader

There's a part in the Foundation series by Asimov where the Galactic civilization has forgotten how their technology works. They have all this amazing technology, things that generate free energy and terraform planets or whatever, and all that's left are technicians who kind of know that if you push this button that thing is supposed to happen, but they have no idea what the machine is actually doing internally or the physical principles on which it operates. When the machines break, no one is able to fix them.

I think about this a lot and the world seems more and more like this every day.

View on HN

Here's a talk Preventing the Collapse of Civilization [0] for the same idea from Jonathan Blow.

He mentioned Bob Colwell, who was the chief microprocessor architect at Intel, once encountering some issues with chips from TI. He went to TI, and those people told him they couldn't figure it out either because those fundamentals were done by first-generation magicians. And it's not just TI, Motorola, and Fairchild all have this kind of problem.

https://www.youtube.com/watch?v=ZSRHeXYDLko

I think in the old days there were people who actually understand how the whole system works in a fairly detail manner. A Polymath in Computing. How your Code > Compiler > OS > CPU > Silicon Design > Manufacturing works.

I remember there was an old Intel fellow who understands Fabs, designed CPU, has experience with OS, wrote VM and does Web Programming for his hobby. ( He gave a talk somewhere but my Google fu is failing me )

Now everyone is so specialised in their domain no one has a good overview of anything. And no one understand how all the puzzle were fitted together in the first place.

And after one or two generation, the Why it was crafted in the first place was lost. And we are left with people specialise in that domain to "Maintain" it. Or reinventing the Flat tire as Alan Kay calls it.

There are certainly places in the world, today, where machines from the 60s-70s-80s are running critical processes, and all the original people involved in design and implementation are either dead or retired - so you're stuck with technicians that only have repair manuals, along with spare parts of everything. Bespoke, one-of-a-kind circuits / machines / etc.

It's the problem of keeping things in living memory. The craziest thing about human civilization is every 75 years or so, we have to complete a 100% knowledge transfer of everything we know to a new set of humans who start out not knowing anything about it.

Obviously it's a little more complex then that, but the importance and value of education and documentation is very much in that realm: drop the next generation in a desert, and we go back to the stone age instantly.

Cixin Liu’s SF novel “The Supernova Era,” in which a supernova irradiates the earth in such a way that everyone above the age of 13 will die in a year, takes this as its focus.

Oh man, got stuck half way through that one for some reason but hadnt thought of it from that angle somehow. Im inspired to pick it back up... Like right now.

Which is why when we happen to dig out some form of document from 5000 years ago, and eventually translate, we end up discovering that besides technology and religion of the day, the society was somehow pretty much the same.

You're probably thinking of the oldest written customer complaint, from 1750 BC.

https://en.wikipedia.org/wiki/Complaint_tablet_to_Ea-nasir?w...

Everyone wants the new shiny thing but writing things down is how we got to where we are today.

The problem isn't that these machines are from the 1960s. It's that we stopped making machines like that. I'm sure someone somewhere along the line convinced someone that it'd be better to do it all over from scratch, but if these machines lasted 60 years, then they can't be all bad. In Italy, there are textile manufacturers that use looms that were built in the 1920s and still make cloth perfectly well, just like the old times.

We do this all the time with technology. It's because we're addicted to change, and we have no judgment about how much change we need and where.

That scenario isn't caused by an addiction to change. It's just not economically viable to continue manufacturing obsolete equipment for a shrinking customer base.

And what causes that shrinking customer base? Apparently there is some kind of replacement, which is brought about by ... change.

Whether we are addicted to change or not, might be another question though.

Right, just like I want to go back to a car that gets 3 gallons per mile.

I agree that the problem isn't that these machines are from the 60s. They may have even been cutting edge then.

Indeed, the problem is that the user of the machine faced a choice between 3 possibilities: replacing the machine as expertise in the old tech dwindled; building and maintaining that expertise over the course of decades; or doing neither, and letting the equipment inevitably lapse into its inevitable disrepair; and they chose the latter.

Change is constant, which is good, because there is no progress without it. The only choice we have as individuals is how, and how well, we adapt to it.

> we're addicted to change, and we have no judgment about how much change we need and where.

^ this

> so you're stuck with technicians that only have repair manuals, along with spare parts of everything

Tell me more about how people hate writing documentation, and then tell me how they can't shut the fuck up about not having it when they need it.

This is a peeve of mine.

What do you expect?

What's so surprising about it? I hate cleaning my apt but also hate when its dirty. There's no big contradiction here.

Last time I cleaned my apt I managed to delete my grub conf somehow and had to spend the day learning to manually mount an encrypted zfs pool. Luckily I have some etc keeper thing that keeps a git log of all changes to my conf files so I could restore it.

If only I could pay someone to write the documentation for me.

You would still have to explain to someone what it is that should be documented.

A better approach is to learn just enough to do it yourself, but it's not easy to document things well.

I like to think about documentation like this: Without it your work will be almost wasted as nobody can make use of it. That helps a bit :)

Thus the "if only" - I can pay someone else to clean my place pretty well, but I can't really pay someone to take knowledge out of my head.

I do think it's a bit telling that we have a lot of automated software that exists to help us style our code, but there doesn't seem to be nearly as much investment in documenting code.

Simple things like a linter that yells at you if a class has an undocumented method would probably at least be a step in the right direction. People may complain that it would lead to overdocumentation but I'd argue that it's probably better than underdocumentation.

Even something as simple as providing a warning when there appears to be an long code block without documentation would probably be a step in a decent direction. What seems trivial and obvious today likely will not in a matter of weeks - and it generally just gets worse from there.

In Rust you can just add `#![deny(missing_docs)]` to your code and compilation will fail if you have any undocumented items.

Of course that can lead to developers just adding a single line comment with the name of the type ... but it's still a nice feature.

That's actually super cool; I wish I had more reasons to reach for Rust on a regular basis but this is definitely something I'll keep in mind should I ever get the opportunity to use Rust in any capacity.

Visual Studio nags you if public methods and properties aren't documented. It's not a panacea, in fact it's easy to work around but at least they're trying.

IME, the only thing that works is to attack it from a Project level: make usable documentation (needs a review before being signed off) a deliverable or the project isn't done.

> Tell me more about how people hate writing documentation

The thing is, I don't this happens in isolation. At least not for me. I hate writing documentation because time is not allotted for it. So I have to rush it, and in the process I feel like I'm doing a poor job, which is demoralizing. I'm gonna dread it next time I have to do it.

I saw this a lot with Y2K. Suddenly all those places that had been ticking along for a couple of decades on their COBOL line-of-business systems with minor maintenance had to make some major changes.

The choice, really, was "hire some ludicrously expensive COBOL devs, or replace the entire system". Replacing the entire system failed on every sane business evaluation: high-risk, hugely expensive, no guarantee that the new system will even work. So they hired the ludicrously expensive COBOL devs (because the devs needed to understand 1970's-era COBOL, and they were rare in the OO-frenzied 1990's) and patched up the old system.

But as time wears on, those systems fall further and further behind, and the COBOL devs who actually know how to maintain them get more and more expensive (or actually unavailable). The costs and risks of replacing the system is still too high for any given marginal change, but the marginal changes are getting very, very expensive.

And then 2038[0] rolls around and they'll face the same choice. And the same risks will come up. It'll be interesting to see what they do, and what choices are available. Because patching the old system may well not be possible at this point, because there's nobody left who understands it. And migrating the complex business logic to a new system may well not be possible, because there's no-one left who understands the old code.

[0]https://en.wikipedia.org/wiki/Year_2038_problem

I don't think them falling behind is necessarily bad. Modern systems, languages and programming practices are much nicer, but they also have a lot more effort behind processors, compilers, frameworks, operating systems, etc., so overall the system is much more complex.

If anything, I think having a separate, archaic, mostly frozen system base for critical infrastructure is a good thing. The current ancient COBOL abandonware situation is probably not the simple and mostly standardized solution one may wish for, but it's a lot closer.

I'd agree, if nothing ever changed and it was feasible to run them forever. But it's not. At some point (probably 2038) the whole thing will fall over and a new system will have to replace it.

Fortunately we still have 17 years till then so surely all these companies have started preparing for this inevitability already, right? (i know the answer)

> the COBOL devs who actually know how to maintain them get more and more expensive

COBOL doesn't actually pay that well, no matter how badly the business depends on it. The law of supply and demand would suggest that it should, but it still really doesn't.

it did back then... during Y2K I worked with COBOL devs who were enticed out of retirement with $500+ hourly rates

You're not stuck with it, you just have decided that's cheaper than replacing it.

CuriousMarc youtube channel, to extent, contradicts your statement.

Even if that's true, those are exceptional examples. They're rare. So why should we care?

They are a sign of a deficiency in the tech business model. Maintenance is a bad business, and so the only way to keep things understood and working is to build new things.

This is fine as long as the "in maintenance only" portion of the industry remains small. Which will only remain true while the industry is growing exponentially. However like industrials, there will eventually be a physical limit to how much better computer networks, microchips, web search, cameras and other activities can get.

Once this happens growth in the core parts of the industry will slow, and more software will be in maintenance. However software components are both opaque and orders of magnitude more complex than physical parts. As an example, Losing practical knowledge of the linux kernel involves losing knowledge of ~27.8 million lines of code.

There may come a time in a number of decades that getting a driver patched is an impractical activity due to the scarcity of knowledge ( or potentially even the ability to build the driver... )

Linux drivers are already a mess, and have been since the beginning. But the problem is due to IP concerns by hardware vendors, not a lack of knowledge about the code base.

which makes the problem more likely to emerge when the various hardware companies stop developing new drivers due to lack of significant new hardware evolution.

e.g. why update the nic drivers if the nic hasn't changed in 20 years? this could easily turn into "how do we even modify and release our companies nic driver?"

I don't understand what you mean. Most software spends the vast majority of its lifecycle being maintained.

> Maintenance is a bad business

Could you please elaborate why?

I would agree that such business has relatively low ROI. Still, for example, Linux distros and *BSD foundations are to an extent a maintenance business.

Operational Budget vs Capitalize Budget, for one.

Build something new, affect bottom line directly vs maintenance.

Blame the tax structure for placing an emphasis on new.

This is actually good news. It provides a lever for the government to change our entire culture (in a good way). Much easier than convincing every individual that maintenance is required.

Or bad news, that the govt's unreasonable bias is the reason why it is the optimal economic behavior to neglect maintenance?

Even that is good news. We have 1 knob to turn instead of 8 billion. even if today that knob is on a nonoptimal setting. Convincing 1 government is a lot easyer than convincing every individual.

Old IBM mainframes in banks and government institutions aren't that rare I think. But I don't have any numbers to back this up.

not rare at all.

COBOL still runs the world, despite what many on this board think.

COBOL is rare outside the USA and Western Europe, I'm pretty sure.

central banks all over are running on IBM.

Reliance on deprecated behavior, but without the compiler warning. There may be some hidden risk out there, and we have no idea how much it'll cost to fix or replace.

The example that springs to mind is the social security check printers. I think they wound up reading the wire voltages as checks were printing to duplicate the behavior for the y2k fix. Was urgent as many people relied on that income.

There are rare events that you can't do much about up front, they're external. A pandemic might be a good example. There are other rare events you can simply avoid, but it's often tempting to just skip the maintenance and let someone else deal with it when it breaks. It's rare right? Not like we're going to get blamed, or even be around to have to deal with it.

Alas, those rare examples does includes nuclear power plants.

That is probably less of a concern than most other things because the industry is so highly regulated that everything is formally specified and documented.

Now guess what happens if we forget how to make something intrinsic to modern farming.

like dirt?

Not being snarky; just want to point out that current soil erosion rates due to modern farming mean the world will run out of dirt by the end of the century.

http://large.stanford.edu/courses/2015/ph240/verso2/ https://world.time.com/2012/12/14/what-if-the-worlds-soil-ru... https://www.theguardian.com/us-news/2019/may/30/topsoil-farm...

A similar analogy from MacIntyre's After Virtue is also apt:

"Imagine that the natural sciences were to suffer the effects of a catastrophe. A series of environmental disasters are blamed by the general public on the scientists. Widespread riots occur, laboratories are burnt down, physicists are lynched, books and instruments are destroyed. ...Later still there is a reaction against this destructive movement and enlightened people seek to revive science, although they have largely forgotten what it was. But all that they possess are fragments: a knowledge of experiments detached from any knowledge of the theoretical context which gave them significance; parts of theories unrelated either to the other bits and pieces of theory which they possess or to experiment; instruments whose use has been forgotten; half chapters from books, single pages from articles, not always fully legible because torn and charred. Nonetheless all these fragments are reembodied in a set of practices which go under the revived name of physics, chemistry and biology. Adults argue with each other about the respective merits of relativity theory, evolution theory and phlogiston theory, although they possess only a very partial knowledge of each...

In such a culture men would use expressions such as 'neutrino', 'mass', 'specific gravity', 'atomic weight' in systematic and often interrelated ways which would resemble in lesser or greater degrees the ways in which such expressions had been used in earlier times before scientific knowledge had been so largely lost. But many of the beliefs presupposed by the use of these expressions would have been lost and there would appear to be an element of arbitrariness and even of choice in their application...What would appear to be rival and competing premises for which no further argumentation could be given would abound. Subjectivist theories of science would appear and would be criticised by those who held that the notion of truth embodied in what they took to be science was incompatible with subjectivism."

Interesting talk by Jonathan Blow based on a similar (though software-focused) premise: "Preventing the Collapse of Civilization" https://www.youtube.com/watch?v=pW-SOdj4Kkk

Fascinating. Although I've never read Asimov, I've often posed the following thought experiment:

Suppose humanity were set back to the stone age. In the sense that all working technology is 100% destroyed into ash. We don't even have a proper hammer or knife. What we do have is written documentation about how the modern world works, and an army of the brightest engineers.

How long would it take to re-create modern tech? And with the documentation/academic papers we currently have, could we even do it? What I mean is, what are all the steps we need to take from building a forge out of mud and flint and steel, to producing an i9 intel chip, 5g networks and space ships?

Think of this as an ansible script that bootstraps cavemen to modernity.

The "How to make everything" YT channel (https://www.youtube.com/channel/UCfIqCzQJXvYj9ssCoHq327g) does that. He started from digging up clay with his bare hands to make pots, and is now roughly in the iron age IIRC, melting iron using only stuff from earlier 'tech tiers'.

One thing I found interesting is that apparently many metals used to be collected from ores just laying around. Those sites are all exhausted today. So a full 'reset' of the earth wouldn't be possible today. Of course metals could be scavenged from the machines we have now (in a nuclear disaster style scenario), but we could never fully 'redo' the technical evolution we did over the last 10's of k's of years.

Without technology, we wouldn’t have the resources to sustain our population, leading to widespread skirmishes for local resources and a rapid devolution of society. Many people would die and we’d probably be set back a thousand years at least. Pockets of the world would escape this fate, having successfully organized around the right leaders. They would probably only be set back a few hundred years and become the new governments of the world a hundred years later. These may feel like long time scales, given that we would somehow have access to all human knowledge, but I feel they are correct. There is an unfathomable amount of work to be done, and a lack of societal stability will constantly set us back even as we make progress. We will basically have to relive much of modern history.

I think it would be far worse!

We would lose access to modern tech and ancient tech is lost completely already.

How many people can farm without access to machinery at all?

How many would be able to produce any hammer from scratch? How you would ensure that they are heard and followed?

How many draft animals are present in your area, for example oxen and horses?

Rebuild time just to restore food supply is too long, we would collapse to prehistoric stage with some pockets of places where food stockpiles were not destroyed.

And maybe even worse as animals are not available for hunting!

We would survive as species, but with nearly total population, cultural and technological loss.

People worry about the wrong things!

There is no conceivable scenario in which we lose access to modern technology. OK, in some far-fetched case we can't use complex electronics. Then what?

Well, no one need draft animals because there are still many gasoline and diesel engines around that run without electronics. Since we're not all driving cars around any more, our supplies of fuel will last at least until we can get pumpjacks and small refineries running again.

Believe it or not, the farmers can farm just fine without their $500k tractors, it just means they have to farm a lot smaller area with less output, but their skills will transfer just fine.

This is akin to "preppers" learning to make fire by rubbing sticks together: in what situation do they think they'll run out of matches and cigarette lighters?

I was responding to post about 100% fictional and magical situation, see https://news.ycombinator.com/item?id=27021924

I think that noone here was thinking that this scenario is likely?

> all working technology is 100% destroyed into ash. We don't even have a proper hammer or knife. What we do have is written documentation about how the modern world works, and an army of the brightest engineers.

Preparing for such scenario is waste of time, but thinking about it may be interesting. Actual complexity and importance of various infrastructure is interesting.

> This is akin to "preppers" learning to make fire by rubbing sticks together: in what situation do they think they'll run out of matches and cigarette lighters?

This one is actually possible to be directly useful, and not an example of something useless.

You're probably on the right track.

One thing to mention, though, is that all the easy-to-dig carbon fuel has been dug, so "use coal to power X" just isn't quite as feasible. Biomass just doesn't work for a lot of it.

If everyone is still alive then a "reset" is impossible. Sure the electric grid may die and kill everyone, but the reset only happens after lots of people are dead.

You might enjoy this book: https://www.goodreads.com/book/show/39026990

>where the Galactic civilization has forgotten how their technology works.

this reminds of a recent article on the Tartarian empire conspiracy theory I posted

https://news.ycombinator.com/item?id=27013834 - no discussion - actual article https://www.bloomberg.com/news/features/2021-04-27/inside-ar...

In a few generations of nobody knowing how their technology works anymore will there be conspiracy theories regarding how that technology came to be and then became impossible to replicate.

I think about the same thing a lot lately. It was not supposed to be a prediction but just a warning, I thought, but apparently it was a prediction all along.

It's very easy for people to slip into self-indulgence and pleasant illusions but when something important breaks then nobody has a clue what to do. This happens more and more around me.

I never thought I'd live to see something like this. And I am only 41.

I'm sure there are plenty of EE types here who can tell you how to refine silicon, design and print a circuit, program in binary, assembly, C, and high level languages, explain network stacks, and build a web app.

But really, can anyone really explain how computers work nowadays? Look at how optimisations in cache stuff on the CPU led to horrible security flaws, surely that should be obvious if anyone really knew what was happening under the hood?

Everything is obvious in hindsight; these "horrible security flaws" you speak of are only a problem now because we're treating software as adversarial. If the machine's owner is 100% in control of the software that runs on the CPU, there is no security boundary between the CPU and the software running on it, so there is no need to even entertain the notion of leaks through speculative execution.

Ubiquitous software delivery from questionable sources is the problem, that's what invalidated the design assumptions underlying the cache optimisations. You might argue that it is the hardware designer's fault to not see how the software world would change in the next 20 years, but you can similarly argue that it's the software designer's fault to not keep in mind the security limitations of the processor it's running on.

Not really.

Timing attacks are not all that hard to understand - if the timing (or anything else) a user sees depends on information they shouldn't be able to see, then there's a (theoretical, at least) information leak and that's a security flaw.

I'm not a security expert, and this is something that's always in the back of my mind when considering security. It's ... one of the only things in the back of my mind - you need a constant time string compare for passwords is the most obvious example (and one that literally any decent programmer should be somewhat aware of). I only know the basics of security, and this is one of the basic principals. It's up there with "escape strings".

People simply don't think about every layer of the stack. If they did, then it would be obvious (but there's so much in the stack that this is infeasible).

JavaScript came out in 1995. Timing attacks on the CPU have been feasible since them (and heck, if you go back further then everything was terminals and multi user operating systems anyway). There's never been a time when CPUs haven't run potentially hostile operations. But virtually no-one (myself included) thought "can a timing attack happen"? despite timing attacks being a very common type of attack.

This is a key part of the Warhammer 40k lore too; a huge amount of tech is ancient and the technicians who care for it have slowly turned into a religious order that considers their maintenance routines to be religious rites.

I was thinking - maybe wikipedia will cure this...

then I thought... could this happen to knowledge repositories too? could knowledge degrade under "need source" for this and other administrative controversy?

What if we sort of devolve to https://simple.wikipedia.org over time?

This is also humanity’s situation in Warhammer 40,000—but seeing as how randomly lobotomizing people is the norm in that universe, I hope we don’t end up there :-P

Don't forget that humanity turns into an exceedingly fascistic/fanatically religious and almost absurdly authoritarian dictatorship, with literal superhuman stormtroopers enforcing its might. Also cosmic horrors from beyond time and space, civilizations that literally feed on the souls of tortured prisoners, terrifying unfathomable alien hive minds and warmongering sentient fungus-grown brutes who irreversibly infest every world they touch.

Fun times.

Also originally a scathing commentary on Thatcherism and British society, similar to Judge Dredd, but that has been toned down a lot over the years, unfortunately.

That's it. I'm joining the T'au.

Or the fact that planetary genocides are de rigueur in that universe.

Well, in their defense, the planetary genocide is partially justified. When do they call it: - Ork infestation. Without scorching most of the air and ground, Ork spores will just give bigger nastier Orks, and a possible staging ground for further infestation. - Tyranids. Yeah, you're just doing whatever Tyranids would do anyway. Plus denying them resources - Chaos. Even if there is a single heretic, they can spread their belief around and then use psykers to turn the planet into a Warp world, which is a fate worse than a virus bombing.

(I do know there have been a Tau and Necron Exterminatus, but I did said partially justified)

It's funny. In Warhammer 40k lore, any society that didn't practice witch burning was essentially wiped out by daemonic incursion. Therefore Warhammer 40k human world evolved to a racist, xenophobic, psyker-hating society.

Fair, still a shit place to live in though. Even if killing your planet is justified, you’d rather not be there for it.

This happened after the fall of Rome (West). It took Europe well over 1000 years to have indoor plumbing and sewer systems again.

Pixar’s WALL-E was similar, also The Machine Stops (which is oddly prescient given how old it is and what this year was like): https://en.wikipedia.org/wiki/The_Machine_Stops

A part of Dan Wang's 2020 letter discusses this https://danwang.co/2020-letter/

This "industrial literacy" is dominated by Asia today.

This reminds me of the current influx of software guys into the microcontroller space. Arduino's, pi's and the like. You have these super knowledgeable people who are unable to execute the electronics side of their project, so now we have a huge market of things abstracted up to their level (i.e. LEDs with all the necessary electronics already embed, just call the address and go).

This isn't even just a hobbyist phenomena. Electronics design itself is becoming more and more "turn key hardware", where you just slap chips on a board, string them together, then get chugging on the software.

Well it's a lot like a doctor fixing your sneezing :D

The difference between tech and other machines like cellular life, is that we built it from scratch in a giant writing civilization so to absolutely lose the ability to reverse engineer or reproduce by copy would require quite a dumbing down of the humans or a catastrophic loss of resources.

Something I saw somewhere is that in engineering you can copy without even looking at the thing. If you know it s possible, all countries seem to somehow access it the next decade. Sometimes it's just a psychological block.

Two short stories of his in that vein..."Profession" and "The Feeling of Power".

https://www.abelard.org/asimov.php

https://www.cs.utexas.edu/users/vl/notes/asimov.html

Theres a pretty good Liu Cixin short story that involves this as well, ashamed I didnt know Asimov set the precedent. Not surprising though :)

Also a central part of the story in Monica Hughes' Isis Trilogy.

for a delightfully humorous take on the same idea hunt up theodore cogswell's short story "the spectre general"

Why can I not upvote this more than once!

Reply on news.ycombinator.com