Institutional memory and reverse smuggling (2011) | Hacker News Reader

Institutional memory and reverse smuggling (2011)

landley.net

129 pointsannapowellsmith49 comments

Open articleView on HN

I could have sworn that I read this before, on a different website. Turns out that I was right according to my notes.

https://web.archive.org/web/20111228105122/http://wrttn.in/0...

Previous discussion on HN:

https://news.ycombinator.com/item?id=3390719

https://news.ycombinator.com/item?id=3311015

And here I was wondering what the odd title meant, but with your hint it totally makes sense. From the page:

    <!-- from http://wrttn.in/04af1a -->
    <title>wrttn:04af1a</title>

and of course if one visits https://landley.net/history/ you can see that Rob has archived some things he found interesting:

    I'm writing a book on computer history. You can look at my mirror of stuff I found elsewhere on the web, or the scans of stuff I found at garage sales and such.

...which links to https://landley.net/history/mirror/index.html

Apologies for those who lose hours if not days to the rabbit hole therein.

A while back there was a discussion about employee retention and churn, with some parties claiming a complete staff turnover over some period was a good thing, and that documentation could serve as the complete institutional memory.

I hope they read this.

Three problems with documentation. Sometimes it doesn't exist, sometimes you can't find it, and sometimes you don't read it.

Well I guess the fourth problem is it could be wrong, but that just comes with the territory.

There could be a fifth problem too, it shouldn’t be referenced.

Because it was compiled incorrectly by someone/some group without sufficient permission to do so, so referring to it would lead to corporate politiking.

Writing and maintaining good documentation is hard, which in business terms means expensive. If a company doesn’t want to pay enough to reduce turnover it likely would not invest into documentation either. And even is it will no documentation would quickly turn a novice into a seasoned expert.

> My job now was to smuggle these documents back into the company. I would be happy to just hand them over. But that doesn't make any sense to the company. The company officially has these documents (digitally managed!), and officially I don't. In reality, the situation is the reverse, but who wants to hear that? God knows what official process would let me fix that.

This is beautifully written :-D

> Oh, and as an external consultant, I'm not allowed to know some of the trade secrets in the documents. [...] I need to smuggle these trade secrets back into the company, so that the internal side can handle them. They just have to make sure they don't accidentally repeat them back to me.

he's not supposed to know things he made and left behind. this all gets me thinking about the difference between the employees and "the company"

the employees are always the company, they embody it, like he used to. but as soon as he retires he's supposed to suddently not-know the secrets of the company? best to laugh at the absurdity

At least the company hired him back as a consultant with a big paycheck.

It's ironic how managers and executices in organization, especially but not limited to companies, are always stressing about the clock for a few minutes lost here and there by employees, yet are totally fine wasting days of productivity by not having a proper documentation process or practices.

"[...] our group was a mix of people over 55 and under 35, with few in between."

We will be seeing this with nuclear soon. The only people I know that studied nuclear engineering or related fields are in their 60s. The renaissance of nuclear will put these people in high demand and a fresh cohort of young people will be drawn into it.

no - for a specific example, plenty of math majors at UC Berkeley got pulled into Nuclear Physics more than ten years ago.. William Gates was pushing some money around.. graduate students are pulled in via backdoor channels perhaps?

I know lots of mid career nuke Es. They have great jobs. Programming, industrial automation, carpentry, middle management. Not, you know, reactor design or operation, as those aren’t jobs that exist.

This doesn't smell quite right. For me, it wasn't uncommon to run in to NRE majors at college 15 years ago.

(now maybe they all work at NRC, I'll give you that)

> The alien machinery hums along, producing polymers. [...] The more I look around, the more the engineering world, once you go back more than a few years, looks like subterranean New York City. A mass of strange engineering feats humming away out of sight, produced by long-forgotten ancient peoples

These sentences made me think of the game Factorio. For those who don't know, in the original game (before the recent Space Age expansion) the official objective was to build and launch a rocket, leaving behind the still running automated factory which produced the parts for that rocket (and yes, one of the intermediate products is polymers, simplified in the game as "Plastic"). One can imagine someone, much later, encountering the long abandoned factory, still running and still producing polymers.

I wish there was a game that did this on purpose. You are given an undocumented alien half-broken contraption and your goal is to repurpose it for something different.

Space Station 13 was a little bit like that sometimes.

If someone wanted to be such an archeologist but instead for figuring out how some code or project worked (where perhaps the only remaining thing was the compiled code and some limited documentation), what sort of role/position would that be?

To my mind it just seems like something one would get assigned or throw into when looking into improving a process similar to the company in this story, but perhaps I am missing something.

If you're a strong technologist, ideally with some reverse engineering experience, and you befriend a bank CTO, you will probably be offered a job that is part this and part saving projects that are trying their best to jump off a series of cliffs.

Just make sure to set it up as a consulting gig and never, ever, work for a bank.

In "traditional" engineering, there are a lot of old code bases still in use that few people (if any) have a solid understanding of. I've seen code comments dating to the 70s. Often the code was "documented" in internal reports. Many of these reports are now lost. These reports are often incomplete or unclear when they can be found. Plus, what you read in the ancient reports may not be current.

The job ad will probably not directly say anything about code archaeology. If the job ad mentions some sort of in-house simulation software, and the organization is 50+ years old, I'd say it's a coin toss as to whether you'll have to be this sort of software archaeologist from time to time.

>what sort of role/position would that be?

Go work a normal engineering job for a bank or government on something close to whatever was the core system 30 years ago. You should see the signs when you get close enough.

My only-mostly-joking answer is that you can go work for a typical Silicon Valley style technology company, and stay around for two or three four-year vesting periods. Most of them have Google-influenced design doc processes, and few if any have a process for documenting what actually got built, much less what it eventually turned into.

One of my hobbies is feeling dumb for not understanding something, being willing to ask, realizing _nobody_ knows the big picture, and trying to document it.

tbh though I have also occasionally fantasized about finding a job that was _only_ software archeology.

It is great that the URL for this website is:

https://landley.net/history/mirror/institutional_memory.html

In other words, it is in the "history/mirror/" sub-directory, being preserved for future corporate archeologists.

This technical archaeology thing, it's a rare skill. It's not hard, but requires patience and empathy (in addition to technical expertise).

The few engineers I know that have this, went through some sort of ritual passage. A project or idea that forced them into digging.

This reminds me of the Saturn V. If I remember correctly, no one knows how it works anymore.

https://en.wikipedia.org/wiki/Saturn_V

Lol, NASA literally left an instruction manual on how to go to space, and return safely. The contributions those folks have made over the years is related to just about everything you depend on these days.

People just choose to ignore the documentation, as there is less political mileage in throwing money into space these days. =3

NASA wastes billions and billions of dollars on space projects that will likely never fly. Rest assured, there’s still plenty of political mileage to be had from space money.

Is Artemis really that upsetting... =3

Honestly yes. Imagine the cool stuff we could have had if that money was put to good use.

>Lol, NASA literally left an instruction manual on how to go to space, and return safely.

This, I'm so sick of this "no one knows how x works anymore" when it comes to ancient technology. Aerospace engineering is a whole field, do people seriously think we've just forgotten how to get to space?

I've explicitly written things down at work, starting with the sentence: "A note for future archeologists".

This smacks of Exxon...

The big merger coming in the 99 being the creation of ExxonMobile which would create duplicate documentation systems spanning the dates the author gives.

Was listening to a podcast where the interviewee talked about this problem in aerospace. We are quickly approaching a point in the industry where Boeing for example won't have anyone left that has been part of a clean-sheet design and production assembly line for a new passenger jet.

The 787 design & assembly line spin-up work was done 2004~2007.

I love when I run across minor versions of these sort of archeology projects in the course of my job.

Git makes it a lot easier though.

Git is only 19 years old, I suspect the bigger problems in this area are more likely to be 30+ year old, as most of the people who built such systems are likely to be retired.

Some git repos were ported over from SVN repos that were migrated from CVS. I never migrated anything into CVS, so I don't know if that was common at the time. But there are a lot of git repos out there with commits older than git itself (the Linux kernel itself I'm sure is the most prominent example).

> But there are a lot of git repos out there with commits older than git itself (the Linux kernel itself I'm sure is the most prominent example).

No, the official Linux kernel repository is only as old as git itself (being one of the first three projects using git, the other two being sparse and git itself); the older history of the kernel was deliberately not imported into it. There are later repositories which did import the previous Linux kernel history from Bitkeeper and older tarballs and patches, using mechanisms such as graft to tie together these historical commits and the new ones, but the result has never been part of the main repository.

Yes, I was talking about the unofficial "historical" repos, not the official git repo that everyone works off of. I should have been more clear.

That's true actually, I'd forgotten about that possibility. I believe that FBs monorepo was originally in Subversion and that codebase turned 20 this year.

I did migrate several projects from tarballs to RCS, and then a decade later to CVS. The biggest thing back then. Renaming dirs and files directly on the central server. Good old days.

Several embedded colleagues still work on pre-tarball workflows though: zip files and word documents.

Back up: https://archive.is/1AxnM

Thank you for your digital archeology.

>Unfortunately, the internal team doesn't know what the secrets are, while I do. I even invented a few of them, and have my name on some related patents.

I'm confused about how something can be both a secret and patented. Isn't the point of a patent that it is public and no longer secret?

I was confused by the same thing. I guess that the patents are only related, and don't actually contain the trade secrets.

That's the game. Put in enough detail to get the patent but not enough to actually implement the invention.

The phenomena is hardly industry specific:

https://en.wikipedia.org/wiki/Conway%27s_law

If anything, a factory is a result of the communication structure the organizations culture chose at its launch date.

People wasting each others time with inconsistent documentation... sounds like the place is falling apart. At least get a private LAN wiki up for people that work there... Good luck =3

It sounds like an episode of Better Off Ted [1]

[1] https://en.m.wikipedia.org/wiki/Better_Off_Ted

Needs (2011)

yet another working class chutzpa who cannot see the golden opportunity he's sitting on top of. this is a very sad history. executives fired people and operated under sane staff levels in a bet their knowledge was unnecessary, now thanks to this guy lack of both class conscience and self worth they don't have to suffer any consequence of losing that bet. guess they did bet right after all.

This is something ive seen at three defense companies ive worked for. They will toss the new hires in bad times but hang onto the old timers and let them do as the please. The rto never applies to them and they horde like smaug. The best job in defense is to be one of these. Boeing made lip service to passing on knowledge but when it came to true core ppl they were never part of the process.

I was supposed to do knowledge transfer at one job but I burned out and quit before it got done because my job was knowledge transfer, training new hires in C++ itself, putting out fires, fixing bugs, and shipping new features that had already been promised to customers I'd never heard of.

In terms of "Ars Longa, Vita Brevis" [1], I'm not as good of an indexer / teacher as I wish I was.

https://www.slatestarcodexabridged.com/Ars-Longa-Vita-Brevis

Reply on news.ycombinator.com