Theory-building and why employee churn is lethal to software companies

This means if you want to buy a struggling or insolvent company it will most likely have bad documentation and tests.

But one of the most common steps when buying a struggling company is firing people (making the company fire people before buying it) and the selection of who goes and stays is rarely a good one. And it sets signals which make it mote likely for further important people to quit.

I'd suggest just using the documentation. Manuals get used for doing testing and implementation. Using the documentation is like testing for it, and finding those problems right away is the best way I've found to ensuring that the documentation is correct and up to date.

This reveals the secret goal of documentation: To build that model, not to document every last detail. The code can do every last detail, it's the model that is hard.

Set out with that goal and... yeah, I can't guarantee success for you either. It's a fundamentally challenging task. But at least you'll be off to a better start than if you think the primary purpose of documentation is to document what happens if you set this combination of flags. That has its place too, certainly, but if I can only pick one, I'd take the high-quality high-level overview over the stuff I can read the code for in 30 seconds.

At some point I got so pissed off about this that I started assigning code reviews to him - he contacted my manager and asked that I stop, because he had no understanding of the project in question.

What did this mean from the documentation side?

I would have loved to accurately document what I was doing and how the project was structured - but I actively avoided doing so.

Every time my manager's boss knew what I was doing, I was about to be shat on from great heights.

Months of work and planning erased at the drop of a hat - time and time again.

Dependency injection? Fuck off. Spring's too heavy.

Modularity? Fuck off. The project's too small.

Hierarchical profiles? Fuck you, keep using a properties file (with an increasing number of dynamically generated keys).

I was constantly asking for a new developer to be brought in for the transfer of knowledge - at best I was told "That's not a bad idea."; yet nothing was ever done about it.

I took an unmaintainable pile of bugs and made it stable and maintainable. But nothing I ever did meant that I was someone worth listening to.

Eventually I got so burnt-out (and consumed by feelings of uselessness) that I walked away without finding another job.

I was damned near suicidal from fighting for acknowledgement.

I cannot imagine that my departure went too well for them.

I believe this is partially due to lack of training (technical writing rarely gets trained, and companies rarely invest in it as a skill), and sometimes probably also due to lack of exposure to high-quality documentation (which serve as role models you’d then want to live up to in your own documentation).

In addition, companies often do not reward expending time and effort on writing documentation, because it isn’t crucial in the short term. It’s similar to tech debt, just documentation debt.

While writing good documentation isn’t easy, the fact that it usually gets neglected is a cultural issue.

That whole process sucks and I’m sorry you have to go through it. It’s probably the best solution of the time. My one question is… Isn’t that why we have github issues?

Documentation can describe a system state, and how to use or approach it, but in my experience it really lacks in capturing the full depth of experience when it comes to interacting with it. All the little shortcuts and workarounds. All the debugging steps. Seeing something, and realising you've seen it before and solving it in 5 minutes instead of the entire day it took you last time. I could write a story about that day, but it wouldn't do anyone any good - they won't learn from that experience like I did and it won't spring to mind instantly. Assuming they even know where to find a piece written about their specific problem.

So when someone says "How do I X?" all I can think about is all the incidental things I've learned when trying to do something else. This builds the bigger picture. I can answer your question, but you'll just paste the code in and be back to ask more tomorrow. I valued the coworkers who treated such things as more of a discussion and learning opportunity than a transactional question-answer routine.

- A defined and documented process

- Project planning or organization changes

- Staffing, purchasing, or contracting

These are fine tools for a lot of business problems, but they are rarely helpful, and often harmful, for software development.

waterfall by any other name smells just as bad.

I think the reason is that senior management has no trust in the developers, which is the ultimate cause of the need to have waterfall. Each reporting level wants to show concrete evidence that those under them are "doing their best", rather than to have trust that they are.

Of course, it is understandable that people want concrete "evidence". But waterfall is the result, and you end up with big-bang designs, RFCs that are out of date because a change in circumstances necessitate a change that is not documented in an RFC.

It is a management problem, not a software development problem, imho.

Just because it seemed obvious what you were doing and how things worked while you were writing the code doesn't mean it is, or that it will stay that way. Especially if that code ends up getting abstracted out. In line comments demystify code for the next guy way better than any amount of well written documentation.

As for what they're making you do with the RFC documents for changes, try writing your stuff as an ordered multi-tier list. The first tier should be the "manager" description of the change. The next level should be the more technical description of changes. The third tier is the location in the code where that change needs to be made (like a file path sort of thing, but you could use name spaces and even method names if you want). The forth and final tier if you need it would be cross references to other related items in the list and implementation notes/challenges specific to the file path. I've used this approach in the past myself. It's much faster, and it allows you to write a spec doc that isn't a huge waste of your time because it allows you to half write your code at the same time. It also provides a sort of semantic map from how the manager thinks about things to how you think about them, making it much easier for a manager to discuss it with you without having to write nearly as much.

Another benefit to this is you can often use stuff from the second and forth tiers to add in line code comments.

My currently company only did a specific niche in digital marketing but for sales goals needed to sell outside of that niche. Unsurprisingly, the product didn’t work for those use cases.

This is key. Most engineers want to build frameworks instead of applications. But leave the frameworks to the framework builders unless you were hired to build a framework.

I can jump into a 10 year old rails code base and be productive right away if they stuck to the framework defaults. But invariably someone read some blog posts about how "rails doesn't scale" and built some rats nest of service objects/decorators/portsAndAdapters wtfs and it takes some serious code spelunking to figure out what's going on (hint - the end result is always creating or updating a database record, or making a single api call).

I would argue microservices also encourage bitrot. You can leave some service running on some ancient libraries for years it doesn't need changes. But if you have a monolith where you constantly keep the dependencies up to date - everything stays up to date.

I was baffled by this as well for 15+ years until just last year I had a key insight: most developers don’t like to read. In fact, most people don’t like to read. Most people are intrinsically lazy and since growing your skills involves hard work, they don’t do it unless pushed to do so, either by life circumstance or necessity.

Time and time again people have asked me: “how do I get really good at $tech or software engineering?” I recommend a few books or send them my standard list of must-reads. In 999/1000 cases the person approaching me gives me that defeated look that I know means: “Oh. I was hoping you’d say something not involving working hard.” Most people are polite enough not to say it to my face.

When folks have problems with HTTP I ask them if they’ve read RFC 2616, and if not, I offer to bring them a nicely printed and bound copy. It typically goes unread though some use it as a reference. Only once did someone follow my recommendation and read it cover to cover.

Folks seems to lack a kind of basic economic perspective. Spend a week, month, something along those lines growing your skills (or even just 30 minutes reading the docs on that thing you just spent 2 hours declaring idiotic). Yes, this means probably giving up your favorite Netflix binge, video game, or drinking on weeknights in favor of study. But once these skills are internalized they pay back for a lifetime with good returns. Yet for some strange reason people prefer to keep thinking short term and so they never learn what they’re doing. It used to make me sad but I’m making peace with it.

This. This is how I felt trying to understand legacy core code. I kind of like that though. The idea of being able to improve/rewrite a decrepit piece of code excite me. But you won't be productive, and getting a (somewhat) complete and precise understanding of what it does and doesn't, take a lot of extra time.

But.. that is a dead-end job. There is no career path being the maintainer of a large legacy code-base. It was pretty good if you wanted to rest-and-vest, I could do whatever I wanted. But there were no great features, no demo's, no selling of the work. Even small changes were painful.

Moved to a different team where everything is new and building from scratch and suddenly opportunities open up.

- Was the original team slowly replaced due to cost cutting measures and thusly the people who joined aren't compensated well enough to dive super deep into the system to understand it?

At some point people need to be compensated to care is my theory. Engineers who inherit large systems are either scared or simply don't care, possibly a mixture. But compensate them enough in both salary, time, freedom to understand the code which is clearly critical in maintaining the long term viability of the system.

Thoughts?

As a fresh developer out of college I had the mindset that frameworks were something you use and they always work and all you need is the documentation. My first job had a lot of "core" code written by another team in another office we would use with exactly 0 documentation. I was trying to get our UI to do something specific but couldn't figure out how to do it by just looking at the functions available, so I ask my manager and he immediately knows what to do, makes 2 small changes (in my defense it was pretty hacky how it was set up imo) and I get what I want.

I remember being moderately pissed because I was already feeling the heat of asking my manager too many simple questions and I just apparently asked another one, so I say something along the lines of "How the hell would I ever have known how to do that, there are no docs available to say that is possible and that is not obvious at all."

He just looks at me and says "You read the core code" and I remember having my mind blown at missing something so obvious for so long. After that I voraciously read almost all "framework" code I could get my hands on, I asked more intelligent questions and my relationship improved my manager and eventually came to be considered a generally good developer.

Once I had this breakthrough I was seeing it in younger developers all the time, and now any time someone asks for general advice on becoming a better developer one of the things I say is "read a lot of code", and more specifically "read the code in your frameworks"

By documenting your work and/or mentoring a junior employee to pass on your tribal knowledge.

Reasonable amounts of churn are manageable if you do a good job of acculturating new people into the team. This has actually been one of my concerns about the switchover to WFH. I know a lot of that acculturation used to happen by osmosis and people working next to each other. I think my team doesn't do as good of a job maintaining it when we're all working separately. We're a lot more efficient now, but I think we haven't figured out a good way to share and transmit context like we used to and I worry what happens if we hit a period of flux in our company. I worry our team culture is a little more fragile now.

To me this is the a huge incentive for building strong teams. Nobody should be indispensable; it's a disservice both to the company and the person. Pair programming, collective code ownership, intentional cross-training, good automated tests, continuous deployment, and good automated monitoring have all been very helpful to me there. And vacations! Everybody should take long vacations where they are out of contact. By managerial force if necessary.

Alternatively: just promote and pay your people as they develop, even if they're in the "same role".

What I learned from that project is that being indispensable sucks. That lesson has recurred in my career either for me, or when I got wiser, for other people.

It's not enough to write reasonable code that 'works' or document it as it is now, there's a whole unmeasured and thus under-encouraged aspect around maintaining a legible edit history. I often am trying to get into the head of the bug author to figure out why they thought this was a feature in order to preserve functionality while fixing bugs. They aren't always around, and they aren't always fun to talk to. If you have the history, the person is free to work on other things. To gift them to people they mentor, or otherwise re-home them. If you don't have that, if it's all in their head, they have to leave to get away from the project.

If I can figure out what you're doing I can figure out how to change one or two conditional blocks to fix a problem. Your code stays intact. If there's no history anyway, and the code is sketchy, I'm far more likely to take the code away from you by rewriting it. We can't have black boxes on the team. That's a ball-hog, and ball-hogs get benched.

If you don't show your work, then you're going to find you have fewer and fewer interesting responsibilities. Because "how we got here" is also a "why" and we ask Why all the time.

Grow a garden, not a tree. Let's disambiguate the word "project" from the word "system" (not "product" -- that's sales). Let's call a project a time- and scope- limited effort to create or modify some system(s) in your suite (your garden). I think it's a common mistake to mix these two up. If projects just drag on forever, then yes, employees can feel stuck. By this definition, I certainly would not want to work on one project forever.

What about working on one system forever? The problem there is that to stay busy, you're forcing attention on a system that might not need attention. At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn't. There should be no urgency to keep messing with a working system.

So let's work in one garden forever. Systems will grow; we'll prune and re-plant them; we'll give the healthy ones the barest maintenance it needs; we'll give the dying ones the sweet release of death; we'll shape and braid them together. All your institutional knowledge about how System A works will be great when you're writing System F that depends on System A, or when we're splitting System A into Systems A1 and A2. I would absolutely love to work at a company where I can work in one garden for the rest of my life. That's my dream job.

Sidebar: do not make the mistake of assuming "system" means "microservice"! That's only one of many ways to define system boundaries.

The first project is maybe a bad example but if we say the "nth" project instead then:

- not always changing members doesn't mean never changing, companies should just not want you to jump between projects all the time (except maybe as junior programmer) and you shouldn't want this either as it's hard to build up proper deep expertise this way (instead of a lot of shallow expertise)

- if you are still in the same company and the company is healthy then it should be fine if you get asked questions about your previous project for some time after you switched (through not arbitrary, just as an "emergency" thing)

The problem is that the concept of carrier and seniority is seriously messed up in the software industry. Often shallow knowledge about a lot of things is more valued then actual expertise in the things needed. Similar you will find a surprisingly lot of people believing that a senior engineer which doesn't try to get into team lead/people managing positions is doing so because they "surly are incompetent and shouldn't be hired" (IMHO completely rediculus but I have still seen it often enough, just most times in a less directly phrased way). But in the end senior software development skills and team lead skills are distinct skill sets. Implicitly pressuring someone who is good at solving technical problems into a team lead position (or the other way around) is just a pretty stupid decision from a human resource utilization POV. (This depends a bit on the definition of team lead and yes team leads also need to have technical skills but most important they need to have the skill to listen to their team and not get stuck on their opinions, etc.).

When TSHTF and we need some expertise from the original builders or operators of a system, it's a lot nicer to have them on another team than at another company. If they still work for us, somebody can make the case to them (or their boss) that borrowing them for a few hours to fix it is in everyone's interest. A lot of times they are in some kind of management position now and they kind of smile and say "Wow, I haven't thought about that system in a long time. Sure, I can take a look." and it's even kind of a fun diversion from whatever budget meeting they were in.

If they work for somebody else, the answer is often "sorry, I'm busy" or just a hangup when you explain who you are and why you are calling.

I relate to this so much. I'd offer an open letter to some poor anonymous soul who inherits one of my dumpster fires. Because I'll certainly never be given the chance to revisit anything.

Yep, my shit's broken. It was written long ago, under different constraints, for a different audience. I made the best trade-offs I could. I documented them! Though you'll probably never find my writings, and if you do trip over them, you won't understand them yet. Today, I'd scrap most of it if given the opportunity. But to whomever is now taking on the rewrite, you know that little nugget at the core which perplexes you so? The one that doesn't fit your mental model of any type of programming you've touched in your career? That there is the core value proposition that makes it all possible, and if you throw that out, you've doomed yourself to learning my wisdom the hard way. If you're at all up to the task, then the next few years of your life and the remainder of your days at this company will be spent fixing the same problems by reading the same papers and obtaining the same obscure domain knowledge. No, it can't be abstracted away. Yes, it's fundamental. No, no one else will really understand it either when you try to explain it, because no one else without the accountability you now have will sink enough of their time into building the mental model you've had to construct. Good luck.

Not sure how true that is, or if it’s more myth. Also not sure how common it is to have original devs participate in the re-write, or if this is more something they do if the original devs have moved on. If any Googlers on HN could chime in, I’d certainly find it interesting!

I used to work for a company that had an internal tool. It wasn't used anymore, everyone that worked on it was long gone, the source code was "lost".

There was some emergency and someone needed to use the tool to look at some old data. Lots of panic and people claiming it was impossible.

I opened up the executable in ollydbg, patched the executable so it worked properly, and life went on.

I'm sure the source code wasn't lost, it would have been on a tape backup that people were too lazy to try and recover.

If the source code is available then it's even significantly easier. I don't buy this whole "nobody knows how to update the code" reasoning.

If the code can't be updated, it's a matter of incentives and prioritization.

Software product is delivering value, maintenance isn't needed much, people move on to other projects[0], people forget about it, then a couple of years later a change is needed.

But how would the first response in that "couple of years later" scenario be "throw away all that investment and start from scratch" instead of "dig into how it works"? That's not a long time, really.

[0] this is a good thing, IMO. If every project needs constant maintenance from its "owner" you as a company are going to rapidly run out of resources to do new things.

(There's a closely related thing, here, though, that as an individual or a low-to-medium level manager, keeping track of these risks and low-level maintenance areas is part of your job to find ways to keep it from ever turning into a disaster. At some point - 10 years, 15, 20... maybe the knowledge gap gets way too big for the current team - then get proactive.)

I worked on an application where I knew just enough to make spot changes, while the other developer who had made the most core changes continued on with the core. Then he left.

So as more core changes needed to be made, none OMG due to Production issues, that made me dip into more and more of the legacy code, and I grokked what was going on. I wasn't happy with it, but I understood it enough to know that to make it truly maintainable would require a total re-do. But the code that was there worked, and as time went on I even took other developers' bolt-on additions and made their functionality take advantage of the sub-optimal idioms the core used, so that at least if someone else had to make changes, they'd only have to deal with one pattern. I've made wiki pages to describe it, but still any other developer who works on it ends up asking question after question and complaining why the documentation isn't in the wiki (it is; they just don't want to look).

We eventually moved to another servlet container, and I filled in the missing functionality the new server lacked, still keeping the original core code.

I've done that with two other applications at our company as well. Why haven't I moved on? Hey, I'm older now, and it's an 8 to 5 job you can't easily find any more.

Agree with the theory of programming as theory*, but here's the problem with that essay in an era of software-as-a-services and platforms:

A SaaS offering or an Enterprise runs on a stack or assembly of software, each of which has its own Theory.

The problem with no churn is that most of the Theory is not in-house, it's elsewhere, and evolving. If you can't tap into that, you're falling behind. If you don't know it, you can't remix it into your own.

It's just math to convert an org's "refresh rate" of outside-in knowledge (aka churn) into the average lag of that organization behind current best theory and patterns. A 10% churn means an average of 5 years out of date. A 20% churn means at least a fifth of your org is 5+ years out of date.

A couple key antidotes, though (and there are of course more):

Realize that knowledge transfer from (a) outside in, then (b) among, is vital to theory maintenance and evolution.

That means (a) have a great "always be hiring" pipeline of fresh Theory and build processes to make knowledge sharing of outside and inside theory frictionless and routine, and (b) have a great "always be learning" approach to exposing developers not just to the internal theory but to the best and most up to date outside theory.

The better an org is at (b), the less it needs (a). The worse it is at (b), the more it needs (a), but -- crucially -- it must refresh with candidates who know more recent theory than the ones doing the hiring!

Churn is not bad, that word is spin. Internships are a thing, so are externships (firms should do more of this). People, and theories, grow by exposure to novel challenges. Circulation is good for the whole ecosystem of software development because we are building a collective Theory bigger than any one org.

This remixing of ideas is how our collective Theory evolves.

* Most recent decade of my career involved a lot of evangelizing to enterprise management that developers are not project teams to form and disband as fungible resources. Think in terms of capabilities to garden, not projects to remediate.

And, in fairness, that is a reasonable thing to want. The problem is it's not clear whether or not that's possible, and it is clear (but completely ignored by the people who ought to be aware of it) that our current software development methodologies don't even come close to allowing it.

So I guess it doesn't actually work for plumbing problems either.

The problem is, just like how at the start you find lots of devs who don't want to write unit tests, you usually have a lot of devs who resist writing (and reading) docs. They'd rather interrupt someone to ask, or thrash around, or move onto the next task the moment some code is written.

What helps:

1. Keep your docs alongside the source, in version control. This lets you gate code reviews/merges on docs being updated. Someone changed the way the code works but didn't update the docs? Then the PR is rejected until it's fixed.

2. Leading by example. If the TL for the project updates docs with their code changes, and visibly spends time gardening the docs, then they can't be accused of hypocrisy when they demand the same standards of others. Praise quality docs work when done by others.

3. Make the docs beautiful. Let people have pride in their work. It's so easy these days with great static site generators.

4. Answer questions by linking to the docs when possible, not ad hoc Slack/email/ticket discussions. If the docs don't tell someone the answer ask them to hold on for a moment, update the docs quickly and then link to them. Obviously this requires a fast / lightweight turnaround process on review and merge of docs changes, but it can be done.

5. Avoid wikis. Quality requires ownership, someone who is directly responsible and can enforce their will. Wikis don't let you do that so they become a commons, and you end up with a tragedy. Use Markdown or equivalent in a VCS repo so you can refactor docs as they expand, make sure there's someone (e.g. yourself or a trusted lieutenant) who is expected to spend time paying down docs debt every so often.

6. Docs bugs should be a thing. File them. Expect people to fix them.

7. Make it clear that people are expected to read the docs, not just write them. If someone is regularly asking colleagues questions that they could have answered by reading, ensure that's treated like other performance issues i.e. their managers know and the need for improvement is clear to them.

There's more that can be done but these things do work. The biggest problem is always passive resistance. Just like with unit testing or code review, if a culture of avoiding it grows then you'll find it hard to introduce, people will passively resist because it's not as much fun as just banging out more code.

With these approaches the robustness of a program to churn can be improved a lot, you can scale teams up quicker and productivity improves.

Samba is a great example of how this is necessarily an imperfect model, and definitely not an optimal process:

- https://news.ycombinator.com/item?id=488089

Development as theory-building is intended as a metaphor to understand the impact of the collaborative mental model that is created as part of any complex system, and the losses incurred when this mechanism is not appreciated.

Anecdote: Worked for a company that had a large software component (graphical design tool) that had been written in C with a very complicated object model/macro layer, file system format etc. Zero documentation, hadn’t been updated in a decade and not a single person remained who understood it or had done more than simple bug fixes. Couldn’t rewrite as customers had huge investments in assets built from the tool.

One principal dev and maybe 1-2 months? And we had the theory of mind back, explained to everyone else and a new team of 5-6 devs working productively. Component was brought into the 21st century, and this time some documentation was written.

Naur's argument is first that there is a theory at all, and that the question of whether it can be transplanted to another person is really unanswerable. We can attempt to answer it, or build confidence towards an answer, by doing things like writing/reading design docs, having experience in the industry that shows us how other eng tend to think and design, talking to the person who wrote the code, testing our theory against the code, etc. But we can never be certain of what is in the mind of another person.

Anyone whose ever taught anything to a student, I think, has come to see this. It's hard enough to teach people how to do a thing in response to a known circumstance. The real trick is to teach them well enough to have them extend their knowledge on known circumstances to new and unforeseen ones.

And Naur, IIRC, also argues that churn is inherently bad, and that programmers are not replaceable cogs, because one of the valuable contributions they are making is holding, in their minds, the theory of how the code solves problems:

> More generally, much current discussion of programming seems to assume that programming is similar to industrial production, the programmer being regarded as a component of that production, a component that has to be controlled by rules of procedure and which can be replaced easily

That if a new team takes on a codebase, they are rebuilding the theory and reviving the program.

The new theory may not be exactly the same as the old but it is probably still adequately compatible.

In a sense, the new team is reverse engineering the theory, from the source code. Even though this is a much easier task than doing so from a binary.

- Get the “change engine” running smoothly (productive and reliable local dev environment, reasonably fast/reliable build/deploy jobs)

- Get people understanding, documenting, and teaching others the high level functionality and architecture

If you skip those steps, and hop straight to new features, you’re just asking for regressions, outages, poor architecture/design decisions, and just a very slow pace of work. But invest in those 3 up front and you can get to a point where you’re evolving the product well, with few regressions, at a good pace, even without vets to teach the newbies.

There are many open source projects with great documentation and they manage to keep abreast of changes.

This attitude among programmers that documentation is hard, so don’t bother is just that - attitude.

The frequency of job hopping is inversely proportional to the time spent working on a career.

Gaming the system for a large salary early on without much experience only hurts the employee with burnout and getting replaced with better developed talent more deserving of the role.

I think this needs to be said more carefully. For instance, problems could become manageable when a well-thought-out mental model is built. On the other hand, they could become easy when the problem itself is simplified. For me, the e-voting problem fits quite well in the picture where thousands of systems are being built with cute mobile UI but, in the end, are really holding on to the assumption: trust us, we will be honest.

I locked myself in a really hot closet behind the cafeteria with several pieces of our soon to be purchased hardware and started coding. The whole thing ended up amazing and I am still proud of it to this day. =)

Some of my primary goals where simple: No manually edited configs, ever. If you swap a mother board there should be no mac address tracking, no serial number editing, etc. If we can detect it automatically we did and updated the machine database automatically. Hardware techs could simply walk out, swap a drive, and walk away and we would track that.

So we wrote a tool called vex that would export all the machines various bits and bobs. Anything we could access, serial numbers, mac addresses, etc. So once the machine was running we knew where it was, what it was doing, etc. The biggest challenge was getting it running in the first place. The plan was to use DHCP option 82 which includes port information when forwarding DHCP requests to an upstream server. Thus our dhcp server (written by an awesome coworker in like two days) was 100% stateless. It knew the IP range for the datacenter, the ip range for the rack, and the port that the request came from and could respond with a static IP every time. Swap the mother board and it wouldn't matter because the machine would still be plugged into the same port on the switch. Each machine would then request a tftp boot image from the server listed in the dhcp reply. This server was smart, it would look in the machine database to see what state the machine was in and where it needed to go "next" so to speak.

Step 1) Machine has never booted before. It would go into an identify utility that would allow the installer to verify that each machine was plugged into the right ports. The tech could watch the ipmi identify lights on the machine as it activated each in the rack in order and if any where out of order network cables needed fixed. From this point forward we knew that port 3 was the 3rd machine in the rack on the prod and ipmi networks. Once identified the machine would automatically reboot.

Step 2) The machine would run memtest86 to ensure that memory was valid. Sadly I didn't have time to make it automatically update with results and reboot so this step required manual verification when I left. A engineer would check IPMI and mark the test successful.

Step 3) The machine would run stressapptest to push the CPU and memory to the extreme. This allowed us to ensure that they would not overheat and that the rack could pull the load needed to get all the machines to 100%. Once done it would mark itself successful and reboot.

Step 4) The machine would run fio to ensure that every attached disk was in good state and fully functional. Again, it would then reboot automatically.

Step 5) The machine would install our default operating system assigning hostnames and such via parameters from the database.

Lastly once the machine booted into its "real os" for the first time it would setup puppet, install all the back packages, look into the machine database for its assignment and then install all the various bits ad bobs required for it to do its selected function (hadoop, web server, dhcp server, puppet server, etc)

Everything was driven by the machine database, and you could mark a machine as needing burn in and it would repeat the testing steps. Super useful if you suspect a machine is bad in some way but don't have time to debug. Once running vex would export all the details of the machine as well so we could track failure rates, hardware events, etc.

Best part is that the lab ran the same software as the datacenter so we could always test the various bits of this process to see what worked and what didn't. In fact I added several optional steps to do things like get performance profiles and build all kinds of cool graphs from tests on demo hardware.

I ended up bootstrapping three datacenters and all it required was an "Ark" which was just a single server with puppet, dhcp, tftp and audubon on it ready to setup all the other servers in the facility. During our initial power up we had a DHCP server doing something like 100-500 requests per second sustained over the course of like a half an hour (until the memory tests started)

Sorry, not sure why that memory jumped out at me but it seems you did something alike and it reminded me of how much fun it was to be involved in mine. =)

But the content more than makes up for the style!

People leaving means that there's nobody there who can fix the decay when it happens, though.

1. It takes more time for the rot to show.

2. Twitter devs were amazing at building really solid software

3. The culture of twitter was really that 1 person did the work and 9 others watched, enough of the productive devs stayed.

4. The effects of rot are overestimated.

I have no insight into how twitter worked as a company before Musk's takeover though, and I still think it is quite likely major problems will occur in the next couple of months.

The solution to the problem, at least according to the article, isn't "no churn ever", but "low enough churn that the newer developers acquire an accurate mental model of the system."

High level capitalists (such as VCs) treat rare people unfungibly e.g. Paul Graham “the success of a startup is almost always a function of its founders” or http://paulgraham.com/founders.html

Edit: also fungibility is by no means a capitalist idea, and it is a central concept to some non-capitalist doctrines. What is equality?