Reflecting on the Soul of a New Machine
dtrace.org
dtrace.org
I saw this zipping by on Twitter. Heya I’m Tom West’s daughter and I always pop onto threads about SOANM to bang the drum of good work-life balance as well. My dad was an amazing man and we got along well, but he was a pretty distant dad and basically left the raising of my sister and me to my mom who might have wanted to do something professionally as well.
The things he did at work, and his and Kidder’s ability to talk about them, were quite interesting and have lessons that can be relevant in many different timelines and lifestyles. But you know, I wound up moving to Vermont and helping people use technology to solve problems in their lives. You need all the things–technology, an understanding of people, and a life–and my dad possibly only had two of those at any one time. Thanks for writing this.
I’m not particularly talented, far from it actually, but many times I’ve worked in teams with non-junior people who were tasked to do some portion of a project, and simply delivered garbage: a terribly over engineered solution in a component where over engineering was actually very dangerous (e.g. doing distributed systems using several small bespoke moving parts rather than a single solid leader election mechanism, or even better getting away with something leaderless and more elegantly stateless, to give a recent example).
In those occurrences, I inevitably “broke it down to pieces” by keeping pointing out all the exact scenarios where the code delivered would fail, until the other person really had to finally move to a solution much more similar to the one I had in mind (much much simpler). Sometimes this required an embarrassing feedback loop of 10+ round trips, where every time I’d just look at the newly delivered code say “this doesn’t work in a X scenario”. And I was not in a better position from a knowledge perspective, we were essentially peers.
I felt like a prick and definitely felt resentment from the other party (especially since code reviews are public), but at the same time, how could I have acted differently? The end result was really orders of magnitude better, more robust and smaller.
There have been colleagues though who do exactly what you’ve described. That brutally painful loop of breaking pieces of an overly complicated solution over and over. My unfortunate confession is that I usually end up giving up and let them fail on their own. I’m always willing to help, but if someone doesn’t want help, I won’t spend much energy forcing it on them.
What happens is that the code will be released, some user will hit the bug in the field, and the issue will be escalated to the engineer, who will proceed to spend entire days or weeks to troubleshoot the problem, and then just put a dirty patch that fixes the specific issue, leaving all the other corner conditions still exposed and ready to explode. At that point, major redesign of the solution, which would have been possible at the code review phase, will just be deemed too risky by both the engineer and the management.
On top of this, Management, usually ignorant to concepts such as “over engineered solution”, will praise the heroic effort of the engineer who spent days troubleshooting the issue, clueless of the fact that it was a completely self inflicted wound because the engineer decided that the best way to calculate 2+2 is to really do sqrt(2^4) and then realized that the user really wanted to do 3+3, so they’re going to put a special case to evaluate 3+3 as sqrt(3^3+3^2) rather than just doing the damn x+y.
Nobody will even realize that the issue could have been completely prevented in the first place by adopting a solution that was much easier to reason with on the whiteboard, rather than a house of cards of custom logic.
I’ve seen this script dozens of time.
Circa 1980, around the time of the events in the book, the time delays involved in making design changes to boards built from MSI building blocks were bad enough. Physical wire mods to a board took hours or days. New boards could be made in weeks or months. There was also some flexibility because parts of that design used PALs, which were easily reprogrammed.
But in today's world the time cost of a re-spin of a CPU or a highly integrated SOC chip is huge. Months for simple changes, years for complete re-designs. Just look at the time estimates for hardware changes to x86 chips to mitigate Spectre and Meltdown. Some months for new microcode; many years for new silicon.
Nevertheless, it's perhaps easier to let things happen and afterward think "I told you so". Because, unfortunately at most companies, if you try to impart those lessons earlier in the process, e.g. at design reviews, you quickly get the reputation of not being a "team player". Sounds crazy, but that's simple human nature.
Other than that I haven’t found any great solution, but I have found it stressful and counter-productive to try and do EVERYONE’S thinking for them. They do not learn in that case, they just continue to outsource their thinking to you.
e: also 10+ rounds is excessive for a peer review imo, after 2 or 3 I’d be having a higher level design chat with their lead/manager.
Excellent code reviews take time and attention, which is why you need auto code formatting and $lang proverbs to take the uncontroversial stuff off the table early. Then you still have brain power left to give value from the review.
Also: work hard to earn a reputation as a fair but tough reviewer. Then when you need to be a prick, people will know it's important and let their ego down.
Is there a difference in that? If I sign up to do a job, that's quite different to having something assigned to me. Things I sign up for are things I have chosen, based on my abilities and enthusiasms; things that are assigned to me.. well, not so much.
my conclusion is that people don’t really like simple things and they believe that complex=good and it’s a sign of how good/smart you are. so people are suckers for complexity. nobody is going to receive a medal or a promotion for well written and simple code that “just works”. you won’t know it’s there.
now, the reaction you’re describing comes from the “i’ve seem some shit” syndrome where not only you can see the layers upon layers of unnecessary crud but you can also see how it’s going to break in the future. it’s terrifying.
There's the problem, right there. Using a code review to expose a design flaw.
I worked at my first startup starting in 1988, not long after SOANM came out. Most of the engineers at this company read the book. I was struck by the similarity between the situations and dynamics described in the book, and what I was observing first-hand.
Our approach to engineering was academic in nature. This was remarkable because there was only one PhD in the bunch -- me, the most junior person -- and everyone but me had a spectacular track record of designing, building and delivering innovative products.
We started with whiteboard discussions. These would go on for hours at a time, across multiple days. When we had the outlines of a workable idea, someone would go and write a design document. And that got reviewed extensively, over a few days. The "breaker" sorts of analyses got done at this point.
So by the time we started implementing, we knew we had a very solid design. From that point on, bugs were just bugs. They weren't these initial cracks that revealed a rotten foundation.
I thought that this approach worked incredibly well. And I was dismayed to observe that it wasn't practiced elsewhere. I did try to institute this approach a few times, but it doesn't really take hold unless you have other senior people who want to adopt it. And at my last startup, we had smart but young and inexperienced developers who were bought into Agile, and code reviews substituting for design reviews. The predictable (and predicted) hilarity ensued.
Engineers, once they are hired and passed the interview, are mostly seen as a commodity, in the sense that, in the eyes of management, every non-junior engineer should exactly be able to come up with the correct idea (modulo a short phase of code review, just in case), so the output they produce can just be as equally produced by a peer (just like a commodity): in that world view, design meetings spanning multiple days don't do much, and they just make you lose time against the current jira sprint.
I would love to work in a place where management understood that collective intelligence is almost always superior than the initiative of a single person, and thus it's very important to have a design phase where as many engineers as possible are involved.
Every single time I had to lead a significant project, I always spent so much time involving as many people as I could find to discuss the idea, constantly self doubting my ideas. Incidentally, very few significant bugs have occurred when the project was delivered, in contrast with the example I was quoting, where the thing was developed completely hermetically by another non-junior engineer, who failed to account for many non-to-rare corner cases.
https://www.agilealliance.org/resources/experience-reports/s...
A few years later, in 1986 to be specific, I started a job at an OCR company. My cube was directly across from the office of Carl Alsing, the guy who managed "the microkids". I didn't put two and two together at first. Alsing was one of the people who interviewed me for the job, and I got to know him a little bit for a couple months before someone told me about the book connection.
So I went to the library, got the book out, and read it again. When Kidder got to the part where he introduces Alsing, it was amazing. In a paragraph he captured details about Alsing that I hadn't even realized myself (eg, his personality projects the image of a smaller man, though he was actually decently tall).
The other thing he got right was Alsing wasn't much for schedules and tended to work on things that interested him instead.
Anyway, having verified that Kidder had indeed captured a lot of truth about Alsing, it gave me trust that the rest of the book was similarly accurate, and not just sexed up in the process of storytelling.
My perception of the team leader hero, Tom West, was also permanently altered by an essay written by his daughter Jessamyn (a quasi famous technologist in her own right), who described him as basically a depressed alcoholic.
In the late 1980s I got to learn a ton about an operating system that had threads (tasks), virtual memory, shared memory, rich filesystem ACLs, symbolic links, and a bizarre shell language that forced you to do all your looping via recursion.
I learned how to program in C on that machine, I wrote some of my first assembly on that machine, and I spent hours pouring over thick black three-ring binders filled with manuals on OS usage, system calls, and the hardware architecture.
Friends and I competed to learn arcane details on how the system worked. My main claim to fame was I figured out how to share memory between unrelated processes, and how to use the atomic-test-and-set instruction to safely synchronize access to it.
One time while we were studying the virtual memory system, a friend wrote a program to see what would happen if you allocated the entire 4G address space of virtual memory and then randomly touched different pages of it. Predictably, the storm of page faults ground the machine to a standstill, causing our lazy sysadmin to have to put down his his copy of Datamation and address the gathering throng outside his door.
I have this recurring dream that I'm back at college for some event, and I wander into the Computer Center and look around. In the very back, there is a dusty 'Dasher' D210 terminal next to some stacked up old furniture -- plugged in and emitting a dim green glow. It's hooked up to our MV/10000 Eclipse, which I imagine is still in the basement of the Library Building humming along after all these years. I try to log in, but I can't remember any of my old accounts and passwords.
You knew when an Ada compilation job had been started because the other terminals would start getting sluggish...
At one point the CEO injects a critical piece of wisdom by telling the engineers they are not allowed to use a mode switch to preserve backwards compatibility.
In any system, the presence of mode switches often points to where things will go wrong. If a system has two modes, then inevitably, one of them will be more unloved, untested, under documented and unreliable than the other.
* Roll out the code with support for a new feature
* Flip the mode switch on a few nodes to use the new feature
* Wait a while, watch for anomalies
* Enable the new feature on more nodes, until it's on all
* Disable support for the old feature (another mode switch)
* Eventually, roll out new code with the old feature entirely removed
It's not quite a whole-system mode switch as in SoaNM, but that's kind of the point. In a distributed system actions do not affect the whole system all at once, so different rules apply.
Also the Mushroom Theory of Management. Easy to encounter that firsthand. "keeping them in the dark, feeding them shit, and watch them grow."
https://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine#Them...
A couple of points struck me then that may be of interest to the HN readership.
First, the instruction set originally proposed by the architect was very VAX-like. It would probably have been great for those writing in assembler. But the end was approaching VAX-like instruction sets.
Second, retaining the old Nova/Eclipse instruction set meant that one kept three general-purpose registers (well, sort of general-purpose: by convention, I think the stack pointer was in AC3). One also had no register-relative byte addressing.
I liked the Novas and Eclipses, and learned a lot working on them. But the RISC-based UNIX machines took over from the minicomputers pretty quickly.
[Edit: I don't think I've used an MV since about 1995--take any remarks with a grain of salt.]