Accidentally Saving the Day
essays.jwatzman.org
essays.jwatzman.org
I recently joined a budding open-source project (PhobGCC, a custom gamecube controller), saw that it was using a rather overbuilt full kalman filter, and within 2 days of work rewrote it into a less generalized filter that ran 7x faster (later improved even further) and performed even better at its filtering task.
It was super basic work, despite it having been almost a decade since I last studied control systems, but it made a really big difference for the project.
Keep it up.
What? Your contribution to your company was actually recognized? I've never heard of that happening before. Where do you live? I'm guessing it's not the USA.
It has taken me a decade to realize this is the key to success. Just do your very best work, as often as possible, and let the rest figure itself out.
I accidentally created a data warehouse that became the necessary backbone to launch a massive new org (100M+ revenue). I was building it for a relatively small near realtime Elasticsearch cluster to generate data reports for an application. But I thought, gee, if I suck in a bunch of other data from other sources and clean it, I might find a use some day. Little did I know that someone else would piggy back on it to build a POC for a giant business expansion.
Last year I pulled off a minor coup by making it cheaper and easier to run a big chunk of our code. One of the senior people was trying to get me to toot my own horn but the actual work was like one week of cleverness and five weeks of book-keeping, and so it felt too slog-like to celebrate.
But a couple weeks later when I was doing some housework it dawned on me that it was 'only' six weeks of work because of a bunch of fairly thankless things I'd invested a great deal of time in over the previous 18 months. I'd been dribbling out appetizers for a while, this was the first big dish, and I'm not sure it's the main course, so it's not like I missed my big chance, but I probably should have honored it more.
I'd had an initial 'I told you so' feeling in the heat of the moment that pushed down as feeling petty, and delivered instead a fairly lukewarm "this is what I am talking about." This is, in fact, one of the things I'm talking about, but I think I'm past trying to recruit people (all the fun people to recruit are gone). I'm sadly just following the Campsite Rule and splitting my priorities between things that help me do my job, and things that might help me do the next one.
In that respect, I've been chasing a dragon (two, in fact) for the last couple of months instead of acknowledging that paragraphs like the one above are a pretty good indication I should be focusing all of my energy on job hunting instead. Wisdom is on a continuum and there's a lot of room still for foolishness even if you've got a lot of things figured out.
There were a bunch of more senior developers who I hugely respected and learned a lot from, including this guy who I'll refer to as $SMART_DUDE who joined at about the same time as me. He knew his stuff inside out when it came to C++ and I remember being absolutely fascinated by a talk he gave on memory management and different types of pointers one lunchtime. It's no exaggeration to say that I literally still refer to things I learned in that talk today, almost exactly twenty years later.
There were some pretty big personalities in that company and I was seriously intimidated by my line manager, who always seemed like he was pissed off with me. I spoke to him when he interviewed me, and then barely ever until my performance review came round nearly a year after I'd joined the company. I remember speaking up in a meeting once, him snapping at me, and I just shut the fuck up.
He surprised me in the performance review. We had 5 levels of bonus. I thought I was decidedly mediocre in his eyes but, out of nowhere, he gave me not the top level, but one down, which was only slightly less than the top level (we had a reasonable degree of transparency about percentages involved).
Still, I didn't think much of it and had started looking around for other roles. One of the motivators is that one or two people I really respected had or were leaving, including $SMART_DUDE, who'd handed in his notice. There was him and another guy, who I'll call, $SMART_DUDE2, who I thought - and still maintain - were just the real deal. I thought they were better than the people they worked for, way waaaaaaaay better than me, and better than everybody else in the company, although, to their credit, I think the people they worked for also though the same.
The other motivator was that, although I was earning slightly more, and on top of this the company had covered my rent for the first 12 months, I was still financially worse off than I had been where I'd lived before on the lower salary (higher cost of living in the new area).
Anyway, in the meantime I started hooking up our client with the backend using XML rather than CORBA. Not even using SOAP. I just realised there was an endpoint that we could communicate with firing XML documents back and forth according to a protocol that we agreed in a late afternoon/evening design session that would allow us to load and update whole trees of objects, rather than just individual properties (surprise, this was mind-blowingly slow), without having to make any major code changes on the back end. Really not that big a deal.
Whilst this was all going on I landed a contract. It paid something like 1.8x what I was earning after tax even with something like two months holiday factored in. It could have been more but I was so overawed by the low end of the pay band that when they asked me what I wanted I said £Y and they accepted without an argument. I was an idiot. Still, because I thought absolutely nobody would believe me, when I handed in my notice I told the company my new role would pay 1.35x my salary. I was an idiot for the second time. If I'd not been an idiot at all I could probably have got 2.2x.
It was at this point my line manager - bear in mind by this point (14 months in) I could still count the number of times we'd spoken on one hand - dropped the bombshell of, "Oh no, I mean you're really great. I think when you leave we'll feel the loss more than $SMART_DUDE. Let me see what I can do on salary." WTF?
I still maintain that $SMART_DUDE was a lot better and smarter than I was or will ever be, and I'd hire him in a flash - literally no hesitation other than to ask him if he might not get a bit bored by the work we're doing.
I still don't really get it, even now, but I suppose I delivered pretty consistently during the time I was there, didn't make a fuss (anyone who knows me now would find that very hard to believe - there's a history here that's too long and boring for HN), and solved a business problem that they had quickly and pragmatically without worrying too much about whether it was the best option from an engineering standpoint.
My advice: you don't need to be super-smart, but be just a little bit hungry, able (and stubborn enough) to deliver, and don't stand too much on engineering ceremonies.
In this way, it’s not unusual to 2x or more your income in those first couple years across many different roles (even non technical). This is the reality of becoming what is often referred to as junior->senior.
Or are you talking about a real life example of a COBRA CORBA implementation?
This is not how projects should work, and we should stop perpetuating these heroic stories. The actual superstars are all those people doing the less-than-glamorous grunt work. Planning things on time. Getting the infra ready, stable, well-tested, documented. Making schedules that have margins. The people carefully designing things so they are robust and don't break left and right. Thinking through their systematic testing. In short, preparing everything so that heroic acts are not necessary.
Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. Pushing all this against pressure that they are just delaying everything and should not worry so much and be more positive and agile. Imagine they had succeeded and everything had gone smoothly. This would have been much better for the company (since the existential risk wouldn't have existed in the first place). But that somebody wouldn't have ended being the superstar with the promotion and the bonus. It would likely have stuck to have been the guy just delaying things with their worries. That's what needs to change in sw eng culture.
I tried to get at this in the last section, that it's not about going out trying to be a hero, and it's unfortunate it didn't get across. That was my point that you shouldn't go look for company priorities to fix, to be a hero. Just look around you, see what's going on, and do good shit. Sometimes yeah that does involve heroics, sometimes it's just gruntwork.
I don't want to go back and edit the essay now but I think it could be much clearer indeed!
> Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. [...]
Disagree here, though it's maybe not obvious from the bits of the story I told why. ChaChing was one of... maybe three or four dozen experiments like it. All of the others failed. There was no reason to believe ChaChing would be any different, and if it did work, even the best-case expectations were well below what it ended up doing. It really was a lightning strike. So it would have been a huge waste of time to do all of that work, delaying Jabberwocky, for a bunch of things which never ended up shipping.
The "to be a hero" is the problem, you can always feel good about solving hard problems and finding your boundaries, but "avoiding a crisis" is a thankless job & we should thank those people more than the "only I can fix it" people.
The core problem for me is that the constructive "avoid crisis" people are baked out of these sort of events, rather than born risk averse in the first place.
> Just look around you, see what's going on, and do good shit.
One of the biggest side-effects of WFH was that this whole "who's around" mechanic has faded away without a proper replacement.
Most of my satisfying work came from random interactions like this (not the most impactful or the most purposeful).
I was a principal engineer and most of my 10 AM to 2 PM was taken up by meetings which mostly involved prevention of work duplication or to stop someone driving down to where I knew there was a dead-end (like, "I see you're using alpine docker images, why?" or more management boundary stuff like "the difference between these two ideas is ~140k over a year for all customers - can we agree that the expensive one is better for customers, because they don't see a 12$ month bill and it costs more than 140k$ to build the cheap one in NRE?").
Most of that work is tiring and always unproductive - the actual work is done by smart people who answer my questions carefully, the dumber the questions, usually the better the results.
But sometime around 3 pm, I'd be line to get a coffee and I would run into a PM or someone from support or dev who would have a question for me which sounds nonsensical at first - but having an hour & half to go down that rabbit hole looking over someone's shoulder usually ended up with mostly "hey, I know what's happening, I've seen this before" or the classic "hmm, that's strange, how could that even happen?".
Six to ten weeks later, I would be in a meeting where I would get some sense of "deja vu" when someone describes their approach and reach back to that day & that problem, to ask "how would you detect X" (something like "machine reboots, comes back with same hostname, but different IP" - would you restablish connections to IP or hostname, what about krb5 ... yada yada).
That part is the real value as an engineer, but entirely unappreciated from the management's point of view.
With WFH, I would go get a coffee, go for a run/walk and as good as it was for my state of well being, I would eventually get pulled into these crisis situations Thursday at 9 pm instead of Tuesday at 3.
I like WFH for the work I actually have to do, but this sort of temporal serendipity is not easy to recreate on Zoom.
That's the most controversial thing at work too.
It comes up so often because it is a measurably good idea when you look at container sizes or startup times (gzip is terrible, terrible way to package container images).
An engineer tries it out, it works great and it does because alpine is lean, fast and generally good.
I work on performance, so any decision to make things slower gets routed through my desk - the perf buck stops at my desk. Now it's my thankless job to squash out her initiative and as gently as possible (there are lots of smart people in the industry, but being kind has a better ROI), so that it is not a "this is policy" form letter appeal to authority, but walk them through the entire list of tickets in my notes.
Because we hit a bunch of SEGVs with the JVM and the JDK team basically closes them as WONTFIX when reported.
And this sort of decision has a shelf life, I can be right for 18 months and wrong the week after - so this is not a hard line.
This specific thing is definitely not a permanent thing, but a temporary headache - the JDK team is working on Portola and that'll fix any issues they had.
https://openjdk.java.net/projects/portola/
If you're going to install glibc as a workaround then you're basically giving up the space savings anyway.
We still break things in production, but counterintuitively I'm the happier when the problem is bang in the middle of code I can commit to, rather than deep inside musl -> jdk interactions on what happens with a longjmp on a SIGSEGV.
Also the turnaround for the CVE reported was much faster across teams if everyone picked the exact same base image - standardizing on Redhat Universal Base worked out (& also AquaSec scans etc is easier if 6 different products have the same image down there).
Amazon Corretto does have alpine images, so this is sort of a 2020 advice with rapidly declining value (I'm funemployed for 2022, so my current "weakly held" opinions are entirely about rust syntax).
>It isn’t about creating bigger and bigger opportunities for yourself — it’s not about selfishly inventing self-serving projects. Rather, it’s about getting better at recognizing and taking advantage of bigger opportunities which are already there and just making things happen.
I remember being at an all-hands company meeting years ago and listening carefully to what the people who were given the top awards got them for, and about three in a row were for diving in, working nights and weekends to come up with emergency fixes to save projects or products that had gone seriously wrong, and these were different projects. It was so blatant then even some senior managers finally noticed "Hey, we have a problem".
One which you've zeroed in on is "don't require heroism". We have a lot of these stories already. (Unfortunately also a lot of exciting stories of "heroism".)
The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. IMO this is also really what "hero" stories are about: something goes so wrong that either people are finally forced to either look for once, or forced to bring in someone who will look.
As a senior engineer, for the past couple years I've tried to pitch parts of what I see "just there" intentionally, either before it becomes a crisis or just as a potential win. Sometimes people still don't see it; maybe they're stuck in "one year of experience, ten times" or sometimes they're just built out of stop energy. But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".
Author here. Yes, this was exactly my point!
> But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".
Indeed. I've mentored a bunch of otherwise-competent engineers trying to get to the next level, and my advice is similar: ask yourself "ok, now what's next". After doing the cool thing, there's inevitably half a dozen other things around it, and things might be obvious to you (as the now-expert in the cool thing) but not others.
Your original example shows you doing extra hours and long days to achieve this: if this was your point, the example is terrible.
I do agree with high-impact work just being there, but finding the time to do it other than "on your own time" is where the tension is between management and engineers who have a different belief of what the highest priority is.
Yeah, I addressed this in another comment here. I attempted in the last section to separate the ideas of "I heroically dived in" and "I took an opportunity in front of me" -- you shouldn't go looking for heroics. This didn't quite get across, and I totally understand why. Ah well.
In my case, the house is the business goals of the organization. In your case the house is the product made by the organization.
It also puts a bunch of people who have actual life commitments (like kids, health issues, family issues or whatever else) at a disadvantage simply because somebody can afford to spend the extra time on something. And if those are the people who properly plan and execute, they get a double whammy.
The moral of the story should not be that when there is an opportunity to make a significant contribution by exchanging your life for your work you do that, but that you should be able to excel at work and be recognized for it without exchanging your life for it.
Sure, the author of the post does not have anything to do with most of the management failures, except the failure to not work extra hours for a very artificial deadline (one does not need a hero to save face if you've messed up yourself — exposing the failure is usually a net win).
Looking back, I am still pretty amazed at the turn-around I helped facilitate at a bunch of smaller companies.
Presently, I am an employee, and this is one of the most painful uphill battles I have ever had to fight. I have tried warning colleagues about issues that seem obvious to me, but face exceptional pushback all the time, only to have things blow up exactly how I predicted a few months (or sometimes weeks) later. In my own work, people generally let me be and do good work, but as soon as it becomes a group effort the “damn the torpedoes full speed ahead” minority steam rolls ahead and makes a mess of it.
The worst part is, I know exactly what the problem is, because I’ve been on the other side of it: organizations pay way more attention to external consultants than internal resources. 90% of the advice I have companies for how to change was suggestions gathered from their own employees, slightly curated and abridged into something easily digestible.
And so, while I’m exhausted from trying to change things from inside, when shit hits the fan, I step up and do the old mercenary thing to solve problems at the last minute and great expense. I keep trying because I do have a ton of exceptionally bright colleagues who I love working with, as well as an actually world changing technology. We’ll see… after this next demo… maybe we’ll take some time to breathe and get back on track.