The danger of hidden functional roles
surfingcomplexity.blog
surfingcomplexity.blog
>I suspect this is failure mode is more common than we realize: there is a process inside a system, and over time the process comes to fulfill some unintended, ancillary functional role, and there are people who participate in this process that aren’t even aware of this function.
Ah, Hyrum's Law showing up in an unexpected context, yet again.
With a sufficient number of users of an API,
it does not matter what you promise in the contract:
all observable behaviors of your system
will be depended on by somebody.These people often do a host of small tasks to that noone is really aware of (themselves included; they don't see how it's special or that nobody else knows about it). It can take the organization a long time to recover and realign.
However I couldn't make heads or tails of what systems this was or how it related to us.
After a bit of back and forth, the customer said "ask Glenn, he always fixes this for us".
Glenn was of course on vacation and didn't respond to messages...
A few hours pass while we make zero progress on this issue and customer increasingly annoyed, when Glenn finally responds to my messages.
Turns out that on a different server from where our software is installed, there was a Java-based message broker service running that needed occasional kick in the pants (IIRC stopping, deleting a file, restarting). We're not a Java shop, we don't use message brokers.
For years, Glenn had been keeping this service running and updated for our customer, as the customer didn't have any internal resources that was capable of handling this. And nobody else at our shop knew this. Why? Well, he just did what had to be done and moved on to the next issue at hand, as he always did...
There's a word for this kind of thing: "covert agency".
I don't remember where I first saw it, but looking back at my own career, being able to exercise some covert agency made miserable jobs worthwhile in some ways.
Even at Amazon, when I left work I left behind all ways to communicate and my bosses respected that. Nearly a decade there and never did a boss try to reach me on vacation.
(I've also seen a person who believed he was indispensable and thought because of that he could behave like an asshole. Very satisfying to see him get fired.)
Maybe time to look for a new employer.
I am a third year junior associate at a law firm, it would interest me how you transitioned.
Edit: And you have to take those weeks in the summer as well, you can't take 4 consecutive weeks in fall, winter or spring. So it's really inflexible. If you want to go hunting in fall, and skiing in March, tough luck, in Sweden everyone "needs" to take all their vacation in the summer for some unknown reason.
It's like you have to retire every year and go through that difficult transition to a jobless person. Many people just end up drinking too much.
Even taking 2 weeks off consecutively, it would not be unusual to require "special" permission for.
Yes, it's terrible.
But we know different people! I don't know a lot of people with six figure engineering jobs.
Apparently here's what the US Bureau of Labor Statistics last said about it:
> One reason for this is that American companies offer fewer vacation days. According to the Bureau of Labor Statistics, 76 percent of private industry workers (who make up 84.7 percent of all workers) receive paid vacation days. After one year of employment, these workers were granted 10 days of paid vacation, on average.
> This number grows modestly as years of tenure with an employer increase. In 2017, the average worker with five years of experience at a company was given 15 days of paid vacation and the average worker with 20 years of experience was given 20 paid vacation days.
https://www.cnbc.com/2018/07/05/heres-how-many-paid-vacation...
Parental leave is commonly 6-12weeks, especially among the HN audience.
[1] https://www.merriam-webster.com/words-at-play/misuse-of-lite...
At the end of the day the company has to be willing to make the investment in adequate documentation and training backups. The only time I've ever seen that happen is when people leave permanently, and of course it only happens AFTER they leave permanently. Documentation and training are cost centers, after all /s
I remember reading about similar experiment in biology:
If you have a petri dish full of bacteria, and you gradually introduce antibiotics, not only will your bacteria population grow resistant to the antibiotics over time, but there metabolism even might become dependent on it.
You see, biology is a system as least as complicated as any computing infrastructure we have.
(Compare also most organisms these days being reliant on oxygen. Or how humans lost the ability to make vitamin C in their bodies.)
One of my first bosses describes the opposite relationship. We tend to think technology is very rationalized. He once told me that we should think about the system we were building as an organism. Build it, but also play with it and get to know it, because it will have its own personality.
Who would set up a virtual machine backup system that offers no alerts when it recycles something? I mean, I might do it myself, so I sort of know, but you shouldn’t do it.
Similarly I’m currently having the “pleasure” of documenting every function I’m responsible for as I’m leaving the public sector for the private sector, and man oh man, would the excel sheet of functions and directions to where they happen have saved myself from hundreds of hours “searching” or “refreshing” my knowledge on things when I had to revisit them after years of not working on them.
It’s also why MBA business process types are valuable in any enterprise sized organisation. Because they tend to both discover and document these hidden functions when they do their work. Of course, 9 out of 10 times nothing comes of their discoveries, because managers are notoriously poor at long term benefits realisation.
It is the same with self-documenting code; if the function and variables names are clear with a sensible architecture in place, then it is easier to keep everything in sync and readable even as different engineers take over; rather than having external documentation which must be sought for, possibly discarded, and therefore not used as much as it ought.
One of the things I really like about Elixir is being able to put executable tests and examples inside the module documentation. If you run the tests and it fails, you know you need to update the examples in the documentation.
I’m not a big fan of executable testing or self-documenting code in smaller projects. Sure you solve a lot with good naming, but it’s usually the business logic that needs to be documented.
I can easily ready how a piece of Python or C# works, but I can’t easily tell why employees who only work less than 7 hours a week and are paid in advance are excluded from the function I’m looking at.
But I mean more generally. We selfhost some things, have others in azure, and have a lot of different databases and projects. It’s nice to have a reference sheet of which belong to each other.
This kind of knowledge is mostly tacit and must be absorbed by osmosis from constantly talking to the person and watching them work.
Tangential to the main point of the article, but I've seen this exact failure before! I worked at a retailer that would reboot all point-of-sale devices daily, after stores closed. The day before biggest sales day of the year, the team decided to leave the devices online overnight - sometimes a device would fail to come back online after the reboot, so leaving them on would prevent that sort of issue.
Of course, one bug or another reared its head after 24 hours of uptime. We ended up rebooting all of the devices anyway.
New versions of our service got deployed once per week, barring delays and hotfixes. Sometimes we'd miss a release, and a version would stay in production for two weeks. Maybe there would be a code freeze and we'd miss a release, and a version would stay in production for three weeks.
We occasionally saw problems. Some cache would fill up over two weeks but wouldn't empty, or something like that. Something that you'd like to test, but which doesn't show up unless you run something in production for a month.
I advocated a simple solution: either check that the process runs for multiple weeks in testing, or restart the process every week in production. In other words, don't test it in production.
When they came back, we had so much mad respect for the things they kept out of our day-to-day job.
That's one reason why so many managers hate remote work.
Before remote, things would sort them out on their own most of the time.
Now they have to think about their job and define processes etc.
When it comes to programming, this is one of the good reasons why simplification matters. Systems that are complicated to the point where you can't understand how the intended functionality "arises" out of the interaction of its parts, is a system that is very difficult to debug and evolve.
https://www.google.com/amp/s/www.thestar.com/amp/news/gta/20...
It doesn’t always work, but this time, it did.
I dunno, man, I think I'd demand to be paid a million a year if someone forced me to attend a meeting at 7AM.
Even with everyone working remotely can we please try not to make this kind of BS a norm in the industry?
I guess it illustrates precisely why it's a bad idea to have meetings very early in the morning.
Our field of work (I suppose most readers here are IT professionals) never had to normalize fluctuating schedules until relatively recently - when our profession has become fashionably decentralized.
And I think we need to be careful and make sure that harmful practices like having too many meetings or mandatory meetings at odd hours have not become a norm.
Programmers don't create things and solve problems at meetings. Most of the time, the meetings are there for other people in the business.