Trying to sneak in a sketchy .so over the weekend
rachelbythebay.com
rachelbythebay.com
If they succeeded, the manager would have pointed to the feature as an example of their “hustle” and ability to get things done where others couldn’t.
If they shipped the feature and it crashed the website, the manager would blame the front end team for making a fragile system that couldn’t handle a simple feature.
If they failed or were blocked, they’d point out their working proof-of-concept and blame the front end team for onerous process or politics.
The real litmus test is how the company reacts to that manager after this stunt. If the company sides with the hustle manager, it doesn’t bode well for engineering in the long term. When management rewards shows of effort instead of long-term results and fails to acknowledge negative externalities or selfish managers, you breed more of that behavior.
However, if management sides with engineering and shuts the hustle manager down, you’ve found a good workplace.
If a startup needed to move quickly, they’d ping the relevant parties at the planning phase and get everyone on the same page.
I was referring to the archetypal “hustle” managers who deliberately try to do an end-run around other teams for their own personal gain.
Startup or enterprise, doesn’t matter. You can’t have management that rewards asymmetrical games that benefit rule-breakers at the expense of everyone else, including the customers.
I had a laptop which I brought with me from Australia. It wasn't in the asset register in the USA, so I was entitled to a computer. I ordered the most highly specced desktop build available, put it into the empty cubicle next to mine, and spun it up as a development server. It didn't have backups, but that was OK because I never worked with primary data on it and all my work got committed back to Perforce daily.
Strictly it was very much against policy, but policy would have meant I spent 6 months sitting on my hands. My manager "hustled" for me and did an end run around process.
That service becomes the source for a management report.
That management report contains useful data that the CEO looks at weekly and uses to build his board report.
The original Aussie guy leaves, but leaves the laptop behind because it's not his. He also doesn't document it (because that would get him in trouble for running a server on a laptop).
...Years pass...
The laptop finally dies. The CEO is furious because he can't create his report. He leans on the IT manager. The IT manager has no freaking idea where this report is coming from or who makes it. They lean on the Support team to find out which server produces this report. The Support team drop everything to work out wtf is going on, because this suddenly became their #1 priority.
Eventually, someone finds the decaying husk of the laptop, and works out what's going on. They put together a plan for creating a supported server to do the same thing. It'll take weeks, because they have to provision a server properly through the usual channels. CEO has a rant at the entire IT department for not supporting critical business processes, and not being agile enough to support the business. IT manager takes a beating in the next management reviews. No-one is happy.
[0]usually a rogue spreadsheet rather than server. The worst case I saw was an Excel spreadsheet in a business-critical department running on a user machine with a post-it note on it saying "don't turn this machine off". If the logged-in user name wasn't the same as the temp who had originally built it, the spreadsheet refused to work and the department ground to a halt.
> ... benefit rule-breakers at the expense of everyone else, including the customers
Maybe I've been in startup land for too long, but seems super normal and fine to ship a feature over the weekend if it goes through the regular CI gates - it's tested, peer reviewed, been QAd in staging, etc. Is this not accepted outside of startups?
[0] https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp...
1) Is stuff really badly broken?
2) Was the really bad breakage introduced recently (this could either be an earlier bad rollout, or it could be external factors changing)?
3) Is this requested deploy either a revert of a recently-made bad change or the minimal possible fix/bandaid?
If all three of these are true, then you can do a deploy RIGHT NOW regardless of the calendar.
(recently - because if this is something which has been broken for years then it's unlikely that it suddenly became urgent absent external change - which is already carved out above)
Environments like those described often have continuous push and automated slow rollouts with health checks, so the idea of doing something on a Sunday isn’t that strange at all.
That said, there’s something to be said for not trying to locally optimize. If you push bad stuff on Sunday, you’re messing up a bunch of people’s well-earned rest and recovery time from work. You push bad stuff on Monday, and everyone’s there to help you fix it without the stress of lost family or other commitment time.
The difference is 24 hours, which likely isn’t going to make or break anything. It’s easy to get sucked into believing things like that matter when they don’t.
In this case, someone was trying to quietly ship things to production on a Sunday without involving the owners of the front end. How would it look for you if some other team crashed your part of the website on a Sunday without even coordinating the change with you first?
My point was that it’s important for companies to not reward selfish behavior from managers who want to make a name for themselves. If you genuinely need to ship a website feature on Sunday, you involve the website team for launch and follow up monitoring. You don’t try to quietly ship it out the door at the risk of breaking other parts of the business, as Rachel explained in the article.
Her story was of an untested feature trying to get injected into the frontend on Sunday to meet a Monday deadline, by someone without the proper access, and with no apparent oversight or process concerns.
I'd do everything I could to block this push as well.
Sounds like these people designed a component that wouldn't work on the current infrastructure. That happens, but if there is a big oversight like this, then you should ship late, instead of risk taking everything down.
I haven't had specific conversations with anybody about it, but I think we have all been around the block enough times to have been burned on a few weekends when it really wasn't necessary.
Not a start up at all though, and not a team of twenty somethings with anything to prove by moving fast and breaking things.
Agree with you though that I have seen this at a lot of places. I did a number of phone interviews looking for a more relaxed place in order to end up here.
In fact most trading platforms have this huge advantage of not being 24/7 operations.
Multiple of these processes you listed often require humans. You either are asking them to do this during the weekend, which is bad, or you gave them ample time during the week meaning you're ok with steps needed on weekdays, so you can do that for the last step too, with full staff present.
(bugs and urgencies notwithstanding, but that doesn't appear to be what we're discussing)
Deployment over the weekend can make a lot of sense in the world of B2B, but there’s a difference between a carefully thought out plan to deploy at a quiet time and sneaking something out when no one is looking.
Once a startup hits some level of maturity, it's unacceptable to be shipping something significant on the weekend (or whenever people aren't around to respond to an issue). Probably post product-market fit, maybe Series B.
I guess it also matters how much your company values work-life balance.
Ideally, once they identify by trying to pull the trigger you can move them out of the company.
Many of these processes may be necessary, but it's also necessary to explain why, and to make them as fast and painless/frictionless as possible - especially as each single process in isolation may seem reasonable, but when stacked on top of each other, the "get stuff done" approach becomes a lot more tempting.
The less clarity there is on the "why" the more creative the management will be.
Of course, managers who say "I don't believe that will happen so I'm going to skip this part." should be walked out of the door to their car immediately. :-)
Process stacking for me is one of the reasons why it's super painful to work in big companies if you want to get something done. As soon as somebody makes a mistake, they will add a little bit of process to ensure that never happens again.
Individually, as you point out, that makes sense. But if you have to go through a 1000 item review checklist for a single line of code, then I can assure you that no human will be able to actually think through those 1000 items. But they will go through the motions to satisfy the process. Then because they have the checklist they don't think they have to think about it anymore. They make a mistake. It gets added to the checklist.
I experienced situations where a single code change would take at least a month. This lead to people trying to save time on a) tests, b) any kind of refactoring, c) adapting libraries instead of writing your own implementation (because fixing the library would be 2 code changes and not just twice the process effort, but an actual committee had to decide about the library change first.)
So a lot of process IMHO is the worst thing you can do for your code quality. Checklists are good, but they should be limited to a manageable number (e.g. 10 items, if you want to add something, you have to remove something less important first). It should also not be harder to do the right thing, e.g. centralizing functionality in libraries should be easier than.
To be fair, the other side of the medal is when it is simply not possible to get certain things done, because the need has not been anticipated when designing the processes. If you don't have the company political clout to get these processes amended, your only option is to wait until a customer is negatively affected, in order to drive the point home. Still, hustling (even if it is well-meaning) is of course not an acceptable solution.
Our sub would dock at the pier the day before, everyone but Weapons Department got the day off/in port duty day. Weapons Department would hold an all afternoon walkthrough of the entire process. Manpower locations and roles. Equipment setup and basic operations. Types, quantity, and sequence of weapons to take aboard. Expected timeframe / pace so that no one was expecting to have to hustle to catch up.
And everything was in binders, with plastic strip edged pages and fresh grease pencils issued to everyone managing.
Every one of those steps was a result of "Ok, crap, what do we rewrite to make sure (shudder) THAT NEVER happens again."
And even so, on my fifth loadout party, I still missed a retaining strap and almost helped dump a torpedo in the harbor, except there was already a step right after mine with a separate checkbox that said "Aux handler has checked strap type/quantity/positioning for weapon type."
Procedures are great for the things that need them. And when you have numerous teams/functions scattered about, procedures are even more necessary.
And I do get that a lot of code is not likely to detonate under the companies' hull, per se.
The issue can sometimes occur when the manager doesn't know that their rag tag team is not this special case, but actually clueless. Or have not learned to spot the difference, or that there is a difference.
> just a counter example
It took me a while to understand why HN'ers revel in "the counter-example."
In mathematical proofs, you only need one counter-example to refute a proof or argument.
Pedantic HN'ers seemingly fail to realize that mathematics and the real world are not the same thing.
Here you're right to raise issue, but it seems the comment is merely trying to point out that 'not all [Scottish!?!] rag-tag teams are bad' and idea draw attention to some such teams being superb. Which seems a fair comment to me.
The things we're talking about here aren't mathematical axioms, they're general trends. One counter-example does not disprove a trend. Every real-life trend has exceptions, and it frequently is interesting to examine the exceptions to see why they bucked the trend.
These analysts say that Soviet authorities appear to recognize that operator errors at the Chernobyl plant on the night of April 25-26 were not the sole cause of the accident, and that technical flaws in the reactor’s design contributed to the worst accident in the 44-year history of nuclear energy.
In particular, they said, a distinctive feature of the Chernobyl design, which sets it apart from conventional nuclear power plants in most of the world, is its tendency to generate a sudden and uncontrollable burst of power if large steam bubbles, or “voids,” are allowed to form in the reactor core, as they did before the accident.
This peculiarity of the Chernobyl type of graphite reactor, called a positive void effect, is now seen as a decisive factor in the accident, one that transformed successive blunders on the part of Soviet operators over a period of hours into a catastrophe.
https://www.latimes.com/archives/la-xpm-1986-08-23-mn-15781-...
But anyway, the main lesson I learned there is that as an ops team (or broader, as an IT department) you need to have Principles, capital P. A short set of rules and goals which you can always point to. Like the uptime goal, which excludes / includes a LOT of things right off the bat - access controls (nobody can touch production directly), testing practices, application architecture (stateless), etc.
It makes me wonder how these organisations don’t collapse under the weight of their ineptitude. Most of the bugs or issues I have to fix are from problems we created by short term hacks. Way beyond simple tech debt.
The engineers are as much at fault as the managers, particularly when it comes to introducing insane complexity to the stack to solve simple problems (how many startups seriously need to invest in tech like gRPC or graphql except to gain cool points?). Management, on the other hand, have no empathy for the people doing the work, so quality dips as we are pressured by both self-imposed and external deadlines which are decided with zero input from engineers.
Half the time we web engineers are building glorified content management systems with some nice design over the top. It’s boring but it’s not a burn out.
I remember as a young Eng. getting caught up in the platform holy wars, and then sitting as a PM looking back on it all like I must have been in some cult.
There's truth to the notion that 'it's complicated' and rarely does anything get done in a weekend, but if there is focus, a decent dynamic process, things can move faster.
I worked at one company that had a messaging product, it has a big team of Engineers and things were at a snail's pace. I suggested bringing in a few talented people and starting from scratch as a re-factor, they thought I was crazy. A young intern left the team, did it on his own with one other person and met with enormous success. The company, even after literally watching an intern out-do them never changed.
Both the old company and the new company are big enough names you've all heard of them I wish I was at liberty to share.
In another project, we were opening up some basic APIs. We did some work with Facebook and they were able to give us a custom API in literally a few days. Our own, simple APIs took 18 months to deliver. The weekly product teams consisted of 10 people rambling on - and the two most important people, the dudes actually writing the code, were not ever present. It was a colossal and shameful waste.
Even though getting a rag-tag bunch of Engineers over the weekend is usually not a good sign (it might actually work for some marathon bug fixes or something), I'm usually sympathetic to the cause.
Who knows which it is. The article says they haven't touched the frontend servers yet, not that they haven't done anything substantial yet, so I don't think we can know unless Rachel decides to comment on that point.
Ship this feature quickly and collect all the credit, or crash the website in the process and blame the website team.
Yes, he was an incredibly capable software developer, but that had led to an ego issue where he thought he knew EVERYTHING, not just developing in his particular stack... so he'd find his way into various systems (software and hardware) he had no experience in, and assume he could just figure it out.
We had half a dozen full-on-disaster level situations in the couple years I was there thanks to this 1 person. Each time it was shrugged off as "an accident" because heaven forbid you actually upset him or hold him accountable.
Some developers put their crazy ideas and experiments on github; some developers put those in production.
However, there are also a few "architects" who are basically free agents with god-like status. They don't architect anything. Rather, they do a lot of counter-productive things such as attending meetings for specific teams without those teams knowing. Then they make large decisions about the direction of a given feature without documenting anything, let alone letting those teams in on what's going on. It's always a fun surprise and works wonders for our ability to deliver a solution on time /s
Beyond that, they'll go off and develop whatever they feel like and force it through the system. We have a code review process, but if you bring up that problems are present in their code, they will often say that it doesn't matter and that the code just has to go through to meet some unspecified deadlines. They'll get management to skip code reviews if you try to hold them accountable. And of course, this results in things exploding. But then they can clean up those exploded bits, knowing what they were, and come out heroically since they "put out so many fires".
I'll stop ranting. But it really is frustrating, especially when you get shut down by management for bringing up any issues with this chaotic group.
In my last job there was this guy who had more than 10 years of dev experience and:
- He tried to convince me it was better to write our own encryption algo instead of using https.
- His mySQL tables used no foreign keys, only int fields.
- He mixed all sorts of naming conventions in his code and databases: English, Spanish, camel case, snake case, kebab, etc.
- He spent 1-3 hours every day getting into the bank web apps, downloading some pdfs, copying and pasting stuff around, and finally producing a report by hand. Every single fucking day. So I developed a small service for extracting financial data from those pdfs into JSON. He only needed to read those JSONs and automate the reporting part. No, he kept making those reports by hand.
The guy was a friend and long time collaborator of the IT manager which was a complete ignorant in terms of dev.
This one is potentially justifiable if speed is a higher priority than data integrity.
Besides performance issues, FKs cause ordering issues with restoring tables when necessary.
However, they're great for diagramming tools to show table relationships.
I'm more concerned when I don't see unique indexes on things that need to be unique (often caused by early versions of RoR.) In my experience, this is never a theoretical problem - it always becomes a major operational problem.
Source: MySQL DBA
also, their ability to shift blame often is the reason they are a senior developer and not a fired developer.
Replying to myself, since I can no longer edit.
However, bigger organisations can get really out of hand with this. They start out with small carve outs, oh, nobody really works from the start of Christmas week until the first full week of the New Year. Soon it's from Black Friday although good luck scheduling anything the week before that, and so before you know it they're declaring that nothing can even be planned for the months of November, December and January each year.
Now, if you really must have this habit, which I don't recommend, what you need to understand, just as much as the "No shipping on Friday" folks is what this does to your perception of time. If you have "Don't ship Friday" then the question in the Tuesday meeting "Will it be live on Monday?" must be re-phrased "Is it ready to ship by Thursday?" and not every manager seems to understand that. But when you have "Nothing can be scheduled for our 90 day Winter Risk period" as policy then the question in that meeting "Will we be ready for the January 31st legal deadline?" is actually "Are we 100% sure we'll have fixed this by the end of October this year?" and I have yet to see anyone who recognises that despite having a policy which makes it so.
Every software release includes risk. There's risk that there are bugs. There's risk that it will introduce regressions. There's risk it will cause a major crash. We can do things that reduce the risk. We can have recovery plans for negative outcomes. We cannot eliminate all risk.
The reason businesses have black-out deploy dates is so that they can control the risk. Deploying code 24 hours before a critical business day, like black friday is for some businesses, is an entirely different risk/reward ratio and should be treated different. Even 90 day risk periods can be valid for some businesses, for example seasonal businesses during peak season.
That's a completely valid strategy, but as I explained you need to actually embrace the consequences of this strategy in your thinking. If the next 90 days are a "risk period" then when somebody says "Next week" they actually mean "Three months from now" not "Next week" and everybody in the business needs to start thinking that way. Sometimes that's going to cost a bunch of money.
For example, if you have such a 90 day period covering November, December, January as in my example, then a system which alerts you to certificates on public facing web sites that expire in less than 30 days time is no good, you can't act for 90 whole days, so that system needs to be re-calibrated for more like 100 days.
Once you accept the 90 day risk period as a reality rather than a convenient fiction it gets very expensive. Make sure the people who insisted on this risk period are paying, because that's the only way they'll learn anything from it. If you don't make them pay for the actual consequences of a 90 day risk period (e.g. hiring a lot more staff, buying more expensive products that don't need intervention) then of course they're going to keep increasing this magic "It's not my fault" period and you'll pay the price in your sanity.
It's really a case of knowing your business and customers.
The magic is in telling the difference.
This was a Sunday!
And it didn’t seem to have been tested, so where was the Dev/UAT/Prod separation?
So many red flags, and I think anyone who has been in development for a while has had some product owner try to break procedures to force whatever pet project that is falling behind to hit some target to get their bonus/gain brownie points. It almost never goes well. But by this time the PO has a scapegoat because it’s no longer their fault it’s not working, it’s production’s fault. (Also it’s not always a PO, it can be a developer or anyone up to c-suite level)
Sigh
That would be a very bad, very cliche joke if it weren't a true war story (2005, good times). Turns out the 15 y.o. had convinced his father that I was "lame" at my job because I didn't tweak regedit. Important lessons were learned by the three of us that day!
How do I install half of an RPM?
Well, I'm stumped... I wonder if he just needed one tool out of suite?
While I agree with Rachel here at a high level, and have been a dedicated follower of her blog, I completely agreed with that comment. You shouldn't be shipping things in the manner described in this post, and you shouldn't be considering your coworker "some rando" and looking down on them for not having the same schedule as you.
Even aside from whether "rando" is contemptuous, the issue isn't that they don't have the same schedule: it's that they are not respecting the company-wide schedule, nor are they respecting fairly obvious norms of professional software development.
I'm a teacher, and I very much believe in educating people rather than putting them down, but jumping on a single phrase/term when there's nothing else to suggest contempt here strikes me as odd. It's especially odd when the entire culture of sysadminship has a reputation of eye-rolling and begrudging wizardry to protect users from themselves.
As far as the company wide/normal schedule goes, at my previous place of employment, major changes were routinely performed (by me) off hours on a Sunday with only relevant personnel on hand. This was primarily for B2B reasons where the vast majority of our clients were doing mission critical things from Monday to Friday. I don't feel that this was the case in these circumstances, which is why I completely agree with what was really said here, and with your reply was well.
I suppose my personal experience in this industry leads me to believe that small snipes like that uncover much deeper contempt than is revealed on the surface.
However there’s often a substitution for a more acceptable phrase when you relay similar information to management.
It’s perfectly acceptable in this context of a BOFH type story, which doesn’t care about the identity of the luser in question.
It's very very off putting and detracts from an otherwise good post. Makes the author sound like someone who simply has nothing but ire for their fellow employees.
Is there some other prestige you should know about her apart from her blog?
It was also interesting to read about a wide deployment of circuit breaking. That's a relatively new concept to me (never ran into it at Google, for whatever reason), but it's another way to prevent weird outages. Last week I upgraded something in my k8s cluster and the API server stopped responding. With the API server unresponsive, I couldn't kill the workload (or even determine if that's what was causing it). I had to kill all the nodes before I could get back in. If the job were communicating to the API server through a circuit breaker, it would have popped after requests started taking too long, and I would have been able to just kill the job instead of having downtime. But, I guess it's still a relatively esoteric concept.
I know this article is about not doing dumb things in production. I just assume those are a given. I'm more interested in the nuts and bolts of day-to-day operation ;)
I believe this is commonly called the "circuit breaker pattern"
Glad I was wrong!
Sounds even better (and more malicious) reversed: "Sufficiently advanced malware is indistinguishable from incompetence". All those debug features "mistakenly" shipped in hardware...
That’s malware in my book.
The frontend systems, taken as a whole, amounted to a bitchin’ botnet. They’d destroy ordinary services without even working up a sweat.
Using the right mechanisms for comms between them was in everyone’s best interest.
A codepath which is perfectly fine in testing/staging could kneel over immediately against a production workload,
- Someone (I heard it attributed to Napoleon, but have seen it attributed to others)
In this case, it's very true -- workloads are really difficult to properly test with respect to production load. A code path that works well in one test can easily buckle under the pressure of a production workload. Stress testing is hard.
I’ve found a lot of things solved by a decent review and deploy process.
My precious company just went through SOC2 certification. SOC2 is mostly about change management. Who can make changes to prod? Is every change approved by the right personnel? Is every change auditable? Etc
Mostly boils down to using git(hub) properly and having a sane deploy/monitoring/revert system.
Doesn’t matter if you’re the CEO or a VP or a yolo manager. You can’t deploy things without oversight. If you want to deploy on the weekend you need approval from the opslead for that system who is responsible for keeping things alive.
On the weekend, no one should be working, even opsleads should only work if they are paged. Only broken things that really affect customers are fixed on weekends. No new features. Period.
I have read the post, btw (liked it - I usually like Rachel's posts) - but still don't understand what .so is.
In this context, I believe .so is the file extension for extensions that can be loaded by web servers during runtime. I think the person in the story was trying to change the Apache2 configuration of all FE servers to load the wanted client library / .so file.
For example libc is a .so file. In windows, think .DLL.
In the ones where she specifically blames other people or companies, I mostly see a competent, reasonable person who is fighting an uphill battle against unreasonable management, which is a trope among software developers for a reason. Like this one for example -- do you really think adding (effectively) backdoors for your special service on a Sunday evening constitutes sustainable software development? I'd rather have that "toxic attitude" than some meaningless sunshine-out-the-bottom screed about "team players who get things done".
I don't but then again I spent years on call for a couple of critical services. SREs / sysadmins are the people who have to work overtime on evenings or weekends to restore services when an incident causes downtime, whether accidental or caused by foul-ups by the development team. After a few years of that, having a rather low opinion of colleagues whose routine bad practices or behavior risks downtime for critical services is more than forgivable in my book.
It's an easy mindset to get into and a blog is certainly a valid place to blow off some steam IMHO.. But when you have the knowledge AND access to fix all of your own mistakes, it's easy to forget that everyone makes mistakes and to empathize with the every-engineer. It can be a toxic mindset to let seep into the workplace; better saved for the pub :)
Even if your colleagues are super-smart they will still fuck up some of the time. In a large enough project someone's fucking up all the time and these people have to fix it.
Of course they're grumpy and tend to complain. That's how they stay sane and don't go postal at work :)
So next time you see an SRE or the like grab them for a coffee and tell them how much you appreciate them. It could save your life in the future.
Sorry for sidetracking, but are they not good working environments (note this is a separate question from whether they're a good thing)?
I've been planning to try to get a job at one of the FAANG companies, and have basically assumed it would be a good working environment based on the reputation, the perks offered, and my prior experience at another very large tech (but not exactly FAANG) company, which was that the working environment was generally very good there though I didn't really like how that particular company operated in the marketplace.
Depends heavily on which team you join within a company. FAANGs are not monoliths; they are collections of business units cooperating (or sometimes not) with quite a bit of variation in culture and work-life balance.
At least, I'm guilty of that, I skim articles and read comments. But this post is not skim-able(?) and thus feels a lot more dense than usual.
However, I went back and read the whole thing and it's certainly worth the read.
Also the time of day/day of week.
But mostly the domain name.
I always click on the article when I see it's from you.
Those intelligent and unbiased enough to learn something useful will do so. I agree with and/or have learned from almost everything you've posted. Some of those things mirror my own experiences.
I also didn't see any problem with tone, you were polite and helpful. It seems quite obvious the only problem with your "tone" is your gender. Not that you didn't know this already.
Thanks for your contributions. At least some of us appreciate them and find them valuable.
HN is always very critical, you'll see it on every article.
HN (and more broadly the tech industry at large) likes to believe that it is an egalitarian meritocracy but it is unfortunately not true.
Besides aside from the name it's not like she puts her sexuality front and center, the reader has to notice it as part of the domain. Her name isn't even in the article.
What I'm suggesting is that opinionated women tend to draw more ire and negative reactions than opinionated men, for reasons too complex to address in an HN comment.
I'm willing to bet that if the same post had been made on a medium.com blog that didn't include a gendered name ("Rachel") then the comments would be less caustic.
(Regarding everything the author does/says/thinks in the article)
Everything she does in the article is exactly what any sane sysadmin would do, it's institutional awareness.
My point exactly.
Misogyny is pretty common in tech. It's not an absurd conclusion to jump to.
Given the up-votes of the thread and the down-votes of the negative comments (even ones only tangentially negative) that seems an odd conclusion to draw about the whole HN community.
The right thing to do would be to help this person achieve their goals while maintaining reliability of the operation. Calling a co-worker who is just trying to get something done a rando is the epitome of toxicity. She wouldn't last 5 minutes with that attitude in my organization.
See another post by this author for a similar tone: https://news.ycombinator.com/item?id=22277582. Maybe work right now is tough. It sounds that way.
All that said, a solid takeaway is that sometimes people are naive and you need to say no. The next step is: alongside saying no, ask what their aim is. You're saying no to the solution and you don't know the problem.
And then it sounds like the person did it anyway, at a later date, and broke shit.
If that person repeated that behavior it is someone I wouldn't want on the team, or at least having access to certain systems.
And I’ve yet to find any set of procedures that cover all possible cases. So you still need people to apply good judgement when the procedures don’t cover situations, or if the procedures currently would allow something that shouldn’t be allowed. Like: “no, even though we have a way to safely roll new libraries out, we won’t allow you to add this library to all front-ends, you should follow best practice and use ...”
Would Facebook really let somebody do this to every external frontend server on a Sunday without some boring authorization first? The authorization could simply be denied with a boilerplate response about the right way to do things, with less emotional resources spent than the author apparently did.
Unfair to call the coworker a rando, but these people did bring systems down at literally the first occasion they managed to ship their code, despite having received advice.
I have a similar situation maybe once or twice a year (used to be worse when working in other places) and do my best to defuse the situation while maintaining politeness and etiquette, but this is the tone I'd then use to describe the story to some friends.
It's a blog post, I took their use of "rando" as an artistic choice in emphasizing this person had utterly neglected to coordinate with any of the stakeholders beforehand, with the intent of going into production the next day.
From the perspective of the stakeholders, the person was effectively a stranger, and that's the crux of the matter being described.
I didn't find this readers interpretation of the author's behavior to be boring. I can see how she may come across as being overly combative (though I agree with her goal).
Your frequent comments about what you find to be 'boring' is boring.
The root comment has been hidden, and the mod (who probably hid it) has replied to it stating why it was hidden.
That we can see moderation is a good thing; that moderation is 'boring' is to be expected.
Another way of putting it is that indignation tends to melt down to a few formulations that get repeated a lot, where intellectual curiosity looks for what is new and different in a situation, and then has a new and different reaction to that.
Since we want HN to be a site for intellectual curiosity, it's important not to feed indignation too much. It quickly overwhelms curiosity if allowed to (especially on the internet); therefore we need to moderate it. You're right that the moderation comments are boring; that's because they're also predictable. If it helps at all, they're even more tedious to write than they are to read.
Let alone on a Sunday afternoon. The whole thing smells like someone knew there was a process to do X, figured their X wouldn't pass, and tried to bypass the process.
And she's always got an overall point about how to run things better / improve process / get newbies up to speed and past the 'every commit breaks things'.
I'd take the time to go read the rest of her posts I haven't seen, and definitely learn a bit about surviving sysadmin work in larger organizations in the process.
Granted that his tone and her tone were very different, but, imo, as a reader of her posts, this wasn't necessary. Educating junior devs is more productive than considering them randos.
I'll take my downvotes here, but personally I think Linus with that apology crossed the threshold into old age. Quoting Tomasi di Lampedusa:
> The Prince who had found the town unchanged, was found much changed himself, whom never before would have used such accommodating words; and since then, invisible, began the decline of his prestige.
I don't mind that way of communicating, but I'm also absolutely fine with Linus' style. The comments in defense of this had a bit of a cultish feel ("she has earned the right to talk that way about people!!111"), but I gather that's more because people know her and may consider her a friend, so their perception is changed.
The usual result is that if someone finds out they have to do something that involves asking the grizzly bear a question, they make sure to do lots of due diligence first, and maybe ask other people who might know the answer, and only bother the grizzly bear if they absolutely have to.
You definitely don't want to have an organization with lots of grizzly bears, but it is also the case that a particularly high-stakes system sometimes does need a grizzly bear paying attention to it, to keep that system from giving everyone a great "learning opportunity" on a regular basis. If that's a less high-stakes system, then that learning opportunity is fine, but if it's something that nearly everyone relies on in a large organization, you cannot afford that many outages.
Not saying you're wrong in your evaluation, just saying that the same personality type is occasionally useful, or even necessary, in certain situations.
Toxic is playing dumb and bypassing controls on a Sunday to get your boss off your back, while dumping a bucket of shit over everyone who is obliged to support the company.
It's actually not, that's just going along to get along. Ship enough features, get a promotion. The higher-ups that enable the circus are the toxic ones.
I think in most places, if your goal is to make major production changes on a weekend, the right answer is ‘no’. Seems like she handled it fairly diplomatically, all things considered. And as it turns out, later on the rando in question broke the internal site. So she was probably right to tell them to step away from the keyboard slowly when they were trying to do it on production.
I feel it’s totally justified to just make sure that this Sunday they don’t break production. Helping the team get it right can wait until Monday.
Rachel doesn’t say what exactly was being communicated. But maybe her response on the weekend made that clear. In fact that’s what I assumed when I read the article.
Why not teach people the proper way to release things to production and avoid the name calling and negativity?