Of all the security post-mortems I’ve ever wanted to read, it’s sad I’ll probably never get to read this one and its tale of how a team of well-paid comfortable engineers got together and decided this patch was a good idea.
Engineer Alice: We should really fix this properly.
Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?
Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.
Manager: And we can just prevent it with an extra regex rule in the front end code?
Engineer Alice: ... Yes. irrepressibly existential sigh
Manager: How much QA effort to call this good?
QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours.
Manager: And for the proper fix?
QA Engineer: Oh, hell, at least a week of stress testing to really be sure.
Manager: Add the regex rule.
The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
curl -A "anything but Curl" x.x.x.x
the problem has not been fixed by a user agent string.
> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.
> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.
> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
[1] The incompetent ones, of which there are many.
I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
Cisco has a bad track record in that regard and software in general.
My employer has reached a settlement with Cisco after buying their "firepower" solution which has been plagued with basic software and usability bugs.
In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)
It’ll be totally explainable as engineer failure if something somehow doesn’t work.
No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.
You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
"I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.
Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!
In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".
Quitting assumes the possibility of finding a job after spending 4 months doing leetcode 6 hours a day and not having people that depend on you.
Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...
I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.
One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.
I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
software in many cases is more art than science.
A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.
It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.
And the answer to that question is almost invariably yes.
That's like stopping 90% of a flood. The other 10% does more than enough.
That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.
This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.
It is bad management to hire incompetent engineers.
It is bad management to fire competent engineers that push back.
It is bad management to make employees feel as though they would be fired for standing up in any way to management.
Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.
We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.
The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.
Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.
The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’.
Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P
1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.
2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.
Test-driven development
It's completely unrelated to any development methodology, this is someone unaware of basic HTTP protocol semantics.
So just bad development, period.
That's assuming that the well-paid and comfortable engineers decided on and implemented the patch.