I accidentally saved my company half a million dollars | Hacker News Reader

I accidentally saved my company half a million dollars

ludic.mataroa.blog

1,076 pointssoftskunk458 comments

I feel this entire post.

My career record (US Navy) for cost savings was something over $50MM. Every time I did something I had to do PowerPoints, present to Flag Officers, etc. Hell, once I almost got hugely punished because I didn't let my boss take the credit (he had no desire to because he had zero idea what it was I even did).

Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects (I hate every single bit of terminology in that entire godforsaken sentence), where it was literally my JOB to save the DOD as much money as possible with the least disruption possible. Those were the absolute worst years of my career. That period of my career was my reward for just going rogue and fixing things that saved millions.

I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.

Sometimes I think of joining government org (3 letter agency) to find areas to improve $ efficiency on, or make decent direct contribution to US as a thank you for accepting me as an immigrant.

I'm an engineer in a FAANG, who worked directly in money flow, and have had experience in diverse areas where it could come handy.

Then I start thinking finding the right person to work with & right area to start at is probably 95% of the job, then give up.

As someone who does this sort of work (but posting personally, not officially), you're 100% right. There are groups of already-networked folks you can join like USDS or 18F that are already connected to the problems. The post reminds me of some of the stuff you might see in gov, though I've never seen someone save money and be punished like this.

What is 18F? That's not the easiest thing to search if you're not looking for the obvious.

https://18f.gsa.gov/

Typical gov. Have a money burning problem? Let's create yet another org to solve it.

No government has a money burning problem. They have expensive service delivery problems.

I mean, if you're upset about government spending on tech, you better sit down before someone tells you what goes on in the large corporate sector...

https://18f.gsa.gov/

Searching "USDS or 18F" helped google understand what I was looking for

> though I've never seen someone save money and be punished like this

It happens. From DoD civilian world, 6 months in with a new org (I was the fng)…

tl;dr — Made some software that moved a project from imminent mission failure to mission success, and got all of my command chain in trouble in the process.

1. There was an 8-figure, 2-year R&D contract for some mission critical software that had functions that my org needed. We were updated regularly that everything was moving along swimmingly. At the end of the contract, only about 10% of the functions actually worked. This is totally ok from their side since R&D contracts are “best effort”, so no functioning deliverables are actually required. The part of the software that was mission critical for my org had 0% working. My org was fucked.

2. We need this software in 3 months or less or we can’t fulfill obligations that we had made years prior.

3. Manager gives me free rein to make something that can solve the problem.

4. I have a prototype in one week. It’s not as automated as the 8-figure solution was supposed to be, but we had bodies to help with that (and also do sanity checks at various stages).

5. Weeks two and three were refining based on feedback from stakeholders.

6. Week 4 was training about 60 people to use the software.

7. Software was iteratively refined over 6 months or so based on user feedback during active use, but no major changes were needed — just minor QoL stuff.

8. Software had the bonus side effect of getting two embattled groups to start talking to each other and resolving process issues, which saved unbelievable amounts of time and lowered stress levels immensely.

9. End result was that we were able to meet our obligations. Mission success! Software worked with amazing consistency (very little down time, almost all of it out of our control), and resiliency (changes were easy and fast to make without breaking the system).

So… what did I get for this? A spot bonus? A coin? A handshake from the commandant?

Nope… I got a reserved “thank you” from my supervisor.

The aftermath tells why.

When my supervisor changed the status update up her chain of command that “mission critical software missing, mission failure imminent”, to “mission software exists, outlook favorable” in the course of one month, people started asking questions?

“You had someone in your department who could do this already?” (No, I had just started, and not for a programmer job.)

“You asked to spend a big chunk of the 8-figure R&D contract on your mission critical needs, and that work could be done in a month!?! And you didn’t know this?!?!?” (Supervisor was not a programmer, so it was all basically magic.)

So supervisor was sort of criticized for “wasting time and resources” and “not knowing her people”, even though she could not have reasonably known that I existed (the R&D contract started two years before I worked at the org) and could solve the problem.

That’s bad, right?

Well, it got worse.

The head civilian of our org had been making updates to DC about a potential mission failure due to the 8-figure contract software not working. When that got updated to “not a problem” after it was evident that my system worked, he got called to task in Congress, with congressmen asking him the same sorts of questions that he asked my supervisor (“You had in house talent that could do this? In a month? And we spent how much time and money on a contract to get the same functionality?”). There were no good answers.

Needless to say, once I heard about the fallout, I was looking for the next thing. I felt like I had done my civic duty as a public servant. It was clear to me that further involvement would probably have similar outcomes (efficiencies made at the cost of negative career risk for my command chain).

You'd want to start with the United States Digital Service (USDS). They're basically a consultancy inside the executive branch to build tech solutions for the various 3 letter agencies. You basically do a "tour of duty" for a year or so (however long you want), tackling one of these solutions. And they go out of their way to hire folks like you, from FAANG and other tech companies, so you work with good people.

https://www.usds.gov/

Is the salary competitive?

to my knowledge, USDS is still constrained by the General Schedule[1] salary table.

(no.)

edit: incidentally, this other thread [2] on the front page today has lots of comments confirming. ctrl-f for USDS.

[1] https://www.opm.gov/policy-data-oversight/pay-leave/salaries... (PDF)

[2] https://news.ycombinator.com/item?id=38067206

Anyone with a current significant salary in tech who lives in a major metro will make $183.5K.

The salaries adjust by location, pick your location from https://www.opm.gov/policy-data-oversight/pay-leave/salaries...

This is the SF area table: https://www.opm.gov/policy-data-oversight/pay-leave/salaries...

The links seem to be a bit out of date. Says for year 2020 and has a % increase for 2023? $170,800 is the top number in the chart for 15-10.

https://www.opm.gov/policy-data-oversight/pay-leave/salaries...

You guys live on another planet. As a software developer slave I make 20,000 after diacounts

Most people do this as a sabbatical. I don't expect it to match my current salary, or be even half of it, frankly.

I interviewed with them. The recruiter, I could barely understand. I got the 3rd interview after 5 months. They told me to click on a link where they would ask me to code live. They could see what I was doing. I wrote a simple Python algorithm to a problem they gave me to solve. Complete irrelevant to what I had applied for. I thought it was a joke at first, then I visited their website. I cannot recommend anyone this service. So unorganized. I too wanted to contribute in the tech areas since we all know how much they lack in that department.

I think I would enjoy working for a 3 letter agency but the USDS website makes it look like you'll be working on very different types of things. Although I suppose they wouldn't be advertising that work.

There are a lot of ways to make an impact in National Defense outside of direct government employment. It’s tricky to find a good contractor, but they exist.

The direct employment option might get better with the recent executive order on AI. USG pays market wages for VA doctor/s; eventually they’ll do the same for tech roles.

> Six-Sigma Black Belt

I'm so triggered by this phrase. The worst (to work for and performing) company I was at hired SSBBs like crazy and none of them really did anything or made a damn bit of difference, but if you took the courses you were fast-tracked for success as far as promotions went.

I was not at all popular among my "peers" in that world. I had too much common sense, and didn't make changes just to make changes. I was extremely effective, but since I didn't play by the rules I was ostracized internally.

That said, it did give me the opportunity to get the hell out of that community and back into IT/CyberSec.

In all honesty, there are absolutely good concepts and tools in that world. The issue is that the people who gravitate to that world have no ability to provide any value themselves so don't actually understand how to USE those tools properly.

Everything is a nail when all you have is an ENTIRE TOOLBOX.

It’s interesting because I found Mech design engineers with green belts to usually be really sharp. I only met like one black belt design engineer and he pivoted careers a decade ago. But most black belts I met in QA were managers. Most of the technicians that did the work weren’t certified but lived and breathed the products for decades.

In that industry it makes sense. We were a services platform -- basically a massive, distributed call center.

My experience was that the tools are absolutely useful. Green Belts are those who learned the tools, but stayed at their day job. Black Belts are when you pivot into doing it full time.

Much like people who underperform in technical fields often pivot to Management, you see the same in the pivot to Process Improvement. That creates the issue of the lowest performers at the actual job being the ones most inclined to pivot into that position.

There are always good ones (I'm biased, but think I was). The issue is that they are a relative rarity due to those (and I'm sure other) factors.

The pivot part rings so true, especially “Engineering Services”, aka process and infrastructure control. They often lead the engineering change committee too. I shudder just recalling those meetings.

> Much like people who underperform in technical fields often pivot to Management

This can't be serious...Not sure if jealousy or just regular hubris.

Neither, it’s been my current lived experience.

That said, it’s anecdotal at best; I don’t pretend to have a statistical sample. I also noted that there are exceptions.

I spent a significant amount of time in management and Process Improvement, so I’m speaking from experience of the inside of those within my section of the DOD.

I thought 30 Rock made up the term. Had no idea it was a real thing.

> I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.

Let’s rephrase this in a less nihilistic way: Understand your organization’s values and culture. And beware of doing good (or bad) things in a way that goes against your organization’s values and culture.

Speaking as someone who spent over 20 years in the DOD:

There isn't a way to make things better in the DOD that doesn't go against the culture. Period. My literal JOB was to make things better and it was the worst time of my career. That was with Flag Officer backing and independent authority. I got a chest full of shiny candy and a pathological distaste for it all after a few years of that.

Bigger organizations tend to be more pathological; they attract and shelter sociopaths. So try to find a smaller one that is compatible with your values.

Instead of finding a new org I retired and am doing my own thing.

Figuring out what said own thing is is proving a bigger challenge than anticipated, but I'm enjoying it.

How did you go about figuring it out? I’m somewhere on this difficult path and would love to hear a good story. Contact details are in my profile if you’d be willing to share.

I'm about six months into figuring it out and only now getting a feel for who I am without all the pressures of making a living in my chosen career.

I'll drop you a line via your profile contact info and maybe we can have a chat!

The most insane people I've worked with were at small companies (under 100 employees). A lot of bullshit is tolerated from early employees and cofounders. Big company pathology seems much more straightforward. It's not easier to deal with, but it's mostly just a bunch of assholes climbing the corporate ladder and a lot of incompetency.

* To add to this, the crazy people were slowly pushed out after we were acquired by a slightly larger company.

That's not the takeaway at all.

The takeaway is don't be good at things you don't want to be asked to do.

> Hell, once I almost got hugely punished because I didn't let my boss take the credit

... I'm mentally tripping over that part. Is that normally expected? ... By recursion, it seems that POTUS should get all the kudos, all the time.

In the military it’s extremely common for someone above you in the Chain of Command to get the credit; it may be your boss, or several levels up.

Some of that credit may or may not spill over to you, but the general thought process is that they are responsible for what happens in the command so they get the credit. Good ones avoid that and push the credit to where it belongs, but then they don’t promote. This leads to an incentive to take credit from those under you in order to get promoted.

Right, but someone /other than your boss/ was mad that you didn't arrange for your boss to receive the credit. That's the WTF detail.

Correct; the fact that my name was on the presentation that went before Congress, vice my CO's, indicated to them that I went AROUND my CO.

In reality, this is certainly possible. It just would have been a career killer. My CO was the one who sent that presentation up with my name front and center. He had to defuse the situation.

Also, and this is absolutely relevant, I was not a Commissioned Officer. I retired as a Chief Petty Officer, though I had a degree (Nuclear Engineering) and multiple "graduate level" certifications (PMP, LSSBB, CISSP, etc., etc.). There is only so far competence and capability can take you without rank in the US military. I chose to exit rather than move over and promote, which was the right decision for me.

I apologize for lacking the context in my previous comments; I often forget people don't understand the intricacies of the arcane ways of the military unless they subjected themselves to it.

It's like a dark-mirror version of a really good principle in leadership philosophy, which was outlined in Starship Troopers by Robert Heinlein. "Everything that happens under your command is your responsibility." If a man fails to do his job in the correct way, it's because you failed to make a system that would train him properly, and possibly because you allowed the wrong man to be hired. Technically, every success is also your responsibility, but a smart leader spreads blame for success and privatizes blame for failure. In reality many leaders pretty much do the opposite.

Starship Troopers is on many recommended military reading lists for career professionals for very good reasons.

Functional fascism is a fascinating concept. Surprisingly excellent in the corporate world.

Presidents routinely take credit for all sorts of things that happen without their involement or knowledge. It's part of the tradeoff of expecting them to be omnipotent.

I had an industrial engineering professor who did a full career in the military, focused on energy efficiently.

Paraphrasing, "The great thing about the US Government is it's so big, if the optimal solution would save 30%, but you find a suboptimal solution that only saves 29%, nobody will notice because you're still saving tens or hundreds of millions of dollars of savings."

One of his big projects was on energy efficiently in remote military camps. The cost of fuel per gallon was around $100 delivered to parts of Afghanistan. They had tons of portable generators running at ~20-40% of capacity. If memory serves, ~70% is most efficient. By some combination of building a small or connecting a few tents to a single generator, they were able to very significantly reduce the fuel required and improve the quality of service.

> The cost of fuel per gallon was around $100 delivered to parts of Afghanistan

This was one of the most deeply insane parts of the whole twenty-year war. Of course, wars have unlimited budgets, so trucking gasoline thousands of miles through hostile territory with occasional losses and then using it to aircondition tents was going to happen. While back inside the US teachers were having to buy school supplies with their own money, because only wars have unlimited budgets.

Also using Blackwater mercenaries so they could just brush any of their deaths under the rug as they typically didn't use Americans for dangerous and unprotected jobs.

How much did Blackwater do? Did they actually fight or just guard?

AFAIK Blackwater was mostly for guard duty, but I'm sure in practice it wasn't just for that. There's also a lot of smaller orgs out there that do "guard duty", not just american (for example britain is also active).

Also there was a point when Blackwater boasted that they could deploy a light brigade (~3k troops) anywhere in the world, people in the Pentagon figured it out and very quickly killed that org. Compare with Prigozhin and his adventures, just for reference he had more than 10k soldiers under his command, on heavy equipment (arty, tanks, ifvs), permission to illegally recruit from prisons, he even had air defence like Pantsirs, the motherfuckers even tried to get nuclear backpacks (man-portable nuclear bombs) from a storage site near Voronezh.

I think those entities are being tolerated and even welcome, as long as they don't exceed some potentially threatening size.

> Also there was a point when Blackwater boasted that they could deploy a brigade (~3k troops) anywhere in the world, people in the Pentagon figured it out and very quickly killed that org.

That... didn't happen.

Blackwater went through several reorgs, rebranding, and acquisitions, initially starting with at least superficially deemphasizing security in favor of other functions because of “business risk” just after the Nisour Massacre, but (as Constellis Holdings) is still doing the things it did before, notably as part of the Saudi intervention in Yemen.

https://web.archive.org/web/20080615215345/http://hamptonroa...

But largely you're right, the story is more complicated. I originally heard it in a non-English interview with some Colonel (OF-5, the rank just below General of brigade).

Personally I am very much against the idea of a democratic nation allowing a private company have a combat strength of an infantry brigade. Definitely too much. (And of course such structures, if they exist, must be penetrated by intelligence, though this is not something you can legislate or even admit).

Right, I wasn't disagreeing that they said that they can supply forces (and that article is a year before Blackwater forces were involved in the Nisour Square Massacre in Iraq), just that the Pentagon didn't shut them down in response to them saying it, though the relationship wuth the government was complicated by what troops they supplied did not long after that.

Yup, mostly guarding important people and locations. Though they had itchy trigger fingers and shot up a lot of civilians and cars, and a massacre or two. Eric Prince also wanted them to be a mercenary army for hire. He fooled some Swiss engineers to adapt a small airplane so it could be armed with missiles and tried to sell that as a service (believe it was confiscated on the way somewhere in Africa to be used against insurgents or something).

Also pitched a plan to the pentagon about privatizing their presence in Afghanistan and extracting their natural resources to pay for it. https://www.militarytimes.com/news/your-military/2018/09/05/... (didn't vet this source)

>Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects

>(I hate every single bit of terminology in that entire godforsaken sentence.

Checks out. :).

A super convenient way to do cool stuff like that in the DoD is to do it for the most senior Flag you can get interested in it directly. Projects at SECDEF or SECNAV offices work...differently than outside, as there isn't really an 'up' left for most of the folks involved (in my experience - most are extremely focused on getting the job done and/or the geo-strategic problems).

During the time I worked in that position I was one office removed from COMFRC, which was the Flag I generally reported to via my CO (who was the one who put me in that position and was highly supportive, and who ended up as COMFRC in later years).

Anything that was directly interfacing with that office was great; as soon as I was detailed to do something downline from that office it got painful despite reporting to that Flag.

I wasn't in a position to go higher without spending more time in the service, and that was a non-starter for me.

My favorite quote: "When you do the impossible, it becomes part of your job description"

"Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing."

No good deed goes unpunished.

A part of me finds I might know you. Another part states: this story is way too common.

"Beware of doing good things at work"

You do have the option of becoming self employed. If you do, then you might discover a very different perspective. I've been doing it for 23 years now, with two partners and around 20 odd employees. IT company, you wont have heard of us but you will know some of our customers.

I'm still sort of working now (its ~0030 here) keeping an ear to the ground - HN, r/sysadmin (int al), The Register and a few others are some of my canaries. I listen to their songs. I also have a BirdNetPi doing the same for actual birds too.

If my company cocks up it will probably be fatal. I/we don't get the luxury of blaming someone else and I certainly don't get to mumble some sort of bollocks about "Your x is very important to us, soz we failed, lol and that - here have some free credit rating checker thingie".

Why not try running a small company and living on your own wits with no mother to land on - you are mother!

one day, maybe. Today, I personally lack the savings buffer nor desirable kinds of skills to pull it off properly. Wasn't born with a silver spoon, so I guess I gotta pay my dues.

Even then, my kind of self work is just as unstable as my industry. I may in fact be good at it and still fail.

You're probably just unhappy with what you do and don't realize it. I am on my journey out of exactly what you're describing, and it came down to me realizing that I'd rather make less and solve real problems, instead of being an overpaid Copilot operator.

I'm a bit angry at the current market, but I like the work I do.

And sadly, I'm far from overpaid (you can probably guess my industry in 1-2 guesses now. Not like I keep it secret on this profile to begin with). I really do want to do my own work without being binded to corporate whiplash, but I still have a mortgage and car to pay off. And I need to rebuild my 6 months savings after the current year of layoff.

I'm going to start laying down the foundations on the side next year, but I see at least 5 more years before I can even consider jumping off.

Aw man, gamedev is rough. That's a bad spot to be in today's market. I'm primarily frontend, but now work as solo eng for a sizeable nonprofit. So much better, I'd say 30% of my work is code and the rest is people work. Absolutely fantastic balance.

There's always career direction change. Lots of folks are eager to hire gamedevs, because they work hard and think different. Not sure if that's something you want, but perpetual burnout is never worth it.

At the wrong gig, with the wrong bosses. I have the misfortune of being at one of those bad places now. I was at a much better place before, where if I went out of my way and saved a couple million, I'd actually be congratulated; boss would even let have the credit.

Never stay at a job where the culture is toxic. Even if the money is better. Nothing eats at the soul like being held down by feckless, petty careerists and control freaks.

This. 1000x this. Even if it's a FAANG, do it, there's no shame. I had a good run at one, then the culture went to crap due to one of those long boring stories that still can't communicate how...easy it is for things to be awful, even when they shouldn't be.

When they are, just move on. Life's too short. Whatever blows you up with $2 million in savings would blow you up at $500K

I've been trying to leave my current gig for the last 9+ months. It's tough to find the right roles these days

I agree that getting out of a toxic culture is a great idea but you can also just check out, give them what they want and save as much money as you can. Especially if your goal is to eventually start your own company.

Ideally, you should be working for a company that pays well and that actually rewards you for doing things that are good for the business. But it does seem like virtually every company has some kind of dysfunction.

Checking out literally turns me into an alcoholic. I can't live like that. I'm not an entrepreneur, I'm just a guy who cares way too much about wanting to do a good job.

I agree though, there is no perfect place; wherever you go, there you are. Still, if you're unhappy, you're not going to be less happy by changing to a better job.

This is the source of so much despair for me, I have lived it in two jobs. I want the stability of nice salaried gig but I know that if I do well, my managers are going to see that as slack and I am going to be overworked to a breaking point. And the greedy fucks would never agree to a rev-share or profit-share, no just here's your annual 5% increase now shut the fuck up and do your job.

A few jobs ago I was asked to list the names of projects I was actively maintaining for the company, that list stretched out to two entire screens in Excel at default font size! I was burnt out to the point of severe depression.

Even worse is I hate the bullshit ritual of submission that is job hunting.

I can relate. The feeling of being undervalued and taken advantage of is poison for the soul. It's gotten to the point where I optimize for financial independence, maximum income, minimum hours worked and maximum free time.

> I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.

Preach. 100%.

This reminds me of some of Dan Luu's stories, https://danluu.com/nothing-works/

Likewise with chip software tooling; despite it being standard to outsource tooling to large EDA vendors, we got a lot of mileage out using our own custom tools, generally created or maintained by one person, e.g., while I was there, most simulator cycles were run on a custom simulator that was maintained by one person, which saved millions a year in simulator costs (standard pricing for a simulator at the time was a few thousand dollars per license per year and we had a farm of about a thousand simulation machines). You might think that, if a single person can create or maintain a tool that's worth millions of dollars a year to the company, our competitors would do the same thing, just like you might think that if you can ship faster and at a lower cost by hiring a person who knows how to crack a wafer open, our competitors would do that, but they mostly didn't.

I think about that, and his culture post ( https://danluu.com/culture/ ) a lot.

There are some companies that represent the opposite of what OP describes. Once you work with one of them, I've found it impossible to settle for one of the bad places.

Have worked in EDA and .. yeah, the software is bad but the buyers are conservative, precisely because it's a risky and expensive business.

He talks about a "cocktail party version of the EMH". The credentialed econ version of "nothing works" is https://en.wikipedia.org/wiki/The_Market_for_Lemons , which discusses the role of information and its asymmetry in markets. The EMH fails precisely because total knowledge is impossible and adverse selection is real.

The whole post is gold.

- managers asked how it was possible we saved that much without help from them

- asked to prepare slides

- asked many times on how it happened

- had to roll it out slowly to make it look like they did it over time incrementally vs one small toggle

- asked for a raise due to impact and did not happen.

Sir, for your sake, apply to a FAANG or something, you'll be at least taken care of better.

Also please implement Twitter card metadata in your blog so it looks better on twitter :)

> “please implement Twitter card metadata in your blog so it looks better on twitter”

Didn’t Xitter recently remove the display of all external site metadata except the image?

Yeah people use image interestingly these days though

See, for example: https://twitter.com/simonw/status/1717768637799706922

Maybe my friends are all just bad devs but I don't hear of people getting treated well at FAANG anymore. (Though I haven't heard anything about Netlifx recently.)

Maybe they are busy working - after while layoffs everyone is working a bit more to say the truth

I'm at a FAANG, still getting treated like shit. The problems outlined in the article are 1000x worse in a FAANG and you'll have the same 99% of the people that have drank the proverbial kool-aid telling you everything is great and it's supposed to function like this. At least you get paid more for it though.

True, and not true. It's nuanced. It depends a lot on the org.

I have moved from admob (google ads) to robotics to facebook, and worked with different orgs, and have seen differences.

I think we just keep the N in there so it's less awkward to say.

Facebook is now Meta, Google is now Alphabet, Netflix is now just another independent movie/TV studio. FAANG has become a term entirely independent of what it originally stood for (also "MAAA" doesn't have the same ring to it).

I think it works. Just think of it as a family with a troublemaker kid.

MAAA! THEY'RE DOIN STUPID STUFF AGAIN AND THEY WON'T LET ME USE MY COMPUTER!

MAAAM or MAMAA

You don't hear "Alphabet" a lot. MAGMA works

As someone outside of the tech industry - Big Tech tends to be the term I use the most and is mostly understood by everyone.

FAANG simply reflects what those companies are actually called. Most people don't say "Meta", they say "Facebook". And Google is still Google, they never changed names (they invented a parent company but still kept the name).

With the rebranding of Facebook, we could go for MANGA, but dropping the "N" would still have an interesting result.

You could call Google Alphabet and say MAAA. I’m partial to channeling my inner cow when pronouncing that.

In my part of the world that would be inner sheep.

It would be quite the GAAF

Outside of the Silicon Valley they are called GAFAM (Netflix out, Microsoft in).

Im surprised no one asked to automate an email to management of the status for the rollout of changes.

“We’ll need the percentage complete as well and a summary of the savings expected. It needs to be sent out every Friday at noon.”

At least at fang you can write a mostly plaintext document instead of PowerPoint. Downside is you would probably have to do it before making the changes, have it reviewed by the entire team and get “alignment”

Yes and no, changes based on the org

But you would get handsomely paid.

Skip FAANG and go straight to a company that'll value you!

Lol, if I wasn't given a raise, I'd agree to everything, and at the start of my presentation say

"hi everyone, I wanted to walk you how we got to a half-a-million dollar savings, basically I spent a day looking at how terrible the original infrastructure was deployed and removed a code test feature that was causing the problems. This was just complete oversight from every aspect of the development, management, testing, and everything. Overall this code is as bad as it can possibly get, and we just launched it. And basically I was told not to say any of this because it makes everyone look bad, so I was to roll this out gradually to make it seem like managers were doing some sort of work."

Then drop the mic and walk off stage.

Honestly the amount of give-a-fucks I would have lost would have been a lot. And this is coming from someone who's done this for almost 2 decades and cares about his job because bills to pay, kids to feed.

Back in the day I worked at a company that if you came up with some long term cost saving measure, they gave you a bonus of 10% net savings for the first year.

A co-worker in the early 90s (he was a tech writer) told me of a cost savings device he invented in the 80s at Texas Instruments to fix a process where occasionally a mirror on a very expensive piece of military camera gear got scratched (I think it was during field disassembly). It was basically some forceps with more metal welded on to make them longer and that allowed you access via a different route than where the mirror was installed. TI gave cost savings awards as a percentage of money saved and he did very, very well with that little invention.

That's a nice incentive.

Also an incentive to sandbag: ship a costly feature, then next quarter ship the savings.

Cloud services could make this really easy to game nowadays

They sure don't do that anymore... At least not at any of the Dallas factories. Even patents reap little money. You practically sell your soul when you join the company and all your ideas are theirs.

This was in Dallas, I believe. But this was in the 80s before the Peace Dividend when there was a lot of defense money sloshing around. I met him in Chicago in the 90s and he, and a lot of TI folks, had left Texas as the defense related work dried up. The team I was on in Chicago was working on the flight data recorder for the F-22 which still had funding.

Oh... I believe it.. Times have changed though. They cut back on travel, corporate crédit cards, etc. We would have no less than 2 cost marathons per year of all day meetings.

Sounds like standard operating procedures these days.

I would say blanketed bonuses are a bad idea, or I'd make some partnership deals with other engineers. However, this was a pretty clear cut "give this man a medal" situation, in a clearly toxic company. This is why my reaction is what it is.

There was an old thedailywtf post about how a company thought they'd incentivize fidning and fixing bugs. Suddenly every engineer had a QA buddy, and they'd make like 50 spelling errors, which QA will find, and enigneering will quickly resolve. They took down the bounty within a week.

> There was an old thedailywtf post about how a company thought they'd incentivize fidning and fixing bugs.

That might have been this Dilbert comic:

https://i.stack.imgur.com/bQOvF.png

There's a name for this anti-pattern, the Cobra Effect[0]:

The term cobra effect was coined by economist Horst Siebert based on an anecdotal occurrence in India during British rule. The British government, concerned about the number of venomous cobras in Delhi, offered a bounty for every dead cobra. Initially, this was a successful strategy; large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped. When cobra breeders set their now-worthless snakes free, the wild cobra population further increased.

(There may be some question as to whether these events actually occurred or not, but there are similar examples of documented pest-control campaigns (and others) on the Wikipedia page[0] where similar things happened).

[0] https://en.wikipedia.org/wiki/Perverse_incentive#The_origina...

ah yes, thanks for that.

Yeah I've been looking into the cobra effect in the current youtube adblocking thingy... It certainly got me to use primarily adnausium which fixes the problem since ads are served, just... you know... maliciously clicked on.

I have seen higher splits than that. You have to remember this is bottom line cost output saving so the 500K if you look at it on the sales side is like completing a +/- $2M deal in terms of net margin added to the business.

I did save one company enough money to finance my salary for the year I worked there, just by removing dead servers provisioned but no longer used. Seems like my predecessors were blind to both cost and the past.

I felt bad for leaving so soon, but good for not having cost them a dime.

AWS architecting class years ago. Awesome instructor said if you want to save money, start deactivating servers. The ones people use... the admins will contact you right away. The others... people started up and forgot about.

Can work great, but watch the incentives. Don’t overoptimize a business process and later fix it with a 10% bonus.

that NEVER happens anymore. and I've only ever heard stories of it, no one I've ever spoken to has ever gotten anything like this.

when I have made large improvements like the article/blog describes, I am pulled off of that and put on something far worse, immediately, except without the autonomy. "why can't you succeed here?"

While this feels good to imagine, the social fallout would be disastrous.

It really depends on your priorities and operating environment, but there are many people who would be leaving anyways if they save the company that much money and were given exactly zero reward, and at that point plenty of people would be happy to burn bridges on the way out. Like I said, it really depends on the environment and your priories.

Tech is a small circle. Unless it's something obviously wrong, you may want to be at least milder :)

The thing about burning bridges is you don't know when you'll need them.

Let's say 5 or 10 years later you're applying to a job where one of these upper-level people now work. How do you want them to remember you? The know-it-all who wasn't a team player and kind of an asshole? Or the engineer who gets things done and has demonstrably shown to land impact and value, an engineer the exec would consider lucky to have?

Some of you will say you wouldn't want to work for one of these executives again. But people change, incentives change, the environment changes. Have you ever made a technical decision you later regretted?

And maybe you don't work for them. Maybe you're applying to a different company where someone knows these previous upper-level management folks and they ask about you. How do you want that recommendation to come across? "That engineer was an asshole.", "That engineer was amazing, I wish we could have kept them. We made a big mistake by not trying to keep them.".

You are -mostly- right but there is also the factor where a specific sort of negative reaction can actually function as a recommendation later.

I've definitely missed out on work sometimes due to having a reputation for being about as subtle as a brick to the face with no lemon, but I've also -got- certain pieces of work as a direct result of being criticised and somebody who heard the criticism thinking "if he annoyed that person that way, he's probably serious about doing the right thing."

I would, however, suggest that probably I would've done better overall if I'd toned it down a bit.

Would it? I mean, the people you're throwing under the bus would hate you, but the top of the company should love because a) you saved them a ton of money, and b) you identified a pile of incompetence in their company.

And who's going to fire you for this?

The people telling you to roll this out slowly are doing so mostly to protect themselves from having their incompetence exposed and to appear useful. Protecting them will help them steal your credit and will get them promoted.

It would, I replied to someone else below to explain why. But in general, it's best for your future career to be remembered well.

That sounds like a very nice and theatrical outcome, but in reality nobody wants to believe they've been horribly wrong about their understanding of things, which means the higher-ups are going to be talking directly to the managers to get their side and figure out how they can show you've misrepresented the situation. They're not going to fire managers en masse because the new guy said "Everyone here is dumb and cowardly," even with evidence.

this assumes that the "top of the company" is a shining tower of competence, blissfully unaware of how incompetent the people below them are.

more likely, the top of the company is just as incompetent but has lucked / gamed their way into promotions anyways.

So this engineer throwing his managers under the bus may be good for the company in theory, but to the senior management this is a red flag. they don't want engineers who do this - they want engineers who give credit to their managers. if they promoted this engineer and fired everyone else, he'd come for them next.

I would assume that the top of the company cares about money, that saving money is a good thing, and that hiring incompetent people is a bad thing.

Of course if everybody quietly agrees that this company is a suitable vehicle for their incompetence, then it's a bad idea to address this. And some companies definitely are.

This is a deep misunderstanding of human psychology.

> the top of the company should love because a) you saved them a ton of money

The top of the company is probably already rich. Being richer is great, but one of the few things rich people generally won't burn to run the money-making engine hotter is their own sense of prestige and entitlement.

> b) you identified a pile of incompetence in their company.

Yes, and you told that fact to people who were already responsible for identifying that, which means you just told them that they are incompetent too. And that transitively works its way all the way up the org chart.

You would make the whole chain of command lose face and do so in front of the rest of the chain of command. It would be career suicide.

Every manager would rather silently waste money than be made to look like a fool. Because the money comes out of the business's bank account, not theirs, but looking stupid affects their personal reputation.

As the common quotation goes, “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

I'd consider that if I already have a job offer from another company starting next week. Otherwise it is self destruction.

"Got management material written all over him"

I worked for a large telco that operates very, very similarly to the company in the article.

About once a year or so one of the stand-out engineers that had the weight of the world on their shoulders would get burnt out and frustrated enough to do exactly what you suggest above.

Literally everyone would just look around awkwardly, leave the meeting and never talk about it again. All of middle management already know all of this, the only way they keep their jobs is by never talking about it, and just ignoring anyone that does. The VPs and President only know what those below them feed them.

Your fantasy resonates. Sometimes I really wish I had F.U.-level savings.

> Sir, for your sake, apply to a FAANG or something

Noooooooooo

You definitely havent worked at a FAANG company.

"Back in the day" I identified a way to save prob a quarter million dollars a year on my project, funded with your tax dollars. Created a spreadsheet model for the cost, documented it all, and gave it to my boss. He said "I'll look it over". I asked his two weeks later if he'd been thru it. His response was "It looks correct. Nice job." I asked if were going to trial it. Golden response: "No, we don't get paid to save money; we get paid to spend money." And that is when I truly understood Cost+Award Fee contracting with the gov't.

Anecdotally I have worked in government roles and have been rewarded extremely well for saving taxpayer money, to the point where I felt it was too much at times. It really all dials down on the common denominator, which is the boss or manager. I've learned over the years in both the private and public world that if a manager is not willing to go out of their way to reward their great employees, it's time to move on to the next role.

>managers asked how it was possible we saved that much without help from them

Wait, it wasnt the managers that designed the overly complex solution! this is all on the engineers

You didn't accidentally save half a million, you deliberately and intentionally saved them half a million, but now you regret it. That's not the same thing.

Large organisations are so woefully inefficient that I'm surprised they're able to compete at all, but they have a ton of money and economy of scale and all that, and along the way there's more than enough money to waste millions on stupid nonsense and inefficiency and nobody really cares.

In my mind, there is where inflation really comes from. For every wage paid that wasn't useful, the resulting product becomes more expensive / less profitable. If we were able to optimize out all the waste we'd probably have deflation while computers and business processes keep getting more efficient

I mean waste is probably a source of inflation but lots of it just comes from unrestrained profit-seeking behavior. If a company believes it can raise its prices without impacting sales, it will do that. That's why so many massive corporations posted record-breaking profits during the recent bouts of inflation.

Government shutdowns that made it more difficult or impossible to shop outside of big stores or online would be as big or bigger issue here.

It wouldn't explain simultaneous mass layoffs.

Except that generalized deflation is one of the largest and widespread economic disasters that can be brought about; you are disincentivizing immediate consumption to on the basis of future larger consumption, you're disincentivizing investing in the economy instead of just holding capital - both things that will make a economy stop working in the long term.

It's the reason why most central banks really want some minimal level of inflation going on at all times.

...no.

Inflation is actually very simple: More money is added to the system than value produced. That is, money is printed faster than value is created through labor. Stop and think for a second. How do we have trillions of USD? Banks create money. And they're doing it faster than ever before.

Now you can absolutely have price gouging at the same time, but the two are independent of each other, even though combined the affects are worse for the price gouged.

I think OP is saying that the creation of high paying useless job is causing the bulk of the money that exist in the system to not mean much. Which is why we have inflation, because the increase in the amount of money does not scale with the amount of "value". Imagine if government is spending half of the money printed to people staring at walls. This would naturally double the inflation because it requires double the amount of money to produce the same amount of "value".

So the government is pumping money, but is not getting the desired signal, so it pumps more money.

Only if those people are actually required to stare at walls for that money. Because having received money, they will want to spend it on stuff, which creates demand. Only when that demand can't be met because people are required to stare at walls instead of creating value, do prices go up and cause inflation.

So there's another reason to replace welfare with UBI. Seems every discussion leads to that these days.

Price gouging can also be inflation. Inflation is simply measured by how much prices go up. If companies just raise prices to reap more profits because nobody is stopping them, as seems to be the case right now, that's still inflation.

Of course this kind of inflation should be fought by competition, but you have to have companies actually willing to compete on price.

It would be helpful if you'd point out what term is appropriate. Given the reference to "my country", odds are that they're not a native speaker. (I can't tell from the English though, but I'm not a native speaker myself.)

Another way to look at it, you intentionally saved the company half a million and you got paid what you think was the fair pay for it. It seems like the author might think it's too much money but the irony is that someone is clearly undercharging (helping skew salaries in the entire industry lower, of course) while execs get bonuses. There's nothing to regret if you can make ends meet, great job.

> I'm surprised they're able to compete at all

They typically don't. When the government doesn't enforce it's own rules, these companies just buy up competition. They strangle the market and they get more inefficient all at the same time.

> Large organisations are so woefully inefficient that I'm surprised they're able to compete at all

It's easy to understand: large orgs are competing against other large orgs that are similarly inefficient, because efficiency in a large org is a really, really, really hard human problem that we simply haven't solved.

And remember they're still providing a valuable service/product. For all their inefficiency, they're still more efficient than not existing at all. This is why they would exist if there weren't competition.

And you might ask, what about competition from small orgs? Well, a small org has less efficiencies of scale, but if they are efficient otherwise, they can sometimes compete effectively with the large org. But then they grow into a large org and will wind up with all of the large org inefficiencies.

It's just how things are. It's not that nobody cares -- to the contrary, company owners care hugely. It's that literally nobody has any idea how to solve it.

> It's easy to understand: large orgs are competing against other large orgs that are similarly inefficient, because efficiency in a large org is a really, really, really hard human problem that we simply haven't solved.

We used to break up companies that became Too Large. Either they became so large that it was impossible to compete against them (as Large Company can demand way better pricing on volume than Small Startup can or has way better access to financial instruments due to better ratings, or like many automotive companies even run their own goddamn bank), or because they could use a wildly profitable business to price-dump competitors out of existence or because they became so large that they felt free to extort their customers to the tune the people would complain about it in the media/their congresspeople too much, or because they became so large their sheer size represented too much economic risk ("too big to fail").

We should begin doing that again. Something as big as Google/Alphabet, Apple, Amazon, Meta/Facebook, Walmart, the Big Four consultancy shops, virtually all major banks - there's absolutely no reason these should be allowed to exist at their current size. Or, if these companies still wish to exist at their size / their existence as one platform, they at least have to be regulated to mitigate the threat originating from that scale.

None of that has anything to do with efficiency, though.

When I'm talking about how large companies have inefficient processes, it doesn't really matter if the company has 10,000 employees or 100,000.

I'm constrasting with companies that have 50 or 500 employees.

> When I'm talking about how large companies have inefficient processes, it doesn't really matter if the company has 10,000 employees or 100,000.

My argument is that there should not be companies this large at all. Everything above 100-500 people tends to breed a layer of pointless middle management layers and a host of supporting bullshit jobs, and in many cases the lack of internal competition leads to ossification.

So, yes, of course one may argue that without this one couldn't build something as complex as an airplane, a GPU or whatever... I'd say, keeping a cluster of teams as distinct companies, maybe with some sort of "service company" that deals with shit like payroll, travel expenses and whatnot, is actually more efficient as it offers a chance for people to get (literally) invested into their work and have the profits end up in their wages as well instead of everything being siphoned off in increasingly opaque ways. And it enforces proper working processes (e.g. documentation about interfaces), and so actually reduces the chance of stuff going wrong.

The only reason 99% of large companies exist is they got bigger than everyone else first, and that gives them the power to keep everyone else down.

Competition is a fairy tale we tell MBAs so they'll start new businesses so it looks like there's competition. They never had a chance.

> I'm surprised they're able to compete at all

Competition is usually eliminated by one or more "moats": huge capital upfront investment, patents, market lockin agreements (e.g. the way Windows is sold to OEMs), vertical integration (e.g. Apple), compliance (it's genuinely hard to start a bank or a hospital), buying out competition, buying out the staff of the competition (why FAANG have so many highly paid staff doing work that gets cancelled) or even less legitimate means.

Large companies tend to end up with similar failure modes to authoritarian state-controlled companies. It's interesting how well China has been able to ride the line between control and growth on this one.

Once upon a time I uncovered a bug that recovered $4MM/year in revenue. It was swept under the rug to protect the team and executives that let the blunder continue for as long as it did. I didn't get a raise, but I made some allies and got to coast for a while.

I did something similar early in my career but it very much went noticed. The system I wrote to correct the bug was basically a MITM attack against pharmacy rx transmissions, it re-priced prescriptions just before they went to the insurance carrier's system because the pharmacies would never apply price updates (i worked at a small independent pharmacy chain, they didn't have a central dispensing system). In my infinite genius I named the system after my current crush. Corporate liked it so much they made an annual award named after my system... so therefore after my crush but i was too embarrassed to tell them the real origin of the name. That was 25 years ago but, to this day, every year a trophy gets made with the first name of my old crush printed on it and handed out. heh if she ever knew she'd be mortified.

You must tell her then update us on her reaction. Bonus if you record it on video. :^)

Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the two machines were ever rebooted. The two servers were related to some feed data and I had noticed that they were cabled together oddly based on how their function was described to me.

The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG.

I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly.

EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.

Keep doing the right thing.

It's 20 years later and I've made my career off of pointing out fundamental mistakes made by very smart people. ;) (just with more skill and tact)

90% of what I do is ask dumb questions as if I'm completely clueless.

I know the feeling. I'm similar in that regard. Debugging complex things by asking dumb questions is extremely fruitful.

Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves.

One of the things I'm hoping I could keep alive in my child

>> EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team.

People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.

On his book "The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit.

Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.

I'm only just realizing now why I was treated so strangely after discovering a pretty severe security issue that had been in our software for about 7 years. I had only been in the team for about 12 months when I discovered it.

The only time I have seen a good response from the discovery of a large security hole was when it was by total accident, and the person mentioned it in a public channel by accident. They almost got fired, but the bug got fixed quickly.

The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.

Why did you copy-paste someone else's comment?

This was a fat-finger mistake. I copied the comment for my notes and somehow fired off a cmd-v while in the comment window. My apologies. This was not the intent.

But then you submitted it anyway?

Yep. Not sure how it happened. I copied the note to my personal notes. I also wrote a comment. I think somewhere between the "submit" click I cmd-v'ed. Woke up surprised today to see this.

I can't remove it either.

I'm open for more questions.

I think you are good - you just had to pass a HN interrogation first. It really delivered though.

> I'm open for more questions

I laughed much harder than I should of at this.

A company of maniacs, but that was a good lesson. Often it’s better to just stay silent and watch. It’s brutal to watch, but better to be the bad guy for pointing at obvious flaws.

Or find and company that values your talent and knowledge.

Reminds me of the Preventable Problem Paradox: https://medium.com/@shreyashere/why-our-leaders-fail-us-and-...

If you had just let the crisis occur, everyone would get a chance to spring into action like heroes, handshakes and champagne all around on a job well done.

This kind of thing has been going on forever. Once upon a time, I walked into a room full of VT100 terminals, and bored CS students waiting for their compiles to finish. (It was a weekend near the end of term) I took a look at the system, and realized they had pushed all the compiles into a batch queue, but that queue defaulted to BELOW interactive priority, so any keystroke anywhere had higher priority... so all the people checking their position of the compile job in the queue, slowed it down even more.

Over the next 15 minutes, I kept bumping the priority of the top job in the queue up, an hour later everyone had their work done, and went home. I had the room to myself. Rogue sysadmin for the win. ;-)

I saved my company $1 million a year a couple of months ago by noticing that there was an S3 bucket that kept growing and costing 80k a month.

I poked around and realized that there was a system that we weren't using anymore that was copying files to the bucket I reached out to the stakeholders and they turned it off and we deleted the files.

The higher-ups didn't seem to really care, My boss's boss told me to reach out to another team that should've caught this and that was about it.

This is what is going on to a Saas provider we are using. I told them about but they shrug it off..VC money I guess :D

"For example, you've run 234,745 INSERTs into [table name deleted]_HOURLY and ALL of them were under 1000 rows."

That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading.

I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.

Long time ago in a galaxy far far away, I replaced the need for N * (Oracle license + Sun server) with a simple perl script (< 100 lines) and one Sun server for a total cost saving in $300M+ / year. While doing it, I also invented map-reduce (it was before Google time).

The problem was to calculate bunch of stats from web servers logs (e.g. 10 most popular pages). The original solution was loading it all into Oracle database running on multiple servers since logs were huge. And then running bunch of SQL queries. Rinse and repeat daily.

The author's gallery of poignant HN commentary on their writing is chef's kiss: https://ludic.mataroa.blog/compliments/

Some of it is fair. The author does seem a bit egotistical at times. They begin from a premise that "Everyone else is incompetent and I am the savior".

There is a high likelihood that many other people in the org were lightyears ahead of the author. It's even more likely that an engineer or manager in their group was "Banking" the inefficiencies to use during a cost-cutting period - and the author ruined that chance... which will inevitably cause massive suffering and possibly poor performance reviews when there is nothing to trim.

The superior tone of the writing is wild to me, because it isn't exactly a story where I think the author looks great. Certainly others look worse in the story, but still, finding an issue and apparently barely trying to convince others it's an issue is not impressive.

It really sounds like they stayed in their lane and did their assigned job, just like every other person on their team.

> They begin from a premise that "Everyone else is incompetent and I am the savior".

I have worked with someone who was like this. Perhaps unlike the author, they were not in fact the savior in a room full of idiots.

This person was no doubt clever and had interesting experiences to learn from. But they were also dysfunctionally arrogant and utterly resistant to ideas that were not their own, to the point where it repeatedly and painfully got in the way of their own progress and other people's progress around them. At the end of the day they got done what they needed to get done, but they left a big mess behind that took weeks to even stabilize (as in, "prod is broken, prioritize above all other work") before the rest of the team could get back to their regular work.

May such people forever find themselves in solo contributor roles, so as to spare others the pain of being their close collaborators.

I appreciate your perspective, which brings up an interesting point for discussion. It's not uncommon for individuals to have strong opinions, and it's equally important to consider alternative viewpoints. While the author's critique may seem overly assertive, it could also serve as a catalyst for evaluating our processes more critically.

In our collaborative environment, it's valuable to maintain a level of respect for diverse perspectives, even if they initially appear confrontational. Instead of dismissing the author's stance outright, we might benefit from conducting a thorough assessment of our current procedures and identifying areas for potential improvement.

Engaging in constructive dialogue allows us to pinpoint inefficiencies and work collectively towards solutions that benefit our organisation. Ultimately, it's our ability to learn from each other and adapt that will contribute to our long-term success.

Reminds me of the sadly dormant @shit_hn_says [1].

I'm guessing it's dormant because the level of discourse on HN has devolved to a point where nearly every comment on HN would now qualify.

[1] https://twitter.com/shit_hn_says

This is such a good article. My skin was crawling.

What really scared me is that I couldn't identify any of the issues raised in my own organization, even though we often run into similar, smaller-magnitude problems caused by a blindness to obvious mistakes. It makes me fear I, too, am blind to massive bleeding wounds. Here's hoping they actually don't exist.

This reminds me of the company I worked at that was paying for dozens of unused Redshift clusters at any one time, peaking over a hundred. There were common data science (read: ETL configuration) tasks that required spinning up a Redshift cluster, and the "data engineers" (read: fresh grads with a few weeks of training) sometimes forgot to shut them down afterwards, or got diverted to different work temporarily and couldn't find the original cluster when they returned to complete the task.

The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was.

The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.

>I am asked to write some PowerPoints, which include phrases like "a careful statistical analysis of user usage patterns indicated an opportunity to more effectively allocate resources", implying that nothing was wrong, we just needed to collect more data before deciding not to let the expensive machines idle all day.

Yep.

If you don't make them realize it was a Hard Problem™ that was only solved by their smart hiring, funding, and task-deciding, you might shatter their whole world view.

I have to play devils advocate here because for every one of these cases, there is probably a dozen of similar stories where the ambitious new guy actually did nuke the system with a risky friday release and then logged off for the weekend xD

Let me hit you with the extra spicy take - since the department doesn't really produce business value anyway, taking the system offline for a week would have dropped the bill to 0 and been the greatest possible cost saving.

Yes, management would have killed me, but I would have been absolved at the Pearly Gates.

Another’s devils advocate is that part of being a senior engineer is writing good public facing one pagers that communicate the problem and the solution.

Sure you shouldn’t have to do that for every single change. But when something significant comes along it’s wise to document it and socialize it.

If OP had done this they may not have been asked to make PowerPoints or dread conversations or hide anything. They could just point at the public doc that explains it all in technical non political language that also educates other teams how they can improve too. This is leadership and mentorship opportunity.

At $large-scale company this happens pretty regularly if you just look for expensive things and ask:

* "Can I just remove or slowly deprecate this?" I.e this job generates data no one uses, or alerts people just bin-bucket or has been already replaced by faster/ better. * "Can I cache this?" * "Can I run this at a lower-priority/off-peak/less available?" * "Can I reduce the frequency or move processing to deltas?"

This is just the easy stuff you can usually do in a few lines of change without even getting into basic optimization rewrites. Most internal things at $large-co are built because someone thought they might be useful, they might get promoted and using that hypothesis and moved on, but few things are actually continuously validated as still generating value > their cost.

Ask whom, though? It's usually really time consuming to determine that nobody is using something, and difficult to get permission. Hence in so many of these hero stories, the critical work is done without permission.

This morning on bluesky I saw a story of a very hasty onprem-to-cloud migration that was facilitated, and accelerated, by the rising floodwaters drowning the premises.

I love this for purely self-appeasing reasons.

  this machine kills imposter syndrome

or at least it helps. Having a background in solid CS theory from High School, and having a degree in Art, I find it very hard to apply for engineering roles, and my mixed bag of experience often lands me in Support Engineer / application admin / integration roles, fuming tremendously when the people with SwEng / Developer titles fumble on with implementing some JavaScript change for a feature I need in Service Now for my application's customers.

It is incredibly reassuring that it is not simply my organization that is hamstrung by the pretense of complexity, when really someone just made it complicated to make it seem important.

And this is the difference between competence and incompetence in tech. We can argue about whether 10x developers exist, but one thing I hope we can agree on is this: Certain problems can't be solved no matter how many incompetent people we throw at the problem.

This is really what people should be looking for in a hire. If this person joins my team, can I reasonably expect them to bring some sense into our code/infra/schedule? Nonsense creep is constant in a big enough organization, and on-ramping is slow, but asking "what does this even do?" is invaluable (wait, no, it can be quantified!)

We should be realign around "0.1x developer" terminology. Like assume many developers are in the 1x or 2x range, but then there are plenty of 0.1x developers writing scripts that start with 500 lines of comment.

Then job postings would have to be, "we're looking for competence and adequacy. We want to pay you a normal amount and get our money's worth."

I saved Amazon $10MM as an intern back in 2012. If only I could have seen 1% of that.

I also saved Amazon a ton of money and saw nothing from it!

On the other hand, employees aren't liable for the company's losses and debts, so it works out in the end.

In trading/PM roles in finance, before clawbacks became more popular, employees would regularly get the upside and avoid the downside.

Are clawbacks common at trading companies?

> employees aren't liable for the company's losses and debts

Never been laid off during a recession or had your pay frozen and bonuses cancelled during a hard time? Employees risk a lot more than most stock holders by working for a company. On average, stock holders are way more diversified.

losing your job is not the same as losing capital

Correct, is way worse. Proletarians don't have any capital to lose. Capitalists do. So if they lose their capital, they can just be like the rest of us. If we (proletarians) lose our job, we risk poverty and death.

> So if they lose their capital, they can just be like the rest of us.

Nah, if they had enough capital to live off of before then, they are far more likely than us to fail upward into a management job.

No, it is wayyyy worse. Although there is a lot of capital investment in a job. You are also investing the most valuable thing you own.

> Although there is a lot of capital investment in a job.

No there isn't. You haven't put up any capital when you join a job, and you aren't (generally) required to invest any as you go along. Your time is not capital.

Auto Loans, Student Loans, training, certification, internships, moving costs, and other small costs like special clothing are all investments made by employees to be paid for before or during hiring. In addition, many businesses require employees to buy tools and special equipment like boots and safety vests.

> Employees risk a lot more than most stock holders by working for a company.

hmm, they can perhaps reduce the risk by not working for a company. They can just be stock holders or launch their own company, that way whatever may happen they will never get fired.

> They can just be stock holders or launch their own company

In order to become a stock holder or launch a company, you need to have capital. Capital is hard to acquire when wages are being actively suppressed by a cabal of employers. It is also hard to hold on to capital when your health care system is intentionally designed to strip away generational wealth from workers. Then to add a cherry on top, you lock the higher wage jobs behind an additional investment that can only be funded by non-dischargeable loans. Then you have the rent seekers, both literal and figurative. Landlords, insurance companies, toll roads, etc...

so you mean riskier?

No, more like a fairy tale.

Pay frozen? Absolutely not. The company would be undergoing bankruptcy by the end of day if not all employees agreed to it.

Pay frozen != no pay

Pay frozen == no pay raises

LOL back in the late 90s and early aughts I had many friends in games. There were many times when payments didn't come in in time and employees were working for repayment promises.

This isn't a risk the employee takes on as a result of doing business. It's a result of the company choosing to do this while also still making profits. So it's not really a risk, it's just mistreatment.

> Employees risk a lot more than most stock holders by working for a company.

Uhm no?

You apparently never been a business owner. Employees get their wages, and even can legally enforce them. If a business go down, owners eat the losses and envy their employees.

I've been on both sides. Being a business owner is much riskier.

But wages are often also only a fraction of what they should be, especially for those working outside of the tech industry. But I suppose that's a separate issue.

Is the risk high enough to justify the ever-growing disparity between owners/C levels/investors/etc and the employees that get the work done?

What with all of the bail-outs through history, running an especially large company seems pretty much riskless. And hell, if you look at the history of technology, say games and game consoles (because I like retro games) the number of times a hugely successful product/project that netted 100s of millions of dollars was "not allowed" by the CEO etc but was hidden until it was too late (see Xbox etc) is super high. In addition to the number of decisions made by higher ups where the business swallowed a loss (particularly easy in larger businesses) is also high.

Imagine if Bill Gates and Steve Balmer hadn't been convinced/swayed to make the Xbox. How much profit has MS made from that? A fucking shitload, and have the guys that pushed it, or for that matter anybody in a similar situation (of which there are many) ever seen any of that success? No.

And we can't say "well the CEXs have the final say because they take on all the risk" they do, technically, but in reality when C levels screw up oftentimes it's just taken as a loss and things move on.

If you only take into account small businesses with single owners you are entirely correct. Mostly because large corporations are squeezing them to death with various forms of rent seeking. However, if your corporation is in a position to casually misplace half a million dollars, I don't think you are in that category.

> On the other hand, employees aren't liable for the company's losses and debts, so it works out in the end.

How can anyone seriously type this? If you fuck up bigly enough, you will 100%—without fault—get sacked. Again, not even talking about long tails (bad economic conditions, layoffs, etc.).

This is under totally normal situations: if you lose the company money, you will be fired. As a bonus, you also lose unvested options or equity. These kinds of posts are exactly why engineers have garbage bonuses compared to finance even though they probably generate an order of magnitude more value.

> If you fuck up bigly enough, you will 100%—without fault—get sacked

which is to be expected - making a big mistake might not be something that can be forgiven and overlooked (depending on the magnitude of the mistake).

But you will not lose capital as an employee, since you did not put in capital to lose. Your time would still have been paid, up to the day you are fired.

Therefore, you obviously have no incentive to take on a risk that can result in a mistake (but which the reward you take no part in). You just do your assigned job, and whether it saves the company money or not, as long as you can cover your ass, you're golden.

Unless the company incentivize you to save money - for example, via a bonus through hitting a target or achieving some goal that was set.

> But you will not lose capital as an employee, since you did not put in capital to lose.

The conversation is a lot more complicated because there's an opportunity cost, you lose time (your time is finite, company time is infinite), you lose reputation, and so on. Besides, your argument is a bit weak as it's not like hedge fund managers put up the cash themselves, either.

My point is only that value-generators should be rewarded as such, and it's a bit weird that engineers are totally cool with not getting a piece of the pie.

> My point is only that value-generators should be rewarded as such, and it's a bit weird that engineers are totally cool with not getting a piece of the pie.

Software engineers are some of the best-paid labor in the world with great benefits and workplace conditions. They often receive equity as a compensation, even when the salary is still vastly above many other lines of work. They are absolutely getting a piece of the pie, and in much greater proportions than almost any other economic activity.

You may be discounting the value of capital, management, sales, and other roles in a successful software-related business.

The remuneration that labor and employees receive is never going to be in line with the value that they generate, precisely because the former group doesn't take any risk. They don't invest any personal capital and they aren't liable for anything. They can walk away any time, sometimes voluntarily, sometimes not. In return, they work fixed hours and get paid on a routine basis. The owners receive only what remains above and beyond all that, which could be great profits, just breaking even, or even losses.

> there's an opportunity cost, you lose time (your time is finite, company time is infinite)

Everyone everywhere loses time, because time passes whether or not you choose to do anything with it. Employees aren't unique among economic entities that they face opportunity costs.

> it's not like hedge fund managers put up the cash themselves, either

This is actually a good example to dive into. Hedge funds are typically paid "2/20", meaning 2% of assets under management every year whether or not there are any gains, and 20% of any gains above some benchmark. It's similar to, say, a commission-based sales role that gets paid a certain fixed salary and a percentage of sales they make. Whether or not 2/20 is "fair" is solely up to those who buy their services, since there is a competitive market of providers of fund management (the "employee") and providers of capital (the "employer").

And in some situations, the "employers" do in fact lose a lot of money, while the "employees" walk away; the limited partners of Melvin Capital, for example, lost many billions of dollars, all while Melvin Capital itself continued to charge the 2% management fee.

And within hedge funds itself, there are again employees who receive a stable salary and maybe some performance-related bonuses on top of that, versus the principals and owners who have personal capital invested. When LTCM blew up, for example, it's estimated that its owners lost $1.9B[0].

[0]: https://en.wikipedia.org/wiki/Long-Term_Capital_Management

> The remuneration that labor and employees receive is never going to be in line with the value that they generate, precisely because the former group doesn't take any risk. They don't invest any personal capital and they aren't liable for anything. They can walk away any time, sometimes voluntarily, sometimes not. In return, they work fixed hours and get paid on a routine basis. The owners receive only what remains above and beyond all that, which could be great profits, just breaking even, or even losses.

The people at the top get an even better deal. They get given stock options, so they get the upside but not the downside. They can also walk away, but they'll get a big payout if they walk away involuntarily. They work fewer hours whether you're counting butt-in-seat time or making-efforts-about-work time (some people, bizarrely, compare the CEO's making-efforts-about-work time to the employees' butt-in-seat time and conclude that the CEO "works more").

> And in some situations, the "employers" do in fact lose a lot of money, while the "employees" walk away; the limited partners of Melvin Capital, for example, lost many billions of dollars, all while Melvin Capital itself continued to charge the 2% management fee.

You're flipping the categories. Being the "investor" can be a bad position, sure. Being the manager, the decision-maker, is where you can't lose. Concluding that that somehow makes employees better off than owners is ass-backwards.

> But you will not lose capital as an employee

It depends on how much you mess up. Mess up large enough as an employee and you can end up sued by your former employer. Losing a lawsuit is losing capital. A probably not comprehensive list of reasons an employer can sue an employee, not all of which are because of negligence or malfeasance: https://www.mylawteam.com/employment/can-an-employer-sue-an-...

Depending on the state you can also have your pay docked (if that's not a capital loss, at least for transportation costs, then I don't know what is): https://www.avvo.com/legal-library/employment-law/paycheck-d...

You've completely missed the point. Businesses can lose money for all sorts of reasons. Owners have to eat the losses while keeping on paying salaries.

No they don't, they can instead fold the company and sell it off in parts.

That's absolutely the last resort and worst thing to happen for them.

Sure. They could instead split the company and dump all of the poorly performing assets and debt into the split off company and on the bond holders.

https://en.wikipedia.org/wiki/Texas_two-step_bankruptcy

https://www.businessinsider.com/corizon-health-bankruptcy-ye...

> If successful, Corizon's Two-Step would avoid a much wider range of liabilities than previous companies who've used it — not just injury lawsuits, like J&J, but the routine debts to vendors that companies rack up every day. If the company succeeds, it provides a "roadmap for eliminating virtually any unsecured liability owed by any corporate entity, regardless of whether that entity is solvent," Ian Cross, a Michigan civil-rights attorney who represents multiple prisoners who have sued Corizon, wrote in a procedural objection in April.

or do a leveraged buyout in which: https://www.investopedia.com/articles/markets/111015/10-most...

> The goal of leveraged buyouts is to make a large acquisition without committing much capital investment.

Sorry - the owners would do a leveraged buyout of what?

Their own company, using a handful of additional investors.

Or they'd sell to other investors, who are using a leveraged buyout, in order to get a better selling price than they would have otherwise. Meanwhile the new investors would take enough in income to cover the amount they put down, plus some, and then the company would eventually fail because it had too much debt.

Who is liable for the company's losses and debts, I wonder?

Sounds like you should move into sales if you want a % paid on value you help a company.

We're all in sales

It just feels strange to save a company a huge sum of money (by my own initiative) and not see a penny.

Flip side, are you ok if your company docked your pay if they found you wasteful?

E.g. why didn't you turn off that temp m3.xlarge instance. $X gets docked from your pay.

In some way, I made Amazon get a lot of profit by choosing to purchase on their platform, and saw nothing from it, though I directly contributed to their profit and could deserve a %.

I once saved a company $20k in infra costs and saw nothing of it.

In fact the team was pretty upset that they'd budgeted that money for infra already and it'd have been better spent instead of waiting till next year to re-budget it.

I once talked AWS into a 5-digit refund for something that was our team's mistake.

It wasn't necessary though because we had a committed spend target to reach and we just had to figure out how to legitimately spend the money somewhere else. :(

My company paid a consultant £25K to increase efficiency. He recovered about 5 minutes a job.

I wrote a tool that saved about two hours a case, in total this saved about £500k. I got a free case off beer.

You are lucky, I got nothing for heavily suggesting a ~1M/year saving.

The only "personal" reward I get from that is: whenever I feel guilty for not having done much in a given day, I remind myself that by this action alone, I've saved my company several times what I would ever cost them.

Helps with self-esteem, but I don't think my company see it that way.

Are you a consultant now?

That's the difference between perm/consultant I guess.

Perm is more "we pay you so fix this", consultant is more the reverse "we need this fixed so we'll pay you".

I always find that dynamic hilarious because in a general sense permanent employees have more value than contractors/consultants as perms usually have a much longer tenure at a company (years and years, vs 6 months to 1 year).

However one thing I noticed after moving to the UK is that the culture is completely different here - everywhere I've worked here there are contractors who stay for years and years like a perm would. It's not necessarily a bad thing, but it does make me reconsider why anybody would ever be a permanent employee beyond a bit more job security (ie long term contractors definitely have to trust that they'll be renewed, even if they usually are it's always possible for the business to decide otherwise).

I saved/earned Amazon $25M/yr also back in 2012. Because the project that was supposed to compute price matching/most-favored-nation status for vendors was never actually implemented. Despite it being in the standard contract for vendors for years, no one ever noticed that we never adjusted prices based on it. My own initiative noticing the problem, my own design and implementation, as a L3(or whatever the fresh grad role level is), and I got zip for it. Big part of my reason for leaving the company. I didn't expect people to fall to their knees and worship me, but it seemed like a project that should be a big part of a promotion, but I was passed over multiple times.

In my company we worked with a platform launched on local machines that had an admin console where you could execute java code. Pair that with almost everyone not turning on the firewall and all engineers connected to the same wifi network, anyone could do whatever they wanted.

I showed a demo how easy it is to read private ssh keys to the head of infrastructure, and after some months people could connect to network only using custom credentials (ldap) which was good, but also asked us to install "spyware" that among other things checked the firewall. I never installed the "spyware" but nobody pushed me. I didn't think I somehow prevented a disaster or did some heroic deed because everyone in the company was professional and nobody would exploit this. But of course I didn't tell about this to anyone except the infra because such information should not be disclosed until is fixed. And once is fixed why disclose it?

I really miss the Mac checkbox to enable the firewall. On linux I use nftables which is really powerful, but with so many possibilities it is easy to miss something during configuration.

I observed a lot of senior engineers don't have sufficient network knowledge. A lot of people on linux don't use the firewall which is really bad if you work on shared wifi.

Also when running docker images, if you map a port when using docker run (ex. docker run -p 80:80), docker will automatically add firewall rules and bypass the enabled firewall, exposing that port publicly.

Tried to save my company 100k per month by cutting down idle compute, clearing out petabytes of unused buckets, and using on-demand compute in testing envs. This resuted in me getting pummelled by directors asking questions, staff engineers saying random what-ifs, until I eventually gave up.

I used to think startups would be one of the few places that actually gave a shit about being lean and efficient - but turns out that's only true if they're bootstrapped.

I did something similar.

Success and super efficiency was rewarded with additional work, including hinting I should help other departments (apparently a joke).

I received no extra remuneration despite asking, yet my company continues to hire new staff weekly.

I've learnt my lesson, just like this author.

Why are corporations so allergic to competence?

Competent people in corporation could show how uncompetent the rest of corporation is, so they are found and eliminated before they can do too much damage to management.

That last line isn't a joke.

I've been on the receiving end of middle management because I've been able to fix things in the business that have always been broken but never got fixed until I worked on them.

Management will claim it's their work, will give you just lip service, will not use your name higher up the hierarchy and will actively down play their own mistakes whilst blaming the rest of the department or developers (e.g. like building a project with wrong requirements they actually provided, then missing the deadlines because stakeholders demand changes).

Where that has happened I quit and then they just go back to the status quo.

Hominids in suits

They are not eliminated, they are assimilated.

A company is like a chain, as strong as its weakest link. There is not much gain to have some links much stronger than the others.

The famous "I've learnt my lesson" which means you really reached the same low level of incompetence as everyone but secretly thinking you are more competent than all the others.

The funny thing is that among all those incompetent peoples / idiots mentionned there are probably smart people who just learnt their lesson years/months and adjusted to the weakest links in the chain.

They are "dealt with". One way or another, competent people are stopped from being a problem for management, either by being eliminated (fired), or tempered down (not assimilated, more like "converted") so they are no longer competent, but that's a small difference for me.

This is called the Tall Poppy Syndrome.

Possibly hot take; everyone wants to be a manager because managers make more, but you can't have a bunch of managers unless you hire a bunch of people, and hiring a bunch of people that are all competent is hard.

Is it possible that the corporate veil is essentially a cloak of invisibility for scam artists?

I think it's just the result of assembling a large number of humans. Beyond some size, relationships get replaced with internal politics and aggregated individual shortcomings become organizational pathology.

Corporations are not allergic to competence; they're allergic to bomb-throwing. Except in times of tremendous strife or revolution, incremental change is the best that can reasonably achieved in large organizations. The definition of competence in a large organization includes understanding the concept of Chesterton's Fence, and knowing what the Overton window is.

Why do seemingly competent people like TFA’s author continue to work at such a broken company, is what I wonder more.

I'm not sure what TFA means (the first article), I appreciate being called competent, though the truth is that I'm just less blundering than some other people.

I'm here because there's some path-dependence in careers. I started at a mediocre company due to not having a permanent work visa, and have been clawing my way up. I should write something else on this, but I've also realized clawing my way up was unnecessary - it turns out that while I've had some skill atrophy from working at these places, good engineers recognize someone that isn't going to cause a spreadsheet dumpster fire, so I should have just jumped to one of the top rungs on the ladder years ago.

>I'm not sure what TFA means

It comes from RTFM, but instead of the Manual it's the Article.

"My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight."

I love this so much

This is the first time that I saw this blog and I've now read a bunch of those posts. There is so much gold there, it's incredible. So much is going wrong in all those companies, that it is just frustrating to be working in those.

A colleague of mine and me got the opportunity a while back to basically work as wanted to in one side project. So we just worked truly agile, without any of the "Scrum" bs around it. Everyone involved was blown away how successful it was in terms of implementation. Two companies tried it before in the span of a year, and they didn't get it working at all. We worked just some evenings and weekends outside of our full-time job over the span of six weeks, and we got 90% done, with everything essential included to be able to start working with it. We then implemented everything else the customer wanted within the next weeks by following the same principles. There were no problems at all (apart from the customer basically almost running out of money because they spent most of on the previous companies which couldn't deliver).

At my day job, the managers usually argue against anything we want to do, question every decision from our side, want to have regular big meetings with 16+ people to talk about everything at length until no one wants to do it anymore, and then micro-manage everyone to death. We tried to at least keep these guys out of the daily by pointing to the Scrum guide where it says that only people working on tickets should participate, so one of these guys just created a bs task for himself, then talked every day about it for a lengthy amount of time without providing any real value at all.

I never got those silly shows before, which play in offices where everyone just acts silly and they never really work. But now I know why they are so popular. They just provide comic relief for people who really have to work in those environments and it is really as bad as in the television shows, if not worse.

It costs a million dollars to run a service that essentially “uploads a 2KB CSV to a database” on Snowflake?

And you can cut this in half by changing some defaults in instance lifetime?

I’m starting to understand why Snowflake’s market cap is something like $50B. This sounds like a nice money-printing business if you can convince enterprises to use it.

I should clarify, the cost was coming from computers that engineers used to write queries against our database - the 2KB CSV was just how we managed their permissions, and the cost of that was negligible.

The immense cost was coming from someone writing a query that translates to "I need one row of data" and then we get billed like $10-20 in idling compute. With multiple computers and several full-time SQL modellers, it adds up very, very quickly.

As the guy that just has to keep Snowflake running smoothly and isn't paying for it, it's a really nice product. I would still prefer something else on principle because it isn't open source, but eh, I guess it reduces my stress at work.

I know of an intern (with a master mentor) who did a one char change and saved our company many millions - the change was to update the threshold of data size of when to not compress small messages - this threshold was updated after many many years - old threshold was based on old compute and network costs.

> I'm not sure what the original estimate was, but I think it was intended to cost something like 200K for a year of operations, but we were now close to a million dollars.

> ...

> I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.

Wow, so they're still over budget by 300k dollars. This is a funny story, but the company sounds incompetent.

> This is a funny story, but the company sounds incompetent.

I've seen "typical" spelled this way several times on Hacker News. Is it a British thing or something like that?

lol I know HN isn't the most 'jokey' crowd, but I thought this was pretty funny.

Yeah, I thought it was funny too, and upvoted it, but I guess more people disliked it than liked it.

The author seems to agree with you, they are still wildly incompetent with their response to this.

Great post on why LLMs will have Management Consultants looking for their future. Average skill is highly profitable. One skilled senior and 20 juniors at an affordable rate is always most profitable.

Developers - Always do this as bonus paid by contingency of saved money and a signed off scope if it works out, especially if you’re d looking into it on your own time.

Specifically if you like you can get signatures from everyone on the hierarchy on how much money or time this will save, cost or make them.

Do it enough times and the right kind of CEp/President will tell you to stop bringing the business case to them for approval and just do them if they make sense and you have your backup.

You will enter a side door, bypassing most politics and c-levels (and maybe triggering some new), reserved for people who say let me see what’s possible instead of coming back with reasons why it can’t be done.

From there, paint a picture of what if you built a team of only doers, minus talkers across the organization.

Tech should never report into Finance, the group that can’t even tame spreadsheets.

Yes, business owners pay.

I'm sad to read that.

I can only testify that this is not my experience at all, so it might be that you're in the wrong place.

In every company I've joined, I've found issues, often large ones, managed to get assigned to them, and had fun fixing them (or at least the satisfaction of seeing them fixed). Then moved on to the next issue.

I haven't always received much recognition from the powers that be (although me and a few colleagues have once been wined and dined at one of the most expensive restaurants in the city for having saved the company 1M$+ with a few days of work each, that was nice), but I've gained respect from fellow engineers and from myself, which was nice, plus lots of things to tell during job interviews.

So, may I suggest getting out while you can and finding a better place to work?

I saved my company $50,000/year. A previous consultant (that they were still paying monthly) had written an application that was the backbone of the entire ordering system. If it went down, no work could be done and they needed to call this guy and pay him $150/hour.

It was a .net app that I reverse-engineered that was basically calling 10 or so encrypted stored procedures in SQL server. Because the older versions of SQL don't actually use encryption, but a type of encoding, I was able to decode them all and replace the entire app with a script that I wrote.

The whole process with testing took about a month (they thought it would take me a year).

The consultant that made it wasn't very happy when he was told they would no longer be needing his services.

Pretty sure that behaviour is as illegal as taking company code with you when you leave, inserting kill codes into software you work on, etc. Being paid to write software for a company and intentionally gimping it in that way probably also invalidates any liability insurance that a contractor may have as well.

The amount of time and money wasted on incompetent people who shouldn't be running or working on things is outstanding.

Fire the majority of people leaving the A players, unmanaged, and the world will begin to work near-flawlessly in a few weeks to months.

But this line of thinking hurts feelings which are (presumably) paramount to actually solving the problems people set out to solve.

Huh, I wonder which of the "big-4" advised them on their practices?

Am willing to bet it was 'KTMJ'... (Batman reference, but... strangely similar to one of the "big-4")

You actually got it. If it isn't a bother, I'd actually like to know how you guessed this, as I thought all the Big 4 were roughly the same.

I think we all want to know!

No comment... whistles and walks away...

This person is a good writer, I'm looking forward to seeing what else they've written.

It is really what you want out of a hacker-stuck-in-corporate story.

There was a tale maybe 10+ years ago about someone who automated their job with a script or Excel sheet or macro and didn't tell anyone about it. Having a hard time tracking it down again, anyone remember what that was?

Google reveals two things, followed by a bunch of shitty blogspam:

- Last year, there was a post on Reddit, with an HN discussion as well: https://news.ycombinator.com/item?id=29994776

- In 2017, there was a question about whether someone should tell their employer that they've automated their job: https://workplace.stackexchange.com/questions/93696/is-it-un...

I’ve found LLMs to be good for those sorts of vague “remember that thing like this?” searches. Bing (with GPT) or even Bard with potentially better search integration are both worth a try.

Now extrapolate this out and realize that 25-50% of our global tech economy is basically waste.

Too bad Scott Adams self-immolated.

This is Dilbertland, at its finest.

I had a friend in a similar situation, he explained the issue and how he would fix it.

I was baffled and urged him to be careful and assume he was wrong that it had to be wrong.

Anyways, he is still fixing it, getting some people to validate it.

I would think he will save at least hundreds of thousands a year.

But seriously, why a company that spends millions of dollars in an area would not hire an expert to try to save some money in it is beyond me.

Because "we don't have the budget".

That "we" may be too low on the org chart, and the qualified manager may not be aware of the problem.

Why would a manager of a multi-million dollar department reduce the department to a single-million dollar department? That would reduce the prestige of their own job. I've seen managers who were overstaffed by double work hard to double their staff again. And the only staff that actually accomplished anything were punished and culled.

It doesn't make sense until you realize that they operate in an environment where there is no competition that can disrupt their business model. If their costs go up, their rates go up. There is never an incentive for costs to go down.

Because it's monopoly money or something man, I don't know.

This post hit right something I dissected in my last role.

In short, I was revamping/enhancing a property insurance policy pricing system as the lead from the business side (i.e. portfolio manager/actuary).

However, much of the development team was truly incapable of writing software in any useful way, so I inevitably dipped into C# and SQL to help diagnose issues as we (as a team) worked on the enhancements.

My business coworker and I found something that remains funny to this day: code littered with references or attempts to use the .NET TPL library to make the platform do more things "in parallel," but without actual knowledge of using such a library in practice.

Nothing worked in parallel! There were blockers all over the place! As someone else said, TPL might well have been a //TPL TO DO.

We suspected that TPL was there as one of those "look what I can do" intrusions that received praise from more senior devs/mgmt. even if it never actually did what it was supposed to do.

> They hired some incredibly talented people to make this happen, and then like five times as many idiots.

I can relate to this.

At a small company, I found a bug in the sales system that was giving away $10k per month by miscalculating customer's tax. CEO of the company essentially pretended that he couldn't hear anything when he was told.

The unsaid part is that this kind of situation is everywhere. It only really hits home when (IMO) when you’ve had a couple of decades in the industry. We have totally squandered the abundance of resources in many sites/situations/use-cases. Whether it be IOPS or threads or memory or network bandwidth and latency. Yes premature optimisation is the root of all evil, but the other side of that coin is pretty ugly looking too. Not even because of monetary waste in many cases, but because of unnecessary complexity and fragility. For what? A slightly higher level of abstraction? A bit more interoperability? A marginal gain in some other metric?

I could fully relate with this post. I personally share a similar mindset: someone who can take criticism with pleasure, who has sorted out thoughts about how the industry operates, who has not subscribed to the belief that people in leadership positions are world apart from us, and who can openly express that wrong is wrong.

I think many of the people who disagree have either been serving in non-engineering roles since past couple of years, or have bought in the pitch of rosy world created by managers and so-called leaders.

TLDR: They changed the idle time settings for running queries in their Snowflake database. Originally, they were set to idle for 10 minutes after every query, but most queries only took about 2 seconds to run.

I know nothing of Snowflake pricing, but how big is their infrastructure that leaving it on for an extended amount of time could see that much savings?

To put it another way, to rack up $500k spend for 30 days of constant provisioned time:

$500k/(30*24) = $694/hour

Presumably, with the 10 minute blocks, there was idle time where the spend was zero, so the instantaneous amount would be higher.

$58 per hour over a year on average. Sounds like a big big warehouse. For scalability.

This is approximately right. They also almost have one warehouse per two data engineers. I've been told there's a reason and usage analysis has been done by actually smart people, but it just sounds incredibly suspicious to me.

He said a few thousand queries per week, which is maybe 3000, or 156k/year. That's 1.56 million cluster minutes, or $500k/1.56M = $.32/minute. A single snowflake node is $.05/minute, so maybe it's an 8-node cluster. They could shrink the node count down to 4 or 2 to save more money.

How long does a snowflake machine take to spin up? Why have an idle time at all?

I have never used snowflake so I'm not sure how this works.

Their virtual warehouses usually take only a few seconds to spin up, but can take longer (especially for large warehouses).

If you keep the warehouse running, there's a fair amount of caching going on behind the scenes that can make your queries run faster on the warm warehouse (how much faster depends on the query) . This article shows one example - a cold query ran in 20 seconds, after caching it ran in 1.2 seconds on a warm warehouse:

https://community.snowflake.com/s/article/Caching-in-the-Sno...

So you don't always want to set the idle shutoff time to the minimum.

There are probably other use cases, but I only think of data warehouses for supporting batch/analytical workloads where latency is not a problem because the entire job takes minutes/hours to complete. In which case cold start does not significant.

Analysts and other data consumers don't want to wait 30 seconds for cache warming every time they refresh their dashboard.

Also, since you're paying by the second, it can be cheaper to keep a warehouse running longer to avoid having to re-warm the cache with every query.

Yeah. I went through a rigorous exercise with sales last year to try to get a good estimate on what it would cost to run the database for a year for our application. They said it would be around 150k.

I threw all of their configuration advice out the window and run all extra small at 1 minute idle. No issues for my use case and we paid 17k for the year.

I have to expect that snow will change their pricing model soon.

Thank you for writing a hilarious and painfully true slice of office life.

what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems

Well duh...of course you'd want that. But it is not so easy is it? I'd also instantly fire half my staff and hire hard working geniuses instead. But that just isn't an option, unfortunately.

Like others said, I absolutely feel this post. I don’t think I can say any other post I’ve ever read feels this relatable. Every one of his posts is both accurate and a depressing assessment of the overwhelming incompetence in this industry.

> I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.

I know he doesn’t work the same place I did but damn, it sure feels like it.

My goal is to make my company the opposite of this one.

Sounds like something out of a Gene Kim book.

Definitely a pattern through the places I've worked. The ones that are smart, have good intentions and go a little bit rogue tend to make the biggest impact.

Executives are too concerned with risk, and honestly, it playing out this way suits them better: if someone goes rogue and it works, they can claim the win, if it doesn't, they can blame and reprimand.

Like staring at a mirror, this post made me cringe with how relatable it was. Guess I have a whole blog to read now.

In my opinion this red tape is why startups are more successful than large organisations… At least in the beginning

Apart from the apparent topic: If the architecture of the software is a mess like he describes I would bail or ask to make significant changes (and threaten to leave if I wasn't allowed to). I would definitely not just try to optimize using settings.

Line manager here.

When I find someone very talented, I go out of my way to promote them because good engineers who want to stick around and not jump when the times are good is hard.

The sad reality is that I’m limited to how much I can bump their pay, give them time off, etc.

Indeed, under a pay-as-you-go model, if there's a lack of precise control over the warehouse, such as a 10-minute suspension, it could lead to significant waste. This is because most queries might only take a few seconds, and the rest of the time is wasted. If you find Snowflake expensive, consider Databend. It's an open-source, cost-efficient alternative to Snowflake, and it maintains a consistent product experience with Snowflake.

Open-source: https://github.com/datafuselabs/databend Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059 Cloud: https://app.databend.com/

I think the most charitable interpretation of this phenomenon is that bureaucracy is hell and the entire organization tends to get put on a leash pegged to the level of incompetence of whomever is in charge of #thing.

(Peter Principle writ large.)

> I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.

Only one thing left to do. Leave 'sleep(600)' calls all over the place and resign.

In a sane world this would have to be satire, but unfortunately I believe every single word of this

A writer cannot make up so much shit. You need a board of managers for that. Effing hell.

It's good to know there are worse places than mine. some things are reminiscent of what I've seen, but nothing so extreme as even half of what's described.

Whenever this happens, I really have to wonder about all of the people I call "good" on my team. Like surely someone gave a shit enough to know this is how it works, right? ...right?

Repost of https://news.ycombinator.com/item?id=38064086, should probably be merged

I'm on the tail end of my career (hopefully), so I can share some unsolicited advice for younger workers.

1. Remember who you work for. Yourself. Not your boss, not your grandpa who always wanted you to become a writer. You only have so many breaths before you get your ticket punched, so make them count.

2. Don't seek work as something you should love. Those should be your loved ones, and hobbies etc that stimulate you. Sure there's a small minority who get to do what they love, but over time love can turn to disgust.

3. When starting a job, figure out the organization's incentives. Some orgs want change, some will fight any change. Figure out what the org wants and your life will be much easier. My job is like the Maytag repairman, kind of waiting for stuff to break. I refer to it as being a digital janitor.

4. Nothing you build digitally will last. Hell, between linkrot, bitrot, and the heat death of the universe, nothing lasts. So don't expect what you've built to last, or to hold value. It'll all be re-factored/re-engineered/re-architected and become obsolete.

5. Large organizations are toxic and often sociopathic. So are small orgs. If you can find a way to start your own business, try to avoid becoming that way. Good luck, most businesses either fail or become that way. But at least you have more choice in the matter.

6. Find a mentor (sometimes referred to as a rabbi) who can provide guidance to you when times get tough. Don't choose a rabbi from your reporting chain. You will find yourself adopting a worldview that might not be aligned to your own interests (in other words, the bastards will manipulate you). This rabbi can be a friend, or just someone you vibe with. Be careful in choosing your rabbi, and take care of this relationship.

7. Illegitmi Non Carborundum

Great advice, especially on finding a mentor. I hadn't thought about the importance of finding someone outside your reporting chain.

Illegitmi Non Carborundum.

How do you maintain such mentor relationships? I found a very inspiring mentor at a place I interned at this summer, that I'm due to return to full time in coming months. But I'm not sure on how to continue this moving forward, as the previous was an intern program relationship, and I'm now full time.

I've found treating mentors the same as friends can work. Keep in contact via whatever means you used during your internship (email/chat/voice). It can be a balancing act where you don't want to be a leech, only taking and no giving. But it is a professional friendship. You have to like and trust your mentor to give you sound, unfiltered advice, without compromising their own integrity. Being a mentor can be a real challenge too!

> Large organizations are toxic and often sociopathic. So are small orgs.

It's important to remember this.

To give an example, most large (multi-billion dollars) companies I worked for where amazing with happy people, whereas the small startups where filled with people backstabbing each other to make sure to be at the top of the food chain IF the company were ever to make money one day.

The moral of the story is: anything can happen anywhere.

Also, you lost Snowflake half a million dollars.

$500K per year saved. Now add in some layoffs in next quarter and pump those financial reports.

Should have just let them burn dude

> I saved my company half a million dollars in about five minutes. This is more money than I've made for my employers over the course of my entire career because this industry is a sham. I clicked about five buttons.

I'm sorry, but this needs a privilege/gratitude check. You are guaranteed your salary, and you're welcome to take on the same level of risk your company is by starting your own. If you think it's so easy go ahead.

I don't understand your point. According to the article it was in fact that easy. The issue was due to hangups in bureaucracy.

This person is asking for a $30k raise to do the baseline of what's expected of them: their job. It's your job to know what needs to be done and assert that it needs to be done.

They didn't make $500k, but saved $500k from a mistake that the company themselves made. Why does that deserve a raise? I've seen people implement features confined to a single file that made over $1m/year but they didn't ask for a $30k raise.

I could see a one time bonus or stock gift and maybe a more modest raise next year if they stick around.

My overall point is this: If you want to make the big bucks, you can go make your own company, but be prepared to take on the mountain of risk that comes along with it. Otherwise, a tech salary with guaranteed income ain't so bad.

I accidentally cost my company half a million dollars, but it's not really my fault HR misplaced a comma.

one of my employees saved an unnamed tech company 250M in regulatory fines. They were flagged as top performer and subsequently laid off in the first wave of the tech layoffs … probably because stock compensation clawback made it very juicy.

> I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.

Pain I feel very acutely. As an employee, I saved a FAANG company billions of dollars in potential GDPR fines after they carelessly declared 'mission accomplished', weeks before the deadline, and never even got a 'thank you', much less a raise.

Mind your own business next time, we need the money here over the pond :P

you mean someone costed your company half a million dollars for x amount of time.

Great! That's worth (let's see... carry the three...) a 1% raise!

"We have always done it this way" comes to mind.

It’s always the cloud costing ridiculous amounts of money.

Reason number 543895 why I like small company life:

When you find shit like this, people will go "my bad" at the worst, and you all are happy the company has more money. Small teams and small firms have their downside, but honesty and transparency, especially when it comes to cost savings... tend not to be one of them.

I think the author should take a vacation.

Yep, can definitely relate to this

No good deed goes unpunished.

good management would fire half the org

if you want to learn more about consulting firms watch John Oliver https://www.theguardian.com/tv-and-radio/2023/oct/23/john-ol...

> The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up by MongoDB, then MongoDB records are passed by another Lambda into S3, the S3 files are pulled into Snowflake via Snowpipe, the new Snowflake data is pivoted by a Javascript stored procedure into a relational format... and that's how you edit someone's database access. That whole process is to upload like a 2KB CSV to a database that has people's database roles in it.

Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.

I’m genuinely curious what a unit test for something would look like.

// TODO

LOL. What did this comment (no pun!) do for your HN karma points?

Checksum on the resulting csv with a parallel implementation of the whole pipeline ;)

I don’t work with large databases so pardon my ignorance. Is there typically a “unit test” bucket you run it on or do you just put in test entries on a production bucket?

We tend to only test what is being tested. So, most DB calls are mocked in our unit tests. For stored procs or other tests that need to be run on a DB, we use a test DB that is setup to mirror production.

I'd bet there are a 100 different answers to your question though. This is the way we handle it.

Normally you'd fire up a separate environment, mock the process and see if it produces the expected results. By the time you put 'test entries in a production bucket' there are so many lines crossed that it likely won't end well even if the tests do pass.

That is probably quite straight forward and of course they have 100% coverage.

That would be an integration test, not a unit test.

We usually use very complicated UIPath flows to “test” these things.

If that doesn’t exist yet (it usually doesn’t), we test manually, but only core workflows.

Of course the answers on stack overflow are partly a result of resume-driven answerers.

Everything I look at these days looks like this. And most of the time it doesn't even solve the initial problem statement but everyone is too naive to even realise that.

The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it.

The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%

I know of an engineer who built a work queue by having a chain from an app to kafka to a processor to Kafka to a database writer.

Literally instead of a table.

This stuff is everywhere. Microservices made it worse and half legitimized it.

This must be common or we worked at the same company, seen this exact pattern.

It went like:

App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka

The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.

I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.

> Database fear is overwhelmingly powerful in a lot of people it seems.

Database are still fairly poorly documented when it comes to administrative work.

There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database.

I mean, from novice admin to DBA-level capabilities.

I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.

This does not honestly seem true for the main sql databases. For everything else, yes, but if people were actually learning databases instead of hiding from them most of the uptake there wouldn’t exist anyway.

But, going the other way, I worked for over a decade on Goldman Sach's SecDB system. It's a quirky steampunk alternative future that branched from our light cone around 1995. There's a globally distributed eventually consistent NoSQL database tightly integrated with a data-flow gradually-typed scripting language (and a very 1990s feel 16 color IDE). I'm sure in the late 1990s/early 2000s (before globally distributed NoSQL was popular and before gradual/dynamic typing had a resurgence) it was more like discovered alien technology than steampunk alternative future. (Also, with source code being executed from globally distributed immutable database snapshots, deployment is much nicer than anything else I've used to date. After release testing, set a database key to point to the latest snapshot, and you're deployed.)

There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.)

It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper.

However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written.

Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.

Message queueing triggers a host of psychological needs. Synchronous jobs rarely need monitoring and management features, but move the same work to a queue, and everybody loses their minds.

I think I've seen enough complexity created by engineering teams given total autonomy, with hands-off leadership, that I'd prefer a much more constrained approach. There should still be autonomy, of course, but proposals for new tech, languages and paradigms should only be considered with due diligence.

The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.

Repeating myself...

Just two examples from my prior gig (fashion e-commerce).

#1 Our hottest dataset (db of current products) stored in DynamoDB. Core dependency for all our code. Easily fits in < 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed to persuade the team to start transition from DynamoDB to Redis.

#2 Tiny (vs micro) service that munged some URLs. Blocker for an important campaign. Prior team of 4 churned for a year, was no closer to delivery. Spring, ORMs, CI/CD pipelines, the works. I spent a week unraveling the requirements (repeated facepalm). A second week banging out a trivial nodejs thing. (My team preferred nodejs, which was their prerogative.) Really trivial. I felt so bad for the biz dev people who'd been dying to get this functionality for so long.

I am currently working with a US government system for downloading public scientific data. You select some data you want to download and add it to a shopping cart. Check out, and select 'create database'. This generates your own copy of an Oracle database, with your own credentials and hostname and db name. Connect to that and construct a query against a table that has some metadata about studies you're interested in. Using the identifiers from that table, join with a LIKE against another table for s3:// URLs. (There are no primary keys and the other table's column is not exactly the same; you need to use a LIKE. This is all documented.) Those s3 URLs point to a CSV which contains another identifier which you use to download manifests which contains links to a web page created on-the-fly which contains to the s3 files to download. By the time you've done all this, your access has likely expired and you must start over from scratch.

voted for the Kafkaesque of 2023

(I assume this is all a reflection of https://en.wikipedia.org/wiki/Conway%27s_law )

I'm going to take a deep breath, and move on to the other stories further down.

So far I'm feeling pretty good about what I've developed over the years.

You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it. These kinds of systems, as the author pointed out, are incredibly expensive and inefficient, but look readable on an architecture diagram.

> You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it.

Where?

IT departments, typically, though occasionally there are whole companies that work in "technology" where this type of work can be found.

I said the above as a jest, but seriously, simplification of complex stacks has been a good consulting gig.

Most large companies. There is a stark difference between the distinguished engineers and the tier below them in terms of asking people to stop doing things badly.

I've typically worked in SRE and platform engineering work and that's where I've gotten exposed to these kinds of Rube Goldberg machines. Make a short list of them when you find them and then use them as a hit list during "cost cutting". Most people don't want to touch these systems because they look big and expansive and generally "work". They're just very poorly optimized.

Dare I say, any time I see a function as a service my brain immediately drifts to inspecting the cost implications of said process.

As opposed to all those people that make similarly comfortable careers in middle and upper mgmt by identifying simple systems and complicating them beyond recognition?

It's a beautiful symbiosis.

These roles aren't opposed at all, they greatly benefit each other :-)

Bringing what used to be the privilege of upper management (wasting massive amounts of resources while getting paid handsomely) down to software developers.

It's that trickle-down effect people talked about, right?

Hah, yes. I will say that while I understand the general disdain here, as I grew more senior in my career I realized the world takes all types. There are "doers" who will rush to an end goal that's highly prioritized and then there's "optimizers" who come fix that mess up into a durable, cost-effective system. Some people are gifted enough in knowledge and have the right business priority to do both at the same time, but usually they're required at different times.

Anecdotally, optimization tasks (in this brain) are multitudes easier than innovation tasks. I spend a lot of time thinking about how to do things differently whereas optimization utilizes many lessons I've learned over and over again with well-trodden patterns. That's to say, I'm grateful for the doers :)

> learning to identify this kind of work/system and proposing ways to simplify it

"I dont think you are fitting in here at MegaCorp"

"You are being negative, the system is great, we don't like that kind of attitude here".

> "You are being negative, the system is great, we don't like that kind of attitude here".

I want to see more lines of code, not less.

Hmm looking at your git statistics it appears you have only pushed 60 commits this month with 12,000 lines of code changed - while Jimmy over here has pushed 200 commits this month with 200 lines of code changed.

If you do not improve next month we are going to have to let you go, we just cant carry a 0.3x employee such as yourself.

"We need this to scale to hundreds of millions of users across many regions"

"But we have no users at all right now"

"But we might have hundreds of millions of users in future"

I am the author and this has caused me psychic damage.

You should take a vacation.

Tried it. Nope. You can get people to acknowledge it but because it's not a fun project or doesn't involve an upsell you can bill the clients for, it'll go in a product backlog for a decade or two.

I don't care any more. I'm just there to tell people what's shit and then laugh when it explodes in their face.

That is what skunkworks are for. You just deliver on time and you are fine.

What is this deliver thing? I haven't done anything productive for years.

The easy part is choosing a better end-state; anyone can do that, and for any of these Rube Goldberg machines at a large-ish company, several people likely have.

What makes someone a staff+ is finding a path to iteratively evolving towards that end-state without breaking anything along the way and while having progress to show off at each step.

Oh yes but part of that is knowing that you need to get others on board. Things get messy when no one knows the end goals with changes.

I see stuff like this every day. It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves.

The worst kind of "DevOps engineer" that doesn't really understand development, operations, or engineering.

But hey, they can run some docker and git commands and piddle around an AWS GUI, which means they are highly technical.

You don't give enough credit to organization chart and project driven engineering.

When developing anything:

1) you don't get to touch anyone else's code. And another department's code? Something another manager's team manages ... that amounts to treason. Never for any reason. MAYBE if they've totally abandoned it and you absolutely need it (but only during unpaid overtime)

2) you don't get to spend ANY time on anything outside of the current project or JIRA ticket. Any time at all. So really, NOT optimizing anything is faster and cheaper. Just look at all the spreadsheets made!

>piddle around in an AWS GUI

I’ve had enough calls with the “Senior/Technical Lead Azure Cloud Engineers” telling them exactly what they need to do that me and them really really don’t get along.

I don’t do any of that shit and even I can muddy my way through it, but these people cannot. The real kicker of it is how much these people make.

And you know how there are those people who, every time you need to work with them, they answer a teams call and then “need to get to my computer, give me 5” and their status is perpetually set to away? I don’t want to RTO at all, but dealing with this team almost makes me think I’m wrong about that.

>It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves.

That's on the industry for not training and gating well. It would be nice to have glue/plumber positions so expectations are not out of line too.

I don’t think most of the coders who end up in glue code positions are actually trainable.

Agree solidly on the gating aspect though.

The problem is that quality hires continue to be rare and at some level you are doing area-under-the-curve reasoning.

That reads like the KRAZAM Microservices sketch.

https://www.youtube.com/watch?v=y8OnoxKotPQ

I've used that video to explain to business people. It's watchable, and communicates important ideas of what a poopshow this can easily become, without having to talk about real partners/teams close to home as problems.

KRAZAM is a prophet and must be protected at all cost

At my workplace (the one in the post), whenever one of the good engineers asks about how something works and it's one of these spaghetti-balls, we chorus "It's the design of our backend, okay?"

And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort

was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ).

I promise I'll rewrite it in python at some point.

I think it's "iterative-StackOverflow-driven development" most of the time, and that actually causes the increased popularity of those resume keywords.

... And a partridge in a pear tree.

This sounds like absolute hell. This is everything I hate about modern software development.

The more experience I have, the more I start to think the Omnigres people are right about "just put literally everything into Postgres".

You have no idea how unbelievably annoying it is to work in a company that doesn't a well defined architecture. Every "buzz word" service should be easily justified.

This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.

I had recruiter pull that in my most recent job search. Has to stick "C#/..." in front of everything because they didn't understand that ASP.NET, WPF, WCF, WinForms and several other C#-specific tech had anything to do with .NET.

So, what tech service can I add to that bloated pipeline as a middle-man to get a fraction of a penny per transaction?

I feel like this might be case of data engineers.

They're not usually software engineers. They're tool users not tool makers.

So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.

Agreed. I've been "the backend engineer who works with the data engineers" for several years now and I've seen their general trend of re-inventing the wheel the hard way a number of times.

I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.

I stepped into a Data Engineering Lead role in 2019. Stepped out of it in 2021. My team was the first in the org to really approach data engineering and we were all software engineers. I'm told that the systems we built have largely been replaced by Rube Goldberg machines pieced together by the folks who came after us.

Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!

I feel like I've never seen anything even reminiscent of this bad in the twelve years I've worked as a software engineer. I really want to believe this pipeline as described is satire. Yet, somehow, it does not quite seem that way. This scares me. But also somehow explains why some companies contain so incredibly much more engineering staff than I can possibly explain looking at their output.

Resume Driven development - RDR

Never heard that before, but that’s so on point.

is this larping?

"this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing."

Came for the engineering, stayed for the blisteringly on-point observations of corporate life

At my previous $JOB I started calling this Promotion-Driven Development because it's everywhere. Take the easy problems that look good and you can finish quickly, get promoted, hand off the facade to team that has to actually solve the real problem, and repeat.

This is very much how Google is on the inside. And that’s not surprising. People respond to the incentives presented. If you spend years fixing hard problems that actually affect users, and don’t get promoted, you’ll almost certainly become jaded, and start doing things that are pointless but do get you promoted. Because that’s how you make more money. And that’s the actual reason to be working.

And yeah, it shows via ye olde Google graveyard. Honestly I do think it's just a hard problem not to have, because the world tends to work on visibility combined with innovation. You can't solve hard problems and make money if nobody uses your solution because they don't know it exists. But I struggle with the balance because yeah, I'm a bit jaded.

I think the crappy part is that it works most of the time.

The purpose of a system is what it does. All development is promotion driven development, different people can either choose to be clear eyed about this or not and, subsequently, good at it or not.

Your job is the venn diagram intersection of an organized system of rewards and punishments from both inside and outside of the firm intersected with your own goals and aspirations over time. Anything not in this intersection is not a job, it's something else we like to delude ourselves into thinking is a job to preserve our egos.

More specifically, your worry about how distorted incentives lead to poor quality software is not your job unless it is your job to worry about said incentives. People keep hoping to go work for one of the magically sane company where all the incentives are correct and everyone is doing the quote-unquote "right" thing at all times. Such a company does not exist because the very concept is incoherent, every system of possible incentives will come with a different set of tradeoffs and the art of operating a firm is to pick from amongst a bunch of shitty options for the least shitty one.

People who do not accept this will be perpetually unhappy and bitter about how others obtain undeserved success because they simply did the unsporting thing of playing the game they were asked to play. You can either be one of those people or you can receive the radical acceptance of what a job is.

I like to think I'm pretty good at this game, swung two promotions and raises by working every request from outside the team out of band to the detriment of my main flow of work -- "everyone loves you and your work is constantly noticed by $upper_management." Wow that's crazy, it's just nice to be appreciated I guess wink wink.

People who do not accept this will be perpetually unhappy

or they go start their own business to get away from this nonsense.

when i deal with clients i actually have to deliver something because i am not getting any promotions.

Client work is its own unique hell of misaligned incentives. You've just replaced one boss with a dozen bosses, all of which have the power to "promote" or fire you.

I'm not saying that any one individual can't find a job that is the unique perfect blend of incentives for them, indeed, the entire point of this framework is that alignment between your goals and the incentives of your job is the most powerful lever an employee can pull. But that incentive structure will necessarily make others working there deeply unhappy as its unique choice of tradeoffs is just shitty in a way invisible to you.

all of which have the power to "promote" or fire you

that's fair, because i also have the power to fire them.

loosing one client when i have a dozen others is not a big deal.

with so many bosses i can focus on the good ones and reject working with the bad ones. i can also unilaterally raise my fees, especially for bad clients.

Not all jobs are promotion driven, at all. My current dev job is for a small family biotech business, and the only path upwards from where I currently am would involve church bells and a flower girl.

But there are still things you need to avoid that would get you fired if your performance dropped and there are still things you can do that would improve or decrease your job prospects on the open market if the company ever goes under or is forced to let you go. I'm using "promotion" in a more abstract sense than literal promotions.

I do not like it. And I personally (mostly) fight against it. But, I believe that this person is correct.

My last place also called it promotion-driven-devevlopment.

It got to the point where some dude built a system that completely floundered but got him a promotion. He then re-wrote it so it sucked slightly less and got another promotion. Last I heard he was working on v3.

Never mind that someone with actual domain knowledge would have either not built the system (since it didn't _really_ need to exist) or would have built a much simpler/more reliable system to get the job done.

He did it like that because thats what he was asked to do, metrics-wise

His boss is happy with it because it looks good for him too

The company is happy because it allows them to sell milestones to themselves and or customers.

Customers are happy because the sales-people took the right person out to a fancy dinner.

I don't feel like I could fault anyone acting their part in the play.

Can't disagree. He's was doing what the incentives in place said he should be doing.

I would find it boring/tedious to keep rebuilding the same thing since I'm more of a solve the problem, operationalize the solution, then move on person.

Ah, but he "de-risked" the project by getting it out the door quickly, he started "delivering immediate value" despite the rough edges, and "has the technical acumen to optimize it further."

I got downleveled in a job offer recently after being interviewed about a project which I labored to make as simple and efficient as possible. They said after careful evaluation, the project wasn't technical enough to merit the title I wanted. Live and learn!

This has to be one of the most insightful comments I've ever read.

Sure. If one is at a company that is entirely run by incompetent sociopaths, then get with the program, right?

But not all companies are sociopath boondoggles. Some are companies where people are trying to do the right thing (make money, grow business), and management would prefer to be successful. In those environments, we still get "promotion-driven-development" because, sadly, most managers are not software engineers and can be successfully bullshitted. And it's not even always malicious: you take someone who is a great people person but a mediocre engineer (huh, that correlates a lot), and they really think that this pile of steaming shit idea is a good idea, and their genuine enthusiasm convinces management who also don't know better. This mediocre engineer really wants to do good but has no idea why their idea is fucking herpes - because they are not competent enough to understand why. Dunning Kruger time. And of course the competent engineers are all Autistic and come across as rude, disparaging assholes in neurotypical management's eyes, and everyone gets herpes except mediocre engineer who gets promoted (possibly into management).

In that situation you have to have "... a handful of good engineers and going totally rogue, we outperformed the entire department pretty effortlessly."

All of my major career jumps have involved "going rogue", and having the outcome being recognized. And one time I was basically fired for it, and the work buried by a psychopath, and the remaining team did it anyway and delivered the solution, averting a product-ending scaling cliff. YMMV.

But not all companies are like that, and in some places you can point out to management that the idea is bad, and management agrees. In my current job we had someone pitch a fabulous promotion opportunity that was not merely a total waste of time but also fundamentally missed (didn't even attempt to identify) the root cause of the problem it was trying to solve, and fortunately management agreed when we pointed it out. Yay!

So I don't think it's as hopeless as you make out, at least not everywhere. I'm having fun right now, and getting paid enough (it's never enough).

> And one time I was basically fired for it, and the work buried by a psychopath, and the remaining team did it anyway and delivered the solution, averting a product-ending scaling cliff. YMMV.

You're triggering my PTSD. Not to brag, but on paper, I was top performer in the dept at a previous employer. Was loyal, busted my ass, tried and many times did save the company money, but because I started posing a threat to my two sociopath managers (yes, two legitimate sociopaths) and spoke up against unethical (and likely illegal) activities of theirs, they made my life hell and eventually "laid" me off.

Getting fired was actually a blessing though, so fuck them. I'm not playing scummy games and larping as an asskisser for brownie points any more and have been much better off.

> People who do not accept this will be perpetually unhappy and bitter about how others obtain undeserved success

This is a good comment, however a giant issue is most companies will not say this out loud, and in fact say the opposite.

So while it's fine if you're savvy enough to see through this, it's not really fair to smirk at people who don't and who honestly take what leaders say at face value.

It's like a weird western version of Japanese office culture where you need to know the wordless rules, but it's not wordless and you need to ignore the BS said while feigning agreement.

Author here! I wrote on something very similar earlier a few weeks ago, down to a reference to Japanese office culture (though not the exact same part of it).

https://news.ycombinator.com/item?id=37806870

>> work for one of the magically sane company where all the incentives are correct and everyone is doing the quote-unquote "right" thing at all times.

Between the “perfect” company, and the shitshows described here, there is a lot A LOT of shades…

I used to work in a company which was far from perfect, but it was great to work there. Until the CEO changed…

I just search for a company where thing are just noy insane. I’ve been there, couple of times, and my current position is kind of ok…

I've seen this over and over with cloud projects. People absolutely get promoted/raises based on optimising/cutting cloud budget. So there's no incentive in doing things efficiently from the start. You're much better off to build a vastly (but maybe not obviously) inefficient first version and then go back later and show how you "saved" the company tons of money by turning off a bunch of unused services etc. It's harder to hide racks of powered off servers which is why this is so prevalent in the cloud.

Fascinating! I'm young enough to have almost entirely worked on cloud services, but that certainly explains the prevalence of this kind of behavior.

* Save 50p: cost-cutting genius

* Spend £5,000: prudent investment in the future

For many businesses the faster time to market is worth the additional starting costs. Doing it quickly and optimizing cost later has been an explicit desire for a lot of organizations I have been a part of.

Some of the best ladder climbers I've seen are amazing at pitching projects. They get the business case done, fully funded, themselves promoted to run it - and then immediately leverage the new title into a job somewhere else.

I've been on the receiving end of one of those projects and once you dive in without the person who pitched it, you see how hollow and nonsensical it really is. But by then its too late. The consultants who wrote it are all paid and the executives who attached their names to it as sponsors keep it limping along for years, draining everyone's will to live.

The other guy meanwhile is doing the same thing again at a bigger company having a great time.

Almost sounds like a piece from the BOFH

i loved the style of writing! what do you call it? Realism? Sarcasm? Idk

It had me stuck reading until the very end - usually that never happens!!

TLDR: Bureaucracy is everywhere.

Incredible.

Meanwhile big tech thinks the way to reduce costs is to wholesale fire 1/2 their company.

For what it's worth, OP also recommended that approach.

Yeah, but corporations rarely fire only the incompetent people. Quite often it's the best paid engineers who get fired first because that saves the most money. And when there's a voluntary get-out program, it's the unappreciated competent people who are the most eager to leave.

The two are not exclusive. Companies can run hundreds of programs many of which are not productive and actually hurt and frustrate engineering teams. Just in the same way government can hurt the environment by throwing up a lot of red tape on green initiatives, companies can hurt quality and productivity by spending thousands of hours on things like employee annual reviews.

Reply on news.ycombinator.com