One way a builder culture can fail
rachelbythebay.com
rachelbythebay.com
... is the crux of her point.
So, um, who decides what belongs in production?
I generally like her essays but this has a weird senseless negative tone to it.
Look... I get it. We've all come across bozos. A lot of these bozos aren't the derpy clowns we (hope? wish?) they would be. No - they're usually very confident, very aggressive, and very very goal oriented. Yes it's absolutely infuriating watching them somehow gain the confidence of the C-level folks, obtain a lot of resources, and burn everything to the ground all while claiming plausible deniability. Wishing that they'd get filtered out by the hiring process is <insert idiom of improbability>.
The real culprit is your director of engineering. That person is making terrible decisions but is so good at avoiding accountability that you place your blame on the bozo. Hot take: if your company keeps hiring bozos then your executives are doo-doo. Leave the company. Oh, you can't because you're making 300k base + 2 years away from 800k in RSUs? Congrats. You're part of the problem (it's not your fault).
The same reason why you won't leave your company is the same reason Mr. Bozo gets a crack at "innovating" at that same company.
People like Rachael (who obviously love this work at their very core) are doomed to end up a jaded curmudgeon unless they 1) start their own company 2) join a tiny company. Big tech is hell. Make your money and GTFO.
Product Owner, or just management?
On a side note: in "mature" markets a bricklayer nor an architect, nor constructor can really build anything they want without anyone looking.
If something Does Not Work Reliably - when it has no reasonable reason for not doing so - then it Does Not Belong In Production.
As for mature markets - useless and/or broken buildings happen all the time. Sometimes this is because a starchitect is good at making buildings that look incredible but bad at making buildings that don't have problems. (Calatrava, Frank Lloyd Wright, many more.)
Sometimes it's because the project is extractive and the goal is money at the expense of quality - or even usability. (Many housing estates and not a few McMansions.)
Both of these are different to a tech co hiring the equivalent of a minor starchitect to Make A Thing Happen, but getting something non-performant and/or impossible to maintain. The difference is that unless it's a greenfield startup Thing is likely to be easy to define and its performance should be easy to measure objectively with simple metrics like uptime, throughput, and so on.
Half of the companies would have died if they followed this -- including Twitter (which had to cron-restart Rails to get it to run for more than a day in its early years, Facebook, ...)
Any high-skill industry has huge amounts of problems with this — rarely outright fraud, but lots of problems with "confidence men" pitching mediocre solutions to problems, and conning their way into running the full lifecycle of contracts based on this. Generally they'll deliver something, but it's usually something pretty mediocre that only kind of does the job. The further a thing is from a company's core competency (and thus, the likelihood that the executives have subject-matter expertise), the more likely this problem is.
People can raise objections all they want, but there's so much "poor faith" lying done in our world, that, as a boss, it's almost impossible to tell which things are libel, and which things are a desperate attempt to warn you something's wrong. Rigorous "professional bodies" like doctor's associations help immensely. These problems are at their worst in fields where things like that don't exist (such as software).
Thanks for summarizing my biggest gripe with my current employer so clearly! It is going on, and all the senior all the engineers see it and there is nothing you can do to prevent it. Only make sure you stay away from whatever these bozo's fancy, as someone will have to clean up or take the blame when everything falls apart.
Anyone know an alternative GTFO? After a few years of complain/ignore/avoid if feel that is my only option.
> Look Mr. C-level, I built this amazing tool. Look at the graphs it produces. It allows _the_ user to customize parameters, track new things, and basically makes unicorns excrete rainbows.
> OMG, I don't know the first thing about how computers work, but I love colorful pictures and unicorns. You are amazing. And, this is the first time you tried this stuff?
> Yes, but these jerks over there, they won't put it on the web! They say it is insecure, risks revealing confidential data to the whole world, and can be used to infiltrate the company network.
> But these are the best colorful pictures I've seen. Off with their heads!
> Look Mr. C-level, I built this amazing tool that makes unicorns excrete rainbows. It is a PoC built in my own time cause I don't have time/money or a project.
> Wow, great! We need this, it fit right into our strategy for Digital! Just go talk to Mr. Bozo, our head of Digital Innovation Strategy Management.
> O, so you have an idea and want budget. Please write a two page memo describing how you can add value, who the internal client will be. Make sure you get steering council approval too and approval from our CISO. If you succeed I will present this to the board as something cool I am personally responsible for! O, and please ask the internal clients for funding, cause you are providing value right? They will be happy to put you in their year plan. The plans for 2022 have already been sent to the board for approval, but this can be the first thing for 2023! In the meantime I will buy an off the shelf solution with <big vendor> and pay a fortune to let them customize it so it doesn't work anymore. Before anyone realizes what a waste of money that was I'll be long gone, haha. Off to the <big vendor> sky box, they invited me to come watch the Super Bowl again.
> It is a PoC built in my own time cause I don't have time/money or a project.
That sounds like a complain, which might motivate others not to listen to you ;)
Absolutely you can do it yourself, but that is also a common pitfall. The belief that something can just be built in x amount of seconds by yourself comes with several risks. Building is easy, committing to a lifecycle is hard and managing the technical debt is an important part of the job.
I'm trying not to build a house of cards.
I'm working in a semi big tech company and not making anywhere near that (in UK) should I cry myself to sleep ?
For context, making GBP175000/year or over would put you in the 99th percentile for the UK.
Source: https://www.gov.uk/government/statistics/percentile-points-f...
Then again, I do live about 250 miles north of London, so that no doubt skews things somewhat.
But take home base pay is usually in line with whatever area your are living in and the amount of exp you have. Everything else is bonus. That bonus is usually bandied about in interviews but sometimes never materializes.
Absolutely correct. This reflects perfectly with my professional career. In the early 2000s I worked at a place which started hiring "bozos" in frightening rate. They killed an internal project for CMS (I was the PM) and proceed to hire external agency for content management, because - "Nobody of our clients is willing to do this complicated technical stuff".
After a heated "debate" with the "bozos", management and CEO of the company, I quitted my well payed job. I created my own company and hired all of my team. For the next 10 years most of my ex company clients came to me by simple word of mouth marketing.
All the customers who leave when your product/service goes to shit because that thing actually didn't belong in production?
> ... is the crux of her point.
No, the crux of her point is that the company is now suffering. If 'X' did belong in production, the company would now be blossoming.
What problem? Sounds more like a solution.
It’s kind of a universal theme in any endeavor. I work in a very different place and have similar experiences. As a founding member of a “startup” within a massive bureaucracy, we grew from 4 people to 4,000.
Remember that people are different. My wife is a finance/auditor by trade - digging through balance sheets and controls to audit stuff is awesome work for her. Conversely, some of the details of how the sausage is made when you build a growing service just seems “wrong” based on her training/profession.
I would say that many companies corporate hierarchy enables/rewards people with political goals to end up in positions of power which slowly corrode the company. It also allows for "bozos" per OP language to propel up the ladder and espouse their ideas without any pragmatic experience or understanding to really solve the problems. People who present well and can play the political game but can't translate to execution.
The worst people for that are the finance people who are sometimes good at numbers, models, they think they understand everything (I.e. MBAs) and somehow accrue power. Once you company has a lot of those at the top level of leadership you are about to enter builder hell.
End of the day everything has a lifecycle. If Alcoa acted like a startup, they wouldn’t be around for long.
Success is not guaranteed, but that can be a well-earned element that at least greatly reduces the odds of failure when they start their own company.
> Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan instead of looking for whether they could do some dumb coding card trick in a 45 minute interview block. I think they'd look very different, and a whole bunch of people would have to find another industry to prey upon.
I would caution against hoping for a personality test that could weed out the phonies, though. "Behavioral interviews" are strange enough already, in my opinion, even if their goal might be admirable (e.g. identify the assholes).
Anywhere there is money and prestige to be had, you can either keep things small or inevitably suffer the phonies.
She's left the positive case out for the charlatan comment, they wouldn't be charlatans (in this instance at least).
If you tell me to build you a system you don't need, and I do, and you insist on using it against everyone else's better judgment, how did I become a charlatan?
because the more likely situation is that the newly hired expert on X is the only one realizing that X is wrong for this case.
But that doesn't suddenly shift all the responsibility to the expert. The power relation and degree of freedom between those who hire and those who get hired is often asymmetric. The ones who hire usually are the ones who make the decisions, so they should be held accountable for those.
Now of course if the expert does have a sufficient degree of freedom then they become complicit.
I don't know the specifics of Rachel's parable, if any. But by and large, the people who are most well known in industry for 'expertise in X' are the devrel / tech evangelists whose job it is to go on stage and be an expert, for one hour at a time (and perhaps a few extra hours on the show floor surrounding the conference). Well, the on-stage demos they execute only have to live as long as the lecture, and you don't have to fix bugs so much as rework the preso to avoid them. It's a long march from a proof of concept to production readiness, and I've seen one or two stories on HN about people who got burned hiring high profile devrels to lead an infra buildout.
> sometimes either their belief in X is too strong ... or, ... the monetary and career-enhancing incentives are high enough to overcome any unease they may have
One scenario I've seen play out: you hire a core maintainer for a product you depend on, but should probably be transitioning away from. The maintainer you typically don't have comparable expertise in anything else, so they might as well apply for unemployment before telling your boss you recommend against it. Which I suppose is covered in 'monetary incentives' but with tinge of regret about the whole thing, especially if hired away from some a more stable role.
When I think of "experts in X" I usually think of the people who created or maintain the thing, or who're active in the Github issues. I guess some of those people might be considered tech evangelists, but I don't think I could name a single devrel person.
The corporate-climber-who-can't-build can often do this and nothing else, and they can sometimes even subconsciously exhibit some of the behaviors that end up with the same outcome, and it often can actually be unintentional.
"Tell me about the most complicated projects you contributed a significant part of the design, implementation to, or ideally both."
Start with the problem they aimed to solve.
Probe for how they chose the solution, and what alternatives they considered.
Get into the details of specifically what artifacts they produced, and if part of a team what role they played.
Get them to describe the solution in technical detail.
Probe into trade-offs they had to make or compromises.
Ask what aspects they found novel or innovative.
Once they finish describing the system, ask if it was successful, and how they measured it.
If they were around after launch, how did they operate it, monitor it, what unexpected challenges they encountered that they didn't foresee.
By the way, all along it doesn't matter if the domain or technology they are describing to you is totally different from your own experience. A senior+ engineer should be able to explain their domain to someone equally technical in a different domain with some competency.
Finally, since you now understand the problem space and the solution yourself, ask them "How would you scale this out to 10-100X" (by some metric).
Altogether, this is a dense 40-45 minutes extremely well spent. And it weeds out the phonies and charlatans.
Some yellow flags that individually aren't a problem but when you start seeing multiple of them turn into a red flag:
* Someone who didn't question the requirements and just accepted them from upstream.
* Someone that wrote a lot of code on the project, but didn't make any of the tough decisions.
* Someone that avoids talking about what they did, and focuses too much on "we"
* Someone who did all the initial decisions but did none of the actual implementation work.
* Someone who describes how X solved the problem, but you realize that's the only option they really considered
* Someone who left before the project finished, or immediately after.
* Someone who isn't sure if the project was a success or didn't see it their responsibility to find out.
I have to say this is an awesome thing to be reading as within a couple of months I'll be taking on my first major project at my new employer. Valuable insight to consider as I proceed.
Nothing, that's expected for a junior developer. The items on the list only become a problem when a "senior" developer does them. One of the parts of transitioning from junior to senior and beyond is learning to understand the business (or other) context you perform your craft in and shaping what you do to help the business succeed.
[EDIT] When I say "help the business succeed", that may sound like management claptrap (even if it happens to be true) so I'll also point out that even if you don't give a flying hoot about the business or its goals, you still do care about understanding whether it's succeeding and profitable/effective at its goals, lest you get blindsided by a layoff or cancellation of your project.
Cheers.
It's only a yellow flag for a senior engineer - which in the framing of the original posting was being discussed. Someone with 7-10 years of experience, I'm imagining.
This is very interesting. I naturally use “we” when discussing past products and achievements. I do this in recognition of the fact that things are very rarely designed and built in true isolation. Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward.
“We” is absolutely not a red flag for me.
Principal engineers know that there are appropriate times to say "we did..." and appropriate times to lean heavily into saying "I specifically did..." (while crediting others for their part, of course, but as briefly as possible) and they also know which one an interview is.
"We" may not be a red flag (I appreciate a measure of humility and recognition of teamwork as well) but it must always followed up by a question about the details of what the interviewee did.
Then you will not get recognition about what you did in the eyes of others. It is hard to trust someone who hides behind a group, because it looks like you have no confidence in yourself or what you did.
Someone constantly using "I" like they worked in a vacuum, and didn't play nice with their team is a bigger red flag for me.
The important piece in the interview is can they answer in detail the rest of the questions.
The worst thing with guys like this isn't that they're going to deliver less than promised, but it's that the aggressive "willingness to lie for their own benefit" bleeds into a lot of other behavior at work.
The imposters/charlatans (and weeding them out was the context for this whole thread) find a way to avoid the question.
Start with the problem they aimed to solve.
Probe for how they chose the solution, and what alternatives they considered.
Get into the details of specifically what artifacts they produced, and if part of a team what role they played.
Get them to describe the solution in technical detail.
Probe into trade-offs they had to make or compromises.
Ask what aspects they found novel or innovative.
Once they finish describing the system, ask if it was successful, and how they measured it.
If they were around after launch, how did they operate it, monitor it, what unexpected challenges they encountered that they didn't foresee.
By the way, all along it doesn't matter if the domain or technology they are describing to you is totally different from your own experience. A senior+ engineer should be able to explain their domain to someone equally technical in a different domain with some competency.
This verbatim can very well apply to many different industrial engineering efforts, regardless of whether there is software involved.
>This cannot be upvoted enough.
I agree, and I mean far beyond the message board.
In my experience, people leave just after the big project ships. They've been focused on a task, working hard and are usually left with "now what?". It's interesting psychology.
Can I tell you a perfect personality test that will always work and never fail? Absolutely not. Even the anti asshole processes don’t always work. But it seems a good thing that this sort of behavior is at least being described and reported on.
I guess it’s better, but the proof is in what the management team looks like.
You'll see this sort of things a lot from Rachel's writing, but I tend to write it off as a stylistic choice. She kind of channels the BOFH [0] sometimes, I think :)
[0] https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell
The incentives are misaligned at pretty much every level.
Facebook didn't gain its users overnight and without a good amount of manipulation and tricks on their part, but they were smart enough to not make it feel as aggressive or directed as Google did with Google+
Personal opinion, they went about it entirely the wrong way. Facebook does technically have a real name policy, but they got the ecosystem of people using their real name that they have not by strenuously enforcing that policy, but by incentivizing people through the design of the product to use their real name. If memory serves, I got my Facebook account via a link that had been auto-emailed to my college email address, and that link pre-filled in my real name in the account setup process.
No, it was a good social network on paper, and they threw the whole company behind it. The issue was that it was meant to be a drop-in replacement for existing social networks. Except it didn't really solve any new customer issues, just solved the google issue of google not having a social space.
They work because they have a quasi-monopoly on something, not because of efficient use of employees time.
Not just attracts, but selects for them as well. The interviews are set up to select people who are morally ok with doing arbitrary and pointless work (months of leetcoding at home) in order to game the system.
The problem was to staff a quality engineering team was going to cost 5x the yearly price of the most expensive service and hiring other people in that space was proving to be very difficult and we didn't have time to take a year to build it and then start migration
In the end I chose to go with a vendor and a smaller team to support a migration. We migrated in less than 6 months and a year later we ended up getting contacted by law enforcement about some activity that was happening our system. The auditing system on the vendors side allowed us to quickly diagnose what exactly had happened and let us give data to folks who needed it.
Had that been our homegrown system, it would've been 100% an unmitigated disaster that took days to unravel. Had we decided to build it ourselves, there's no way we could've hit production as quickly as we did.
Like, maybe I don’t quite want to build an X, but other engineers and managers have their own reasons why we should, and after we discuss/argue for a little while, I decide that, OK, fine, I don’t entirely agree but I’m willing to concede that maybe I’m wrong and give it a shot.
The thing we end up building is OK, but not stellar. We have an outage or two but they’re minor and not an ongoing headache. We have a few wins but it’s not the runaway success we hoped. Maybe we should have gone with Y instead of building our own X? Would it have been better that way? Am I a charlatan? I don’t know.
They're a builder, all right, as long as you want them to build X. Perhaps they have been building X at some other company, and now your company has hired them to come and do that same job over here instead.
Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan
The company specifically filtered them in to build the X product they're renowned for building at other companies. The hiring process is working perfectly if the company decides they want X, hires a famous X builder, and that person then builds X in their role.
I think the point Rachel is alluding to is that the person should have realised X is inappropriate and said so, but if your world is X then it can be hard to see the broader picture. We've all tried to leverage the tools we know best to do jobs they're not perfect for rather than start from zero with something more appropriate, especially when there's a looming deadline or a lot of pressure from higher ups. To call someone a charlatan for doing that seems grossly unfair.
Also, for political reasons, nobody else can or will question it, so there's no safety mechanism.
The x builder can have this affect, on the one end of the spectrum what you describes happens, on the other end of the spectrum charlatanerie happens.
This sounded interesting but I searched for her blog and I couldn't find it. Does anyone know which post this refers to?
- https://rachelbythebay.com/w/2020/02/08/miss/
- https://rachelbythebay.com/w/2019/10/13/firewall/
- https://rachelbythebay.com/w/2020/08/17/potato/
- https://rachelbythebay.com/w/2019/11/10/scale/To see it more easily, here is her post but with some specific, made up example instead of "X":
A chocolate factory wants to build a new dark chocolate line. They hire a very experienced industrial engineer, who just recently finished building car engine line for the Ford factory. However, once the new engineer started to work on chocolate line, they found it extremely boring and devoid of challenges. So they somehow convinced management that the chocolate factory needs to start producing car engines for their delivery trucks. This is obviously a bad idea, and many people point that, but the engineer has secured support of one of VPs, so the project goes ahead. Eventually the line is ready, new engines are produced and installed in all the new trucks. They proceed to break down frequently, the deliveries are late, customers are unhappy, and everyone is having a bad time.
Everyone is blaming engineer now and calling them "charlatan", but the engineer does not understand why: They really worked their ass off building the new engine line, working 80 hours a week to realize their dream. It was not easy -- the factory was only doing chocolate before, so switching to heavy machinery was quite a challenge! And the reason customers are unhappy is because of the management -- this project should have gotten way more resources! Then we could test more and iron out all the bugs instead of having to go to production too early.
Rachel's final paragraph puts the blame squarely on the engineer, with wishes that "hiring processes filtered out this kind of charlatan" and "whole bunch of people would have to find another industry to prey upon", but that engineer is not a charlatan -- they have a legitimate technical talent, and they used it to build real stuff. No hiring process can filter them out -- they will pass a technical interview and take-home test, they have great communication skills, they have a github page with open source code, and they can talk passionately about their past successful projects.
Instead, I'd put most of the blame on the management. In a large company, it is inevitable that someone will work on stuff that is useless or could harm the main product. Maybe because engineers lack perspective, or they overestimate themselves, or they just don't care about company's goals and want to do what they find fun. The company should have some mechanisms to prevent this -- maybe management should listen to feedback from engineers, or there should be technical committee, or at least people should not be rewarded when they ship a broken product. If the company has no such mechanisms, it will have problems as described in the blog.
A senior/staff/principal engineer should be aware of the customer needs they are working to solve, and focused on the problem. Their engineering skills are a tool to fixing it, not itself the result.
In your example, * The industrial engineer should have asked "Do you need me to build car engine machinery?" And if they said no, pursued a different job.
* Once on the job, if management said "let's start building car engines", he should have said "Why, how does this help our customers?"
Actually, in your example: > However, once the new engineer started to work on chocolate line, they found it extremely boring and devoid of challenges. So they somehow convinced management that the chocolate factory needs to start producing car engines for their delivery trucks.
That's borderline unethical. If you're bored and devoid of challenges. Change your job, don't ruin your company.
You place the blame on managers and say "maybe management should listen to feedback from engineers", but that's exactly what happened in your scenario, and was the root cause of the problem! The manager listened to a poorly motivated, mis-incentivized, yet strangely persuasive engineer.
The problem was that neither the managers nor the engineer actually thought about the customers, or listened to them. Incidentally, maybe they would have found out that the customers actually really like the chocolate but it always get delivered too late warm and melted, and so actually investing in building bespoke engines for their trucks would have been correct in that situation. But it has to start from the customer, not the prior skills of the engineer.
Fix the process, and maybe neither the managers nor the engineer need to be fired.
One had written several books on Java and was tapped to head up a new initiative. They "noped" out very quickly, "this isn't a project suitable for java" and recommended a couple of different architectures and platforms and went back to their less prestigious role.
The other was very active in the Java community and shoe horned Java into a very inappropriate product, costing the client $$$ and never reaching the MVP stage.
You could blame management for both of those situations; but one of those companies went on to a modicum of success in an acquisition because the engineer decided NOT to turn the chocolate factory into a car factory.
That sort of exemplary behaviour can save bad management, but doesn't excuse it.
( Of course, management have to be open to these conversations and this alone won't save you from the terrible politics Rachel describes. )
Imagine a different situation - there's a fellow engineer who's also an olympic gold medalist in some sport. They may be awesome to have around. They may have cool stories. They may have connections. But unless they can harness that towards sometging that helps your mission - it really won't make them any better as an engineer.
So when everything is in place, from management point of view it is easier to charge ahead than acknowledge it was a bad decision.
In the end when everything falls appart, they will either no longer be there, or find other reasons why it failed.
Our $company lacked of automation of some process and after months of wasting hours doing it manually I decided to automate that.
I had no exp with real ci/cd tools and thought our process was already crazy enough (using annoying source control tool, heavy dependency between compilation and external soft's specific versions) that we need to build our tool to manage it.
I built MVP as CLI and it worked. As time passed everybody wanted to add tons of stuff to it - like Web GUI, history, logs, managing software versions when creating binaries and even performing code analysis (because we were heavily dependent on 3rd software we needed to detect some colissions)
Overall, it saved us from doing hundreds manual, sometimes error-prone tasks per year. It allowed us to add customizations like that code analysis using compiler's API
But the other hand maybe we could just use some Jenkins or some shit and achieve similar stuff without having to invest time into it and maintain - it's combo of Visual Studio + MS Build + Team Foundation Version Control + .NET Framework, so stuff tends to make problems from time to time
I feel that despite all good things that this project brought, then we could just use existing tool and maybe achieve similar results, but I don't know for sure.
I feel like sometimes I used this project to escape boring CRUDing and used it as a learning opportunity - maybe I abused it too much?
Or maybe should I treat it as "Google's 20%"?
You're duct taping process onto an existing product cycle.
I've experienced the 'X' factor the author is talking about and it's something like "we're moving everything to Linux because that's what I do" and then you go 18 months without a release and the release you do make is a bugfix release to the old infrastructure, your customers leave and you get bought for debt.
I'm not sure how could I rewrite it to avoid that feel, yet still include enough details to let people kinda evaluate whether it was reasonable or not
Some people don't really care about all that and just build whatever keeps them interested and then move on and that's the end of their thought process. You don't seem like that, so I would guess you are not a Charlatan, whatever the correctness of your prior decision(s).
"Fucker's settin' up franchises!"
But..... it is totally generalisable. I have seen this happen myself. Rank-and-file team rail loaded into using Senior Person's X because Senior Person has made X rather than it being the best fir for the job.
In lots of places I have worked there is a particular term that is often used to precisely describe this situation, it is called POA aka "Promotion Oriented Architecture". In POA, you deliberately add a lot of accidental complexity to come up with a fiendishly complicated solution to a relatively simple problem. In short, the complexity stems from the solution space rather than the problem space.
Dealing with incompetent management is never nice or easy, but I'd advocate that as software developers, we're quite privileged because we get paid pretty well, work is versioned in source control and most messes are physically unseen, and cleaning up is "usually" not a big deal.
This hilarious part about it was, they never actually worked at a big company.