Worst of breed software
worstofbreed.net
worstofbreed.net
The manager doing the exit interview started getting defensive and blaming my "attitude problems" [1], and eventually I started explaining that it felt like the entirety of the culture at BigCo, particularly amongst management but even with engineers, came down to "try and justify your existence in the company". Instead of doing things ways that are easy and straightforward, you instead were incentivized to make your code complicated so you can brag about how complicated it is, and then drop constant references to your management about how hard what you're doing is.
The manager didn't like this response, and got more defensive, we ended up going back and forth, and eventually the interview ended and despite taking a pretty considerable paycut I was ok with my decision.
I didn't know the term "Resume Driven Development" until after I left, but that was a pretty accurate description.
[1] Not completely wrong, but that doesn't absolve them of their sins.
It's not as simple as "BigCo" vs startup; I've worked at startups where layoffs were frequent enough that it devolved into existence justification, and I've worked at BigCos that actually did give a fair bit of leeway in how you do things (within some degree of reason).
The closest thing to a "rule" that I've come to is if they use a less-mainstream language; if they're routinely using Haskell or something, they're probably a bit more onboard with experimentation, but that's still not a hard and fast rule.
(Sorry, you struck a nerve with your BigCo depiction. :-)
When I said that the job boiled down to a lot of people trying to justify their existence, Steven said "do you really think that people are doing things to justify their existence in the company, Tom? Really?"
I responded back with "Yes. I think some managers, STEVEN, really like to schedule meetings to make it look like they're doing important work, STEVEN, despite the fact that most of these meetings are useless and could have been handled over slack STEVEN. I don't want to name names STEVEN, but I have observed it on the management side. I suppose you'll need to figure out who I am talking about STEVEN".
This was several years ago so I'm paraphrasing, but barely. I really disliked that job and when he wouldn't just let me answer with "it wasn't a good fit" I got (maybe irrationally) angry and it ended up being an excuse to air all my grievances. I could tell that he was getting upset when I started basically resorting to thinly-veiled insults. Not my proudest moment, to be 100% honest, but I also can't really say that I'm sorry either because I meant everything I said.
Just my view as a dev who's largest co was like 500 people. ~100 engineers.
Big companies are significantly better to work in when you're either (a) in sales with a clear path to hitting/exceeding quota, (b) a strategic revenue generator, or (c) a super hot and extremely well funded corporate initiative (basically all AI projects right now).
The money tap is always on, you get all the cool toys, travel perks are great, and you get to work on amazing stuff without as much red tape.
There were occasional bits of ambition to occasionally work on interesting stuff, but it was mostly a “keep the lights on and then figure out how to make yourself seem important”.
One of my biggest pet peeves is when engineers say that we can’t do something because we would have to learn something new. I got into several arguments because I wanted to rewrite some buggy mutex-heavy code (that kept getting me paged in the middle of the night) with ZeroMQ, and people acted like learning it was some insurmountable challenge. My response would usually be something to the effect of “I’m sorry, I was under the impression that we were engineers, and that we had the ability to learn new things”.
As I said, complaints about my attitude weren’t completely unfounded, but it’s just immensely frustrating for people using their unwillingness to learn new things as an excuse to keep some code in a broken state.
Usually I advocated for doing things a more boring way, and I certainly don’t agree with making every damn thing an “initiative”, which was my biggest issue at BigCo.
I don’t think it’s inconsistent. I wanted to use the right tool for the right job. Usually I can get by with Java’s built in tooling, and that was my initial attempt at a rewrite, but I ended up trying to re-invent a bunch of concurrency patterns with BlockingQueue and I found that literally everything I was spending a lot of (my own free) time was handled in like four lines of ZeroMQ.
I have a single line on my resume for ZeroMQ as a keyword, despite having used it in many, many projects, so it certainly wasn’t using explicitly to pad my resume.
@tombert recognized that the homegrown tech was awful (*) and proposed a mature, reliable, well documented and supported, low-cost, utterly mainstream and mature replacement. That's not resume packing, that's pragmatic, rational software design.
@tombert also knows that every tech professional must routinely learn new things, otherwise they'll be unemployable dinosaurs long before retirement age. Tech dinosaurs aren't a pretty thing in the workplace.
(*) Especially awful because these are mutex and concurrency bugs, and @tombert knew that nondeterministic bugs cost expensive resources to investigate, find, and fix, simply because these bugs are unreproducible. Unlike straightforward deterministic bugs, concurrency bugs are open-ended tar pits that managers and engineers despise. These kind of bugs can eat up a project's schedule and energy.
edited: formatting bug. Fortunately it was reproducible!
Of course there are an effectively infinite number of potential routes you can go down with software, and of course you can't learn all of them, and you can't import every single helper library you'd like to.
We all like to think that the way we want to do things is objectively the best way, and I do think that there are objectively better ways of doing some things involving concurrency and the like, but a lot of the time it is subjective.
But as you said, I wasn't trying to import a library that was the latest hype on Hacker News; it's ZeroMQ. It's fast, well documented, easy to use, and very mature software with very good libraries in every major programming language, and it implements nearly every concurrency pattern that you'd want to use for most projects, and importantly it implements them correctly, which can be harder to do than it sounds.
As I said, I did have an attitude problem at that point in my career. I can blame it on a lot of stuff (untreated sleep apnea being a big one, as I later discovered), but I will admit I probably could have and should have been a bit more diplomatic in how I proposed these things.
I didn't really blame the person who wrote the code for it breaking (who had since left the company), because writing correct concurrent software is hard, I'm sure he had a reason at the time for doing it the way that he did, and of course all non-trivial software has bugs. What bothered me is that I had been designated at the sole person to deal with these issues, so I was the only person who had to deal with the consequences with these actions. The code hadn't been touched by anyone in years outside of adding basic NPE checks, and so I felt like people should let me try and fix it in a way that I thought would be less error-prone, and if it breaks I'd be the one forced to fix it anyway, and I could feature flag the hell out of it in case my code didn't work.
This is key. Writing nontrivial and bug-free concurrent code is extremely hard, it's like writing absolutely solid crypto code. Both look easy, both are incredibly hard and anyone who doesn't know that, shouldn't be writing code at those layers.
Recommending a proven, off-the-shelf concurrency technology is the mark of an experienced and thoughtful software architect.
You're basically stating that people who are hired to staff projects that are superfluous secondary moonshots are more likely to be fired than those who maintain core business areas. That's stating the obvious. When a company goes through spending cuts, the first things to go are the money sinks and fluff projects that are not in any key roadmap. This is also why some companies structure their whole orgs around specific projects and even project features, because management limits the impact of getting rid of entire teams by framing that as killing projects or delays in roadmap.
I see this as a red flag, and an attitude problem typical of toxic employees. I'll explain why.
More often than not, I see this sort of complain from junior developers who either completely miss key requirements behind constraints or are completely oblivious to operational constraints. They invest little to no effort to try to understand the problem domain and what problems are fixed, and instead they redirect all their energy arguing for major rework that they claim makes things simpler albeit their analysis is superficial and simplistic as they are oblivious to the actual constraints. But that doesn't dissuade them.
This analysis failure ends up creating problems within the team because they conflate any lack of support for their half-baked ideas as an irrational opposition to their personal initiative, and thus somehow a part of the problem. Consequently, you start to see those types lashing out at team members and throwing accusations and being an all around pita. They pull these stunts at every single occurrence of any minor inconvenience, as if it automatically means anything else is always better than what they have.
The worst comes if these egregious types get their way. They roll out a change that they depict as a silver bullet for all inconveniences, except that even in ideal circumstances in the real world there are always minor inconveniences. When they surface, these egregious types get all defensive and throw tantrums to bully the team into suppressing any sort of criticism that they themselves spent their time creating for others.
There are many reasons why some companies impose and enforce principles such as "respect what came before" and "disagree and commit". No one wants to work with the assholes who don't follow these principles. Their output is always pristine and the shit shortcuts they took are always thoroughly justified, but he'll forbid if someone else's changes have anything that pricks their sense of taste.
It's a red flag. Clearly an attitude problem. Throwing a whole organization under the bus is a clear tell. If everyone around you is an asshole, the ugly truth is that you are very likely the only asshole around.
I wasn’t trying to rework everything and I am not 100% sure how you reached that conclusion. There were just plenty of times that for certain things there were clear simpler ways of doing things but those weren’t sexy and you were only incentivized to constantly do things to make yourself seem important instead of actually doing work.
I’m not an idiot, I know that you cannot rewrite everything in a 20+ year old codebase, and I wasn’t suggesting as much. At BigCo, instead of actually doing work, you instead were expected to hold several meetings about every minor change you were going to make, come up with a million classes and abstractions that serve no purpose other than to increase your LOC count, and then have a bunch of things to point to at your end of year self-review.
I am pretty sure most of my teammates liked me just fine, because I was generally not critical of any individuals code, because I felt like most of these issues were top down. Management created a situation that if you wanted to get a reasonable yearly pay raise you had to be able to point to a bunch of “big successes” (their words) in your yearly “self review”. Great if you have big projects to point to, but if you’re working on a “keep the lights on” team like I was, you start running out of new proper nouns to list. Instead have to constantly rebrand any small small thing you do as an “initiative”, and you point to the dozen meetings you called for it, and maybe even literally mention the LOC written for it.
Companies that have been around for awhile will always have cruft, and I agree, a lot of seemingly “stupid” code actually has a good reason for being that way, and I knew that even at the time. The code quality itself wasn’t really what I was complaining about, it was the politics around it.
But please go on about how you think that I shouldn’t be able to express my frustration at a corporation’s idiotic policies at an exit interview.
In practice they led to fairly severe security vulnerabilities. "XXE" used to be an OWASP Web Top 10 issue, and the reason it dropped off the list was because XML mostly went away, not because it stopped being a thing.
> But at least, I think XML doesn't have macro expansions, so that's a win.
XML, like HTML, has entities that can be expanded. Unlike HTML you can define them in XML and this led to the "Billion laughs attack": https://en.wikipedia.org/wiki/Billion_laughs_attack
Well, that seems to not matter for the people writing YAML.
> XML, like HTML, has entities that can be expanded.
Lol! Of course I'd be wrong about that.
Expecting XML not to have a well known security vulnerability is a losing proposition.
Haphazard schema is the quickest way to develop terrible performance, loss of referential integrity, and insane queries. Well, outside of sticking everything into a JSON column.
I have occasionally stuffed json into proprietary software data fields to get that data accessible/usable by another system.
(High performance not required)
It works fantastically well, and don't let anyone tell you that you MUST bloat the CLI interface of your program with every possible dial or lever it contains. We should all be cogent of the fact that, in this very young and rapidly evolving profession, textbook and real-world best practice often do not overlap, and are converging and diverging all the time.
And some closed proprietary software does things like adds a few additional fields for pragmatic end user extensibility like this.
The practice predates JSON, but sometimes your bespoke string or ID or whatever in field "Custom 1" is all compromise you need to make things work well.
I want to name names - I could cross-reference some of these with specific company names that you've heard of, but I shan't.
I'll definitely keep this website in my hip pocket to privately throw at teams going forward, as needed.
> "We are doing DevOps now! The developers write Dockerfiles and the Ops team operates Jenkins, which cannot build the Dockerfiles."
I have DEFINITELY seen this done in production back when containers were en vogue! This and Dockerfiles passed around by email.
Far better to shove everything through port 443 and redownload the software over and over until the end of time.
Windows will invariably tell them your software is not secure and block it.
It doesn't matter what the software does, all that matters is that only your company runs it, so there isn't a million people out there with a copy.
Oh boy I have some bad news for thise IT security losers
And we wonder why ageism exists in our industry. Not saying that’s fair or all of it by any stretch, but ouch. It goes to show many of these worst practices are alive, well, and employable.
I'm _not staying you should_, but if it works and makes money, it may not be that dumb.
But seriously, it's not only cynical careerists who are pumping resume keywords like it's a game everyone is playing, and everyone keeps quiet about RDD, etc., because nobody wants to spoil it for everyone. It's also people who are really into one of the keywords, and think it's the most important thing, or the only important thing.
-- Past Case Study #1 --
Context: interview with systems research PhD brand-new founders, after my cold outreach pitch as a "startup generalist" (I think I said it in the headline) who would complement the scientists.
Me: (paraphrased) You're the experts in novel distributed systems research niche X, and I can't help you with that. What I can help with is all the other early startup work you'll need done, like bespoke infrastructure that works with X, Web consoles, mobile apps, some systems programming, product definition, project management, helping academic researchers and industry engineers work together, whatever needs to be done.
PhD founder: (this might be an exact quote) We need an expert, not a generalist.
-- Past Case Study #2 --
Context: interview with a mid-stage startup's CTO, who was hired for distributed systems expertise.
Me: I suspect that a Postgres server on a modest cloud server can handle the entire planet's activity of X. And (since the company's recruiting materials emphasized bias for action, and rapid iteration) we could very quickly build and validate that empirically with simulated transaction load. Of course, in production, we'd set up Postgres with distributed failover, etc.
CTO: I going to need a severalth one-on-one interview with you, an offer is imminent, but it's unclear whether you'll ever be allowed to talk with anyone else on the team, who you pointedly have still not yet met.
But instead it crystallizes the core methodologies of the entire industry.
This should probably be some sort of industry certification or ISO standard.
UUencode all 354 fields (64MB string) and put it in a hidden input field!
Twice!!
update: I'm feeling actual physical pain by reading this and remembering every single piece of software or website I've seen that actually applies these patterns [1].