Drunk Post: Things I've Learned as a Sr Engineer
old.reddit.com
old.reddit.com
This is one of my favorite excerpts. I once worked in a lab where we would have frequent catastrophic failures because there was never any disaster planning or contingency management plan. I personally triaged 3 such incidents alone or with people who happened to be there when the problem arose and attempted to disseminate some suggestions for how to prevent similar problems in the future. No one was interested. People were primarily interested in tearing my head off because I hadn't handled the problem the way they would have done it (of course, they were out drinking beers or sleeping while I was dealing with the issue at 12 AM or on a weekend).
After the third time I said fuck it, the next time there is an issue I am going to insure my own projects are safe and then I'm going home and turning my phone off. Let someone else deal with it. That is the not the culture you want to be promoting.
Because they keep getting shoulder tapped to put out fires. Because they’re the only one who knows the system. Because there is no documentation…
If any random person can break it, it's already broken.
If any employee can break it, it's probably broken (there are very small scales where even this doesn't apply. Ever worked for a company with less than ten people? There's probably something any employee can break).
If any employee that's an engineer, sysadmin or developer can break it, well now you're at least reducing the problem to a more specific set of people.
If only the people on a specific team responsible for a system can affect the system, now you've reached fairly good point, where there's separation of concerns and you're mitigating the potential problems.
If only a single person can break the system, you've gone to far. That effectively means only a single person can fix or work on the system too, and congratulations, you've engineered yourself into a bus-factor of one. Turn right around and go back to making sure a team can work on this.
Finally, realize that sometimes the thing only one team can break is an underlying dependency for many other things, and they may inadvertently affect those. You can't really engineer yourself out of that problem without making every team and service a silo that going from top to bottom. Got a shared VM infrastuture, whether in house or in the cloud? The people that administer that can cause you problems. Don't ever believe they can't. Your office IT personnel? Yep, they can cause you problems too.
Some problems you fix by making it so they can't happen. Other problems you fix by making it hard to happen and putting provisions in place that mitigate the problems if they do.
I say this as someone who’s worked at large tech companies that are “internet scale”.
I think what is needed is a culture of -ownership-. That's basically people saying "I'm responsible". Not one where everyone tries to avoid responsibility, and not one where peopel point fingers.
It's business owner who's responsible, because ultimately he's getting all the expenses when critical event happens, client leaves, client sues the company, and so on. Other people are not really responsible, they just pretend to be.
(Even without getting drunk, I've always found it useful to consider a hard decision once analytically and once intuitively, and if I don't agree with myself, think about it some more.)
I agree but think it is more than just _documentation_: effectively communicating ideas through text was one of the most underrated skills in software engineering. I say "was" as I think there is much more focus on it now with remote work and video call fatigue becoming the norm for many.
learning-oriented tutorials
goal-oriented how-to guides
understanding-oriented explanations or discussions
information-oriented reference material
https://www.writethedocs.org/videos/eu/2017/the-four-kinds-o...
Two important lessons I learned:
1. Formatting and direct communication is very useful. It can make the difference between someone stopping and noticing critical information, or skipping it because they're lazy readers.
2. You probably don't know the correct way to convey information, and the audience probably doesn't know how to tell you how to convey it either. You need to listen for when the docs fail: when somebody says they read the docs but still don't know something or don't do something right. That means your doc has a "bug" in how it gets through to the reader. Experiment, change things around, add/remove information, etc until the bug is gone.
And it turns out those kinds of docs are pretty useful to my colleagues in piecing it together also.
* Rust Library Documentation: Most libraries have complete and up-to-date reference documentation, but are lacking even basic introductions (tutorials/guides) on how to use the library. This is totally just my personal experience so maybe I've been looking at the wrong crates, but with most of the crates I spend several minutes looking trough all the modules in order to find that all the juicy functions are hidden in the Connection struct, or something similar.
* Linux Kernel Documentation: The Linux kernel has excellent in-depth explanations on several high-level concepts, but on the other hand a little more systematic reference documentation on the supporting library code would help a lot.
* While I can't think of a good example right now, a lot of projects have a few basic getting-started tutorials but don't explain advanced concepts or high-level design at all, leaving you to wade through the sources yourself in order to understand how to actually use them.
That said, I find high-level documentation for larger systems to be very valuable. I also find Python's docs to be lacking compared to Java's; I'm often left wondering about the definition of what type is returned, exactly what parameters should be, and which exceptions are raised. Java's docs are very explicit about all these things.
I'm a pretty firm believer that documentation has to live next to the code. Otherwise it's nearly guaranteed to be out of date and/or incomplete
Thoroughly commented code tends to end up a couple changes out of date. A separate file in the same directory ends up a couple major refactors out of date. A separate file in a separate system ends up a couple company-wide reorgs out of date.
It works very well, and doesn't really add any overhead to my work.
I write about how I do documentation here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
It's a long read, because it's a big topic.
Depending on the project, various other documents may be required, e.g. installation guide, user guide, operations manual, architecture diagrams, networking diagrams, module/component diagrams, information flow diagrams, high-level design, low-level design, docs at various "views" (such as "business view", "information view", "technology view"), design decision tracker, ontology...
People managing work, from what I've seen, still prefer to babble over their scribbled 5 basic points than taking the time to do their job and create relevant textual information. Then you listen, you take notes, and then you go and produce whatever documentation of the objective is required to at least understand if it's going to work. Of course you still will have gaps in your understanding, so then more calls, and repeat. In the end, unnecessary/missing features, a whole bunch of time wasted in crap, deadlines missed, burnt time, all of which could have been avoided if someone just had taken the time to do their supposed job - this is not to say it wouldn't have to be discussed, or that there's no need for back and forth and calls, etc, it's just instead of starting halfway you start from -50% or something.
Even in outsourcing platforms there's been a shift contrary to that. At least two years ago and before that, video calls or calls weren't really usual unless you were in some months long collaboration - now even there everyone expects video calls on the interview... It doesn't matter if it's a $100 one time job or whatever.
And management is usually/always ok with this short-term optimization.
Code says what is done. Docs says why
Like dude, tell us why we should read the code in the first place.
Trying to write ops documentation on an unfinished project can be this way.
So I argue that the issue is writing skills, of which technical documentation is a subset speciality of writing skills. I will add, similar to math problems or programming, writing wants you to do it over and over so it can get better.
Agreed, wholeheartedly. Will hitchhike on your comment to recommend two things: Brett Victor's pdf stash [1] and, specifically, the Walter Ong essay "The Writer's Audience is Always a Fiction"[2].
Long story short, we form our audiences by subjecting them to our writing. In writing software documentation, we are implicitly informing the next generation's thought by the simple power dynamic that underlies all technical documentation: "you must understand this in order to do your job properly".
It is no wonder that "form", "inform", and "information" are such closely related words.
We dictate the level of rigor and intelligibility we expect out of our technical documentation, when we write technical documents. It almost sounds like a tautology when put this way, but "bad docs" are exclusively the result of a professional culture that puts up with the existence of bad docs. I've been there; too tired and overworked to care about writing something properly, or wanting to avoid writing a doc enough that I setup some autodoc thing and called it a day. We literally don't get paid for writing documentation.
But good documentation is what made us into good developers (if we are good developers). We should get paid for doing that...
[1] http://worrydream.com/refs/
[2] http://worrydream.com/refs/Ong%20-%20The%20Writer's%20Audien...
Absolutely. So many hour long meetings could be shortened by better communication skills (just more targeted), or even a small email chain.
Communicating in short form confidently is a skill. Many people, including myself (something I've been working on) struggle with saying something that was a complete idea in a meeting and not stopping because they feel like they need to say more. Short and sweet is the way to go pretty much whenever you can, for technical work that is.
I find 'small email chains' don't help at all. Too many people just don't read past the first sentence or paragraph. And email is slow.
I usually 'escalate' quickly. Support didn't understand a ticket comment I made on how this isn't a bug or why there really is a workaround for it?
I send them an IM trying to coax it out of them/get them to understand for a few minutes. Doesn't work? Quick call and do screen sharing. Problem usually solved after a few minutes. Problem is they might stay longer than that took just to 'catch up' ;)
This is really not that different from when we were all at the office just that the last part would be walking over to their office and looking at the computer together. In some situations that makes it even easier nowadays because I don't need to take an elevator down 20 stories and back up again after.
Having asynchronous means of communication is great. But as soon as the back and forth is more than maybe twice on each side, there's probably a miscommunication happening somehow that will be much easier to get solved with a really short feedback loop. Some people you have to call right away coz they just type soooooo slowly ;)
Example 1 - too much details. Morning stand up. Manager: what's the status of that new feature. Senior Engineer: well I tried to call that service but it was timing out so I spoke to Bob and he said to check with DBA on why that stored procedure but it's so slow and turns out index is missing so we tried to add it but mysql and varchar something fckn something...
Dude couldn't you just tell it's delayed due to DB and then expand if needed.
Example 2 - insufficient details I return from the meeting and discover avalanche of emails, chat messages and urgent meeting invite, all with same topic - "Blah service fails, we are blocked" but no details apart from that. On the call I get description of the problem - blah service fails and how everyone is blocked and how infinitely critical it is and what ETA for resolution would be. What endpoint? How does it fail - timeout, connection aborted, 503 response, 200 response but with error message?
Having links to actual code, like GoDocs have it, is something I appreciate too.
And communicating gets even more important the higher up you move in your seniority.
I thought they were going to go the direction of "he's an asshole" and was ready to accept that, but this particular criticism is actually disturbing. People with strong visions can often appear to be "making it up as they go along," when really they are just subpar communicators.
Short story, I am helping my current company switch from a monolith to service-oriented architecture, and in the process have built a framework for spinning up new fully-deployable services from scratch that gets engineers 90% of the way there (minus the business logic). I have a strong vision for how it works and had a dozen-page RFC accepted by the engineering team for it. Yet there are engineers who think I am making it up as I go (I have been asked this indirectly), without any vision guiding the pieces into place. I have chalked up this feedback to me needing to improve my communication of the vision.
So the post's response of "I realized that he's making it up as he goes along like the rest of us" is disturbing because it makes me realize just how difficult communicating a vision is... if this hero that the poster paid $5k to go see can't even convince one of their fans, what chance do normal people like you and I have of convincing people that we're not making it up as we go?
>EDIT: I realize that I am posting under the assumption that the person's hero does in fact know what he's doing. If he truly is making it up as he goes, the above doesn't apply.
Then there's the times where you think you know exactly what you're doing and after going down a road you realize it's the wrong way. Failing to learn and see the signs and make the embarrassing declaration that you got lost and need to turn around is never good. But some people just keep driving. It's hard though when there's a big line of cars behind you that think you actually know where you are headed.
Sometimes it's a hybrid of knowing what you are doing but not knowing the implementation specifics. You know you need to connect high-level pieces A, B, and C with specific constraints, but it won't be until you get into the low-level implementation that you'll know if it is indeed possible. I think that's an example of both having a vision and also improvising as you go.
I am concerned about how to effectively communicate visions to people, because it gets everyone rowing in the same direction. If nobody thinks that you have a vision, when you do, there is no reason they should choose your direction vs just do their own thing.
Nothing if you're good at it. But if you're hoping to learn something from someone, it is pretty disappointing. How to make it up as you go along is far less teachable.
I take this really just to mean "everyone has faults".
People often idealize heroes and think of them as beyond human. If you do that and met your hero then your illusion will often be shattered. But the problem is just that you were putting them on an unreasonable pedestal.
Of course some people are frauds and some people have no idea what they are doing but manage to make people think they do. But I didn't read this as being one of those situations. Just someone they saw as beyond human being only human.
I "know" guy who's conference speaker, so he knows other conference speakers, drinks vodka with them and so on.
He says there's significant amount of bullshit aka things that work nice on slides, things that only work cool in theory, but in practice they arent as great
I think that's what OP meant
I think people like to believe this because it helps them cope with impostor syndrome, or maybe they think it puts them on even ground with people who do in fact know what they're doing.
The foundation tends to be pretty simple, deeply held, and unchanging, while the higher levels are increasingly fluid and specialized. The higher you get on this stack, the more "making it up as you go along" it becomes, but every improvised part is perched on something more stable.
They key to "knowing what you're doing" is organizing this hierarchy well, having the right supports in place to successfully guide improvisation and course-correction while steadily fortifying the foundation.
Your assessment of why people like to believe this seems spot on.
How do you distinguish between what you do and "making it up as you go along"?
In sports terms it would be something like a baseball pitcher being able to throw a great curveball, a great fastball, and an all right slider, and knowing roughly what situations to use them in. There will still be a high degree of randomness and mistakes will be made.
I would much rather work with people who have a good track record of making it up as they go as opposed to people coming in with a fixed idea of how something should happen and are more likely to misapply whatever lessons led to those views (probably someone elses anyway).
I'm guessing that this was based on first-hand experience building such services and witnessing engineers struggle getting new services up. And not so much that you've had specific training or past experience in developing bootstrapping frameworks. This would be my definition of making it up as you go and is great way to do it. Another way is learning how to make bootstrapping frameworks and applying it wherever you can which doesn't go as well.
(I work at Red Hat, and my programming heroes are also my colleagues.)
That's a weird one. I don't know anything about that subreddit, but HN comments are frequently great. I submit stuff because I want there to be HN comments on it for me to read. I typically read the comments first and only bother opening the link if they were interesting.
>HN comments are terrible. On any topic I’m informed about, the vast majority of comments are pretty clearly wrong. ...
>And yet, I haven’t found a public internet forum with better technical commentary. On topics I'm familiar with, while it's rare that a thread will have even a single comment that's well-informed, when those comments appear, they usually float to the top. On other forums, well-informed comments are either non-existent or get buried by reasonable sounding but totally wrong comments when they appear, and they appear even more rarely than on HN.
It is also a reason why I dont want to mention or see HN links in mainstream media. Although I think most reporters sort of know this as well and tend to not mention or link to HN as source.
Full stack compresses two jobs into one. It's purely for cost-savings. They're paid so little because companies revert to "well, you can still only do 8 hours, so you do half as much of each", but really that's just them trying to weasel out of paying for knowledge. They also blur the lines by putting full stack along-side other devs, even though the other devs may not have invested the same time to gain as much knowledge as full stack.
When you take a full stack job, you undervalue your knowledge (and the time invested) and are selling it for roughly half of what it's worth.
Not having to wait for some other engineer to make the backends made us a lot more efficient. It definitely was rewarding to be able to complete products end-to-end.
Hiring standards and pay were the same as for any engineer, at least in FAANG.
[0] Yeah, occasionally we had to optimize some SQL queries or whatever but we're competent engineers, we can figure it out even if it's not what we do every day.
This is 100% accurate in my experience too, and it's also true in the other way around: "full-stack" means backend developer who can put a basic SPA using React/Vue.
From the frontend perspective: they call themselves full-stacks for knowing how to spin up a NodeJS HTTP Server powered by Express with MongoDB inserting JSON in the database. But they are missing:
- AWS/cloud computing: not necessarily creating the infrastructure (although that's a must on more senior levels), but how to orchestrate the different components together.
- databases: why SQL/NoSQL, beyond basic SELECTs, knowing how to properly add indexes, debug why queries are slow, modeling, understanding locks implications and transaction levels, and so on.
- tooling: how to set up a bundler, linter and formatter, testing, CI/CD. This overlaps a bit with the responsibilities of the DevOps engineer, but a full-stack should know at least on intermediate level all of those things. I can't say how many times I've seen "senior full-stacks" that had no clue about how webpack worked at all.
From the backend perspective: they call themselves full-stacks for knowing how to spin up a React/Vue app that does basic CRUD operations using forms, powered by a UI framework like Material UI. But they are missing:
- CSS: most will find CSS hard/annoying and won't bother understanding at all how it works even on a fundamental level, will defer to hacks most of the time to make things work, especially when it comes to adjusting for edge cases like responsive design or cross-browser support.
- the DOM: normally they don't understand it at all, or to a very limited extent.
- Web vitals: how to measure and makes things faster and performant—not really including here overly optimized, but just making sure your app is 60fps or close to that most of the time. Usually when things get slow either on the network side or in the app itself, those engineers will blame is the framework/library, not their misusage of it.
--
Those lists are definitely non-exhaustive, as I didn't even mention more advanced stuff like protocols (how HTTP works? most can't answer), caching, etc etc, but you can get the point I'm trying to make:
The problem with the term full-stack is that only very few engineers really are sufficiently great on both sides of the stack and could say that they mastered both, simply because there's just too much to learn! Frontend has become so much more complex with SPAs compared when it was about rendering static HTML with some CSS and basic behavior with jQuery. Same for backend with the advent of cloud computing and several types of databases.
I've been coding professionally for a decade and I've met only a single engineer that I'd consider him full-stack (he checked all those boxes I mentioned and more). I think I would also include myself, because I spent 50% of my career spent as a front-end engineer, became senior, then I transitioned to back-end engineering because it pays the same or more and it's less stressful (for most of the regular companies that most of us work at). My current title is "principal full stack engineer", but in practice I only do backend/devops, I don't actively code front-end but I keep up with the industry by following the new trends and testing things here and there in personal projects.
Ultimately, I believe for being a full-stack engineer you have to be first a front-end (or back-end engineer) then learn the other, what we have today is most people doing the same from the very beginning of their careers and they either go deep on a single one or in none of them.
My experience is that at some scale it works out okay, but beyond a certain point it just falls flat for most. We deal with insanely talented developers, who will trash a database, because it don't understand how it works. Talented JavaScript developers, who don't really understand how HTTP works... or load balancers, or caching... or webservers. Sometimes you get these fantastic software machines as deliverables, complex, you can't monitor them, or configure much, and the it just implements a basic feature of HA-Proxy or Apache, but badly.
My point is that they should be paid poorly, because they fail to be excellent at every part of their job, but rather than: Yes, this should in most cases not even be a job title. If you find someone who can do all of this well, you almost can't overpay, but are you really sure that you want to tie everything up on one person anyway?
Full-stack developer knows some frontend, some backend, some sql. They are paid good money because they are convenient, not for their knowledge. A dev who does only one thing knows way more about it that a team of full-stack devs.
Full-stack devs earn a lot and will probably earn only more in the future.
This is mostly a result of tech advancements like cloud infrastructure, tools etc that takes all the hardest things from you - like managing a DB, implementing security, managing infrastructure, deployments etc.
You don't need deep experts because of it, generalists are perfect for quickly shipping new features.
You can make doctor level salary at the ceiling if you move to a FAANG as a full stack dev.
The fact that boot camps even exist shows that it’s not that difficult of a job.
Other industries boot camps don’t even exist.
It sounds great because it is great. I don't make that much and I work on boring systems.
I guess those numbers also explain why the author can recommend maxing out the 401k. People supporting a family on less than $100k don't have $19.5k per year to put into it.
- OP uses the phrasing senior engineer.
- Never worked for FAANG. This is relevant because $120k + bonus/benefits is basically a FAANG new grad. Fairly normal for SV tech companies.
- Those numbers likely provide a solid standard of living, but as a senior engineer you are likely underpaid.
- Esoteric and proprietary knowledge means if you choose to go elsewhere, you will be at a competitive disadvantage compared to using industry standard tools. There are of course tradeoffs and lots of general learning that comes with experience, but all other things equal it's a disadvantage.
> I guess those numbers also explain why the author can recommend maxing out the 401k
Yes, but again fairly typical for the target audience I think. There are even startups trying to target this market to optimize the flow of money from salary -> 401k -> post tax contributions/megabackdoor -> other investments, brokerage accounts, etc., e.g. https://www.helloplaybook.com/
Just as a sanity check, BLS suggests the median SWE wages work out to ~$110k.
https://www.bls.gov/ooh/computer-and-information-technology/...
It sucks that I suck. Oh well.
There were a few post on HN recently about fresh grad asking for / being paid $200K and anything lower they felt they were feeling lowballed.
For those of us outside US we could never quite grasp whether something is true or not. Salary across the pond is just incomprehensible.
This one has started to dawn on me. I'm never going to love my job as much as I love my personal projects, so a job that I don't get very excited about but doesn't drain my energy is better than one that I get somewhat excited about but does drain my energy (not that it's impossible to have both, but it's rare)
Life is extremely, infinitesimally short. If it's at all possible for you, you should try to spend as much of it as you can doing things you love. I know many people can't, but it's bleak to just give up and permanently settle, I think. (Especially if you don't currently have any dependents who rely on you; it changes the equation if you do.)
It seems to me that many if not most people I meet fall into one of two camps.
The folks that just don't care at all and do the minimal amount of work they can get away with not to get fired. The other side is folks that care so much about it that they spend so much energy on as to ultimately be unhealthy for them and also making it worse for the rest of us (not taking vacation, unpaid overtime etc.
It is very rare to find people that care 100% while at work. But after ~40 hours each week that's it. Sure I'll stay later for that one meeting that's sorta important and needs to start at 4:30p.m. But you better not try that every day from now on. Want me to be on call for a fixed rate that has nothing to do with my salary? I can't log an hour for a 5 minute call at 1am and I can't take time off in lieu? Good luck getting me on that rotation!
Can confirm this and it was surprise for me when I discovered it. I assumed they would try to maximize $$ in my offer in order to maximize their profit. Yet most of them were trying to get offer accepted as quick as possible. Seems like more offers with less money in each is preferred over less offers with more money in each.
There are “rules” to how the game is played, and how the information is revealed, and a lot of it seems hidden (in fact I have seen agents that don’t understand the perverse incentives!). A seller’s agent will usually need to make sure they have deniability, and they will prefer options where the vendor remains happy (so the vendor will use them again, and recommend the agent to other sellers).
I have only a very little experience, and none in the USA, but that was what I noticed in my own country.
It surprises me how little effort people put into understanding the game, given that playing it well can easily make the same difference as many years of income.
https://www.reddit.com/r/ExperiencedDevs/comments/nnw7yd/sob...
https://www.reddit.com/r/ExperiencedDevs/comments/noa841/sup...
I guess there will be more on their way.
Couldn't agree more
There is still to this day the idea that we're supposed to plan to future like oracles or pad our resumes on the company's dime.
For me, 90% of day-to-day algorithms use is not writing O(nm) or O(n^2) code, and knowing which data structures to use to avoid it (usually a hashtable, occasionally a heap or balanced tree).
However, to the point of the person you've replied to: It's good to know that stuff to be able to assess the situation in the first place.
Amen! Not just junior engineers. PMs, analysts, testers. We all have a lot to learn from the rest of the team!
The other ones are quite obvious but this is the one that really resonated with me. People, with experience, tend to get more pragmatic and less "academic".
I got older, stopped using python for exactly the reasons above (I just really felt that it was wasting my time for trivial reasons), and found mypy which made it bearable again.
Every time I see an example of a bug that TS would solve, it's something that I routinely find in 2 seconds by looking 10 degrees to the left at my second monitor and noticing the screen is white and there's some red text in the dev tools. "Compile time" doesn't mean anything if it consistently happens 0.5s before "run time".
I believe typescript also has options for compiling statically.
Have you ever considered PHP?
The last time I had a job was on a Clojure team, and that team has since abandoned Clojure and some of my former colleagues have come to me and essentially said that they got tired of fighting with runtime errors.
Strong-typed languages (especially in the damn browser) make no sense to me. It's not like we need to manage memory or anything.
20+ Years of experience here, and I hate TS with the passion of a thousand suns.
I don't see people commenting on this one! That may be a good thing because it means people are not questioning it :)
Just put all the expected input/outputs in a big table test and iterate.
TDD is awful when you don’t have a clear idea about the interface and need to mock a whole bunch of internals that are likely to change.
I really should
Highly recommend a journalism class at local community college.
It does help but I am not sure I would describe _my_ education as a panacea.
The main reason it was good was because it helps frame how you think of writing,
The most important things go first. Statement of fact, then you go into what the implications are, then you start introducing less and less relevant elements.
when you’re writing you keep the 5 “W’s” in mind, make sure you answer them. (Who what where when why), for instructive documentation you add: How.
Obviously an education bakes this into you in a better way than I can convey here.
What I learned is probably only decent for writing overviews.
Personally I find structure to be the biggest bottleneck/difficulty when making documentation.
What do others normally struggle with that don’t have this education?
Learning how to write proper instructions is the gateway to proper documentation.
Sometimes fixing the problem will require special access to Production which you don't have, or even a specific role with that extra bit of initiative.
Otherwise i agree 100%.
If it's unfixable, that's when you quit
Maybe... none of this applies to people who own meaningful portions of the companies they work for.
But this one stood out for me!
> Titles mostly don't matter. Principal Distinguished Staff Lead Engineer from Whatever Company, whatever. What did you do and what did you accomplish. That's all people care about.
To say nothing of a lot of brain.
This a thousand times. Having empathy for future devs, maintenance, and bug fixes is so important.
Very frustrating.
I think it happens because most measures of code quality are quite fuzzy, but brevity (which is valuable, other things being equal) is relatively objective. "Have I made the code shorter?" is a much easier question to answer than "Have I made the code easier to understand, modify and maintain?"
I noticed that they put a lot of their self-worth in the sophistication of their code, so it's difficult to criticise without making them feel bad. You need alternative methods of getting them to "see the light" and write code that's more understandable and maintainable by others.
someone then removed all of my logging because it was ugly.
and then had to put it all back in when another bug was coming from the same terse beautiful code.
In fact, I think a third-party auditor would be a valuable service for dev teams to utilize at least once a year. Totally neutral party that can come in and say ‘We’re pretty sure the codebase is too complex and we noticed the commits came from so and so’. The business value here is you can root-cause velocity issues that can come from decent people who need to be reigned in (not necessarily fired).
I’m literally prepared to pay to have these people objectively assessed.
I think this is more prevalent for some languages/stacks than others, too; there's definitely a cultural aspect fostered the language owners or whoever the leaders are.
I've noticed this too. One thing I've had mild success with is the concept that a particular programming document (especially in functional programming) is really a series of mini-documents. Each mini-document has function-level comments, a signature, body, and returns that tell part of the story of what that function does. The minute that the collective of those fail me and I find myself reverse engineering code, then we have failed the team and cost the company money.
Some complicated things must be done, especially at the size and scale of our products, but complex things are painted with a fine veneer of interfaces and documentation.
I think another exercise that can help is putting junior engineers front and center to architecture. Whether it's exposing them to review, the review process of a Senior engineers design, or putting them front and center to design implications. I've seen having to figure out the difference between a controller and a service cause some really positive abstract thinking that puts people on the order of thinking for the group rather than their own merits.
If these systems are not in place, and a senior developer can get away with writing overly complex code, then that is the fault of management, not the developer.
A first-year student is unlikely to understand C++ template metaprogramming, or just about any Haskell code, but that's not to say they should always be avoided in production code.
> The best code is no code at all
This can be interpreted as advice to avoid the 'inner-platform effect' anti-pattern. Good advice, but personally I'd rather express it in terms of the inner-platform effect.
They're just the people to read a book on the topic and try to use it everywhere...
Don't apply corporate best practice designed to withstand turnover to personal programming - you're leaving abstraction and efficiency on the table.
The better code for your own projects is almost definitely inscrutable to a newcomer a lot of the time. It's okay for there to be prerequisites to understanding.
So I would say great code is code that:
1. Does its job well (robustly, performantly, etc. in whatever proportion is applicable)
2. Is maintainable by engineers with reasonable expertise in the tooling
in that order. If you can manage all that and make it accessible to your junior devs, that will of course make your code greater. But don't lose sight of what your customers care about. Your business isn't there to make you feel good about maintaining code, it's to provide customers with value.
Easy to understand code can be more quickly patched and repaired by anyone on the team. If you don't need to call in the senior who built it two years ago to repair it, and you can have someone do it right away, it is better for the customer.
There are code bases at my current employer that are entirely "ok" and still being worked on from before I was able to spell my name. Projects that have had continued development for >20 years by armies of engineers and the code is still readable and simple to understand.
That performant code that you wrote maybe at the expense of readability? It could very well become bad code when you leave the company and it falls to a junior engineer to modify it to fit some changing business requirement. Or, there’s a bug in the code and the amount of time it takes to fix it is a direct function of how quickly and completely that junior engineer can understand the code.
For me, the hard part is knowing when and how to make that trade off. I’ve definitely erred on both sides often enough.
In my experience, the easiest to maintain code is very often the most efficient and robust as well, because people haven't felt the need to hack around it at every corner.
How many entry level engineers come onto a project per year? 5? Is onboarding them onto the project is a deliberate and streamlined way such an undue burden that you must change your programming style to avoid it?
I am not saying the code cannot be made better or more clear. But, it also depends on who you are writing to. Somebody who is not familiar with certain style of programming cannot easily read the code of certain level of complexity.
When I was hacking away my first big program, I could not write functions. Or find reading functions easy. The whole thing was a big wall of glorified assembly sewn together by labels. I am not sure why I was like that then, but I found concepts 'functions' and recursion or any other conceptual stuff really hard. My code was, in its own twisted way, 'most simple' and utterly unreadable.
I find the same sort of difficulties while reading some FP snippets. I confess it was a very short affair, but I had some difficulty reading it and even when I understood, I could not just write or think code in the same style.
There are ways to make your code better, your intentions clear but 'can be understood by a first year CS freshman' is bit abstract criterion.
It is kind of like, vocabulary and prose. You can make your prose clear. But, people have to work on the vocabulary on their own.
> The best code is no code at all.
This is completely agreeable.
Edit : Changed some poor word choices. Added an analogy.
Usually they end up thinking they can do a better job, decide to rewrite the thing from scratch, and take 10x longer to rewrite it than they thought it would take. And accomplish nothing in the end, because the thing they rewrote worked in the first place.
I real life working on a team with ever changing code, that is the barest rank minimum.
As a young programmer, I thought I was a master when I got the code to work. Now I know that is just the start. Making it readable and changeable is where real mastery lies.
This is very hard to convey to young fools as I used to be.
It works. I look at their code and think "that's neat", but you added 0 functionality while making it hard for the lowest half of the engineers to work with. You could have accomplished the same thing with hard-reference to a class.
codex bonus a discipulo prendatur
codex magnus a novo prendatur
codex optimus nullus est
Part of the problem is I couldn't find any good word for "code". "Codex" sounds cool but may not be the best fit here.
EDIT: Forgot a word. Also, I think "prendere" is better for "understood" here than "scire", which is more like "to know".
EDIT2: My friend suggested using the subjunctive for "comprehend" so that it's "may be comprehended" instead of "is comprehended". Also I got the tense wrong initially and I think that's fixed now.
EDIT3: "Ablative agents" are a thing. This is a rough language. Thanks, James.
EDIT4: prendar -> prendatur; aka "oops, should have used third person"
1) Readable Interfaces usable by juniors when parts of the code will be used by lots of devs and will almost certainly change
2) Higher complexity behind the interface for parts of the code that change less often and require more skilled engineers
- The barrier to entry into webdev is low. You just need a laptop.
- An awful lot of websites seem to have been thrown together by morons; shipped; and never improved.
- For most sites, performance isn't an issue - strong tuning skills for PHP, SQL and Javascript are not an issue.
- A "full-stack webdev" generally doesn't do the full stack - few of my webdev colleagues have been interested in networks, for example.
TBH I don't know. In my last position I was paid £45K; I was the most versatile and experienced developer in the company (12 people). The bosses constantly complained that there was a dire shortage of talent to recruit. They recruited quite a few overseas visitors.
Generally I'd prefer to have front end dev + backend dev in my team over two full stack devs.
This got me and somehow bloody true.
Maybe he is on to something.
Two reasons: 1. everyone calls themselves this these days, and 2. they are often quite weak in each of those parts of the “full stack”
Source: over 20 years doing software dev at various companies with terrible code that makes millions. Happily my current job has really great code (imho).
1) People skills dominate quality of work output in terms of value to the business starting at around this level. Really, introverts should avoid programming altogether which is weird because this was one of the few disciplines in which introverts could truly thrive but those days are gone.
2) What company leadership tells you they value has nothing to do with what they actually value. Look instead to what the company punishes you for doing or not doing.
so either i’m doing OK, or we’re both fucked…
Cannot agree more especially if you previously had a blame-free culture and it doesn't look like it's returning in a hurry.
I have seen this happen when new management comes in and don't value root cause analysis and look for a person or team to blame to protect themselves.
Just get yourself a drawing tablet and some whiteboard software like Openboard (http://openboard.ch/index.en.html).
> Good people write shitty code. Smart people write shitty code. Good coders and good engineers write shitty code. Don't let code quality be a dependent variable on your self worth.
That's like saying that eminent book authors write shitty books. Sounds like an excuse.
> I've become what I've always hated: someone who works in tech in a career but avoid tech in real life. Maybe that comes with being old.
It's all about finding the right project. REST + CRUD shit gets exhausting after 5 years. There's an entire world of excitement outside of REST + CRUD.
I see this in myself also. Static typing is such a fever at work especially amongst juniors and mids. Sometimes it’s almost as if they believe object types will spontaneously change at runtime, unpredictability.
Try writing something asynchronous in python with futures and complex nested data types. See how long you enjoy being told that you’re trying to look up a key in something that doesn’t seem to be a dictionary before you switch to static typing with mypy, and can spend your energy on solving the real problems.
I’ve been programming for way over 20 years, and I do not understand how people enjoy debugging something that the compiler would just tell them.
YAML interpreting NO (Norway’s country code) as false being the most immediately obvious.
Bash has a lot of these kinds of issues too, I don’t remember all the rules so I quote everything to be safe and still get bitten by variable expansion sometimes.
Types that change depending on what the value is: that scares me.
AMEN to this. I've been in the industry for 20+ years and at this point I know so many different frameworks (front-end and back-end), databases (SQL and NoSQL), networking protocols, browser differences, ORMs, it goes on and on.
I'm paid what I barely consider adequate.
Now I'm doing the same thing with another company with the web, expect I'm handling everything as a sole-engineer, full stack.
Here I'm dealing with creating an Angular application to replace an aging ASP.Net MVC app, having to rewrite hundreds of SOAP services into a proper REST architecture, and dealing with an Oracle database who's schema we need to keep in place. With a C# .Net Core back-end.
This company needs me, badly, but I doubt this is going to be come a cash cow either (no equity, just a paycheck, and their budget is tight).
Just doing my job as usual.
A lot of us have jobs because companies need to fulfill the image. It’s half the reason why so many people are allowed to do full-stack when in reality they would have no business dealing with those parts of the stack in a real operation.
So no, you don’t actually deserve more money because you are working on more things in an inconsequential space (e.g What the entire Peloton engineering team does is probably bullshit. You need maybe a few devs).
There are certainly better examples than Peloton, but that’s what came to mind (a home gym tech (lol) company). Can’t wait until Bowflex starts hiring out web developers.
I can develop a nice relational database design, write the SQL stored procedures to manipulate it, write a backend API and write the front end SPA for it. I don't think I'm an expert in any of those things though, and if I am then it's more focused on the backend API stuff.
Like, I understand CSS better than most people, but I'm not a guru like some. I can write SQL for anything I need but I couldn't tell you anything about performance tuning my SQL outside of seeks vs scans and the size of a lock.
Understanding the basics of these things isn't really a daunting task and that's why full stack devs aren't paid half a mil a year. But you have to accept at some point that if you're full stack then you're rarely going to be considered an expert in any field. That's not bad though having rounded knowledge is super good.
Companies that underpay fullstack web developers usually don't actually care if their engineers __know__ all that, either. They just want the cheapest, fastest, CRUD app they can demo and ship out ASAP. As a result, these employers fail to recognize (and reward!) the fullstack web developers who do actually __know__ the full stack.
At my last job there was a rockstar web developer engineer who could easily double their compensation by moving to a larger company instead of a startup. My advocacy for them to get a raise or promotion was cast as 'the engineer is complaining again."
Then just assume the front-end will consume it and output a little, inconsequential DOM element here or there.
The way for full-stack devs to profit most is to work at smaller companies and trade that for equity and seniority.
> There's not enough women in technology. What a fucked up industry. That needs to change. I've been trying to be more encouraging and helpful to the women engineers in our org, but I don't know what else to do.
> Same with black engineers. What the hell?
I'll start. I have experienced multiple occasions of men being derisive towards women in the workplace, and have been nowhere near forceful enough in shutting it down. I have offered support after the fact, but I need to become more confident at voicing my displeasure in the moment. This applies to all of my male co-workers, and I would bet nearly all of you as well.
This is gold. I think the whole industry is failing to value an engineer. For 5+ yoe, I think performance is a function of eagerness to code and innate intelligence. Yet the industry is paying engineers largely on function of yoe
All the best outages have unclear blame.
This is the saddest. One more soul taken by the shrinking of the hacker culture.
Another thought that has occurred to me of late: when I started in this field it was "Computers" (capital "C") — a thing really by nerds, for nerds. Increasingly it's the web, mobile. Our "customers" are increasingly not us and so the decisions that would have come so readily to us on how to proceed, what features to implement are instead handed down to us from design, marketing....
I get sick and tired of all the rest of the bullshit, but never with tech and programming itself.
The love and passion are still there, but their buried under the needs of my current mediocre job. I need to find a position that will let be get back to the task of building things again, rather than just tweaking scripts and keeping the machinery humming.
It just doesn’t have much to do with computers and technology in the specific ways that I got enamored with computers and technology. Playing with bare metal stuff at work does. Of course, for someone else it might be the opposite.
OK, you're drunk.
It seems that the entire stack of people in software development is tasked with identifying problems and subdividing them in to smaller problems. This goes for coders structuring lines, architects structuring modules as well as managers structuring teams of architects and coders.
The quality of a software engineer shines through all these layers.
True. You do not know how it is true!
In my experience it’s because they either don’t really know any of the stack well enough to do more than implement someone else’s design, or else they don’t know most of the stack well enough to do even that without very poor, confusing implementations.
I’ll take a backend person and a backend person willing to do front end work before a full stack “dev” any day of the week. And I’d take the full stack dev before a front end one.
So you think one person can't know the entire stack but also half of the stack is not complex/important enough to be worth specializing in?
Exactly the opposite for me. I just can't stand hovering a variable or a parameter and not getting its exact type, or typing "." after a variable and not having my editor gives me all the available methods on that variable, or running my code just to discover that it instantly crashes because I made a typo or forgot an argument or passed the wrong argument or tried to call a method that doesn't exist on that variable or whatever other issues that happens only with dynamic languages. What a waste of my time.
Early in your career, you lean into one or the other. And then years later, after you're confident you're right, you find yourself trying the opposite paradigm and liking things about it.
Both have pros and cons, and if there was a correct answer we'd all just go with that one!
Yeah, same here. I started my career as a huge dynamic languages fan, and Python was my favourite language for over a decade.
But now, after 20 years, I appreciate a static language with proper IDE support and code completion. Offload the work to the computer, that's what we do for a living after all.
However, after spending a year working in Rust, I think this can be taken too far. The safety guarantees in Rust are amazing, but the overhead for contorting programs to a form the borrow checker will accept, and the mental overhead related to async/await compared to goroutines is too much.
My favourite language is now Go, and I find it strikes a good balance between static checks and productivity. Rust is still a more elegant language in many ways with things like generics and iterators and their enum types (algebraic types I think is the term?) and zero-overhead abstractions and clean error handling. Go feels a little hacky by comparison. But it's simple and way more productive for me personally, so I prefer it.
Interestingly Evan Wallace (constexpr here on HN) implemented esbuild in Rust initially, and switched to Go and stayed with it for much the same reasons, but also noted that the Go version performed better: https://news.ycombinator.com/item?id=22336284
> But at a high-level, Go was much more enjoyable to work with. This is a side project and it has to be fun for me to work on it. The Rust version was actively un-fun for me, both because of all of the workarounds that got in the way and because of the extremely slow compile times.
After a year of working with Rust and switching back to Go, I second this. I'm enjoying programming again and finding it easier to put in long hours.
Typing a '.' and seeing the members is very nice, except when the type is not concrete and it's not clear what type is actually being returned. Then you'd have to do trial-and-error using a slow compile cycle. In a dynamic language like lisp, you could just `(inspect x)`. In Python, you can just `embed()` and run, e.g., `x.__dict__`.
The IDE telling you about syntax errors and non-existent functions etc is very nice, except when you use macros and meta-programming and now your stupid IDE won't just shut up (I have this problem even with Python in VSCode).
That, and TypeScript generics can get so freaking complex that the code does NOT become simple to read at alllllll. It's a massive waste of time.
Also read: https://medium.com/javascript-scene/the-typescript-tax-132ff...
There are some problems where you don't want any barriers to getting your idea out of your head.
No rules are best for new exploratory code.
Types and structure are good for existing code.
I wrote Ada years ago when it was newish, and it was hard.
But working on existing Ada code is wonderful.
The main downside for me of the static typing is that it's close to impossible to provide a good testing experience, DSLs, mocks and spy objects kind of require some form of dynamic typing to be usable.
hehe, right
That's ... that's not at all how that works.
OTOH, my experience with same-company recruiters was that they merely existed to act as a simple filter and PR person. They didn't invest anything in me (the candidate) nor really examine if I was a good fit before I got passed further on up the chain.
Now that I really think about it, the quoted advice is pretty much a mirror opposite to my experience.
Yes, something is wrong. But it could be many things. Here is how you can find out:
1. Is the thing that's broken a bug in your code? Then it's your fault, so fix it. This means you need better testing too, and maybe a redesign to resist failures. Try to get alerts at 9am in dev so that they don't come in at 2am from production.
2. Is the thing that's broken a server thing that Ops is supposed to deal with? Probably you should quit. You can also work with Ops to redesign the server stuff to be something less prone to failure. Often Ops can't do this themselves because they don't know enough about how your apps work. Go talk to them, help them out. Or quit.
3. Is the thing that's broken a false alarm, or not important? Quit. Or work with Ops to create better alarms and tests. Ops doesn't know your app, so you need to help them craft the SLIs and SLOs.
4. Did Ops create all these alerts themselves without your involvement? Quit. Or take ownership of the tests and alerts for your apps.
5. Is it a huge slog to try to figure out how the alerting works, to work with Ops to make changes, to add tests, or to figure out what's broken or not and troubleshoot it? Definitely quit.