Ask HN: What do top engineers you know do that others don't?
Just wanted to get into the top 0.01% axons. :>
#1 Rapidly Climb Learning Curves
The ability to quickly learn enough about new subjects to be useful. New technologies or APIs; new algorithms; mathematical or statistical subjects; and most importantly, your problem domain. Some of this ability is a skill, "knowing how to learn", which covers google-fu, reading comprehension, prioritization, time management, etc. Some of this ability comes from having enough aggregate experience that new information just clicks into place like jigsaw puzzle pieces. Some of this ability comes from the confidence of having climbed enough seemingly insurmountable learning curves that this next one is "just another day at the office".
A sign you're doing this wrong: "I need training!"
#2 Understand The Customer
IMHO, the best engineers are half product managers. You can easily get a 10X improvement in productivity by building the right features and not building the wrong ones. Yes, professional product managers are great, but technical limitations often impact product design. A great engineer knows when to push back or suggest alternatives. This requires empathy not only for the customer, but for the product management team. This ability is also tightly coupled with #1 - you need to quickly come up to speed on the problem domain, whatever that may be. If you wan't to be a great engineer, don't study algorithms (until you need an algorithm, of course), study the particular business that you're in.
A sign you're doing this wrong: "Whatever, just tell me how you want it and I'll make it that way!"
Absolutely agreed. I think this is part of the source of the endless fighting around "10x engineers". In most discussions where people are trying to "prove" their existence, they will usually bring up things that are outside of the scope of a traditional engineer, that involves some kind of outside/customers/business perspective on a problem/solution.
It's not that it makes you a bad engineer if you can't thrive in an imperfect organization. All organizations are varying levels of imperfect. Great engineers figure out how to be great anyway.
But I was mostly thinking of stuff that can't be found on Google, e.g. the last time I said I needed training was when I was going to do maintenance at some company, on some internal application that I had never seen. I didn't know the domain either. But I had heard that a two-day training course for users existed (also given by internal people). So I said I wanted that training (developers before me hadn't), people thought that was a good idea and it turned out to be very helpful.
Then some manager tells you, "we need Kafka, can you provide our solution for that early next week?"
THEN, you say "I need training!" (necessitating relief from other duties).
Some of the best ones I know dabble in things like art and music, and do it better than most of the people who are full time specialists. It reminds me sort of like watching sports, where a world class defender can attack better than a second division forward. Maybe they just have really outstanding soft skills.
Is there an implied caveat that this applies mostly to Frontend engineers? Or do you think this is something Backend engineers should also be good at? Or I suppose there is third option that great engineers should do both.
I think it's part of the learning process, where cross-discipline helps them learn a lot faster. Or maybe it's the attitude, where they work to learn as many things as possible rather than optimize for some high paying specialist role.
I find the pedantry of learning how to do the same thing over, and over and over... to be... pedantic?
* Read the source of upstream dependencies. Fix or fork them if needed.
* They're better at finding forks with solutions and gleaning hints from semi-related issues.
* Formulate more creative hypothesis when obvious lines of investigation run out. The best don't give up.
* Dig in to problems with more angles of investigation.
* Have more tools in their toolbelt for debugging like adding logging, monkey-patching, swapping parts out, crippling areas to rule things out, binary search of affected code areas.
* Consider the business.
* Consider user-behavior.
* Assume hostile users (security-wise).
* Understand that the UI is not a security layer. Anything you can do with PostMan your backend should handle.
* Whitelist style-security over blacklist style.
* See eventual problems implied by various solutions.
* "The Math."
Ah, a personal favorite of mine which often makes other people uncomfortable.
1. Users are like Marines, if it can be broken, it will be broken. If it's use is not clearly marked, it will be used incorrectly. If it can be fucked (literally or figuratively), it will be fucked.
2. Users are malicious and are actively trying to backdoor everything you do. Why? Because it's the first thing I do with a new product / what I do with the competitions products.
3. It is the developers responsibility to mitigate the above cases.
In my experience clear markings aren't enough to stop it being used incorrectly.
At the end of the thirty minutes one bowling ball was in a bunch of little pieces scattered all over the room and the other one was pregnant.
Can you elaborate?
A mid-skill engineer might start by trying to diagnose the problem, and usually will get somewhere, but occasionally they'll run out of ideas. They might assume it has to be one of the things they've seen before, but their list is not exhaustive.
A top engineer will start by looking at the big picture and generating hypothesis. They will usually be able to generate more hypothesis and have more ways to test them than less seasoned folks. Often this is because they've seen more things.
For example, off the top of my head, the slowness could be:
- n+1 query problems
- slow remote service calls
- streaming to slow clients
- object allocations
- loops in loops--bad big O
- resource limitation (memory, CPU, IOPS)
- database connection limits
- worker process/thread limit
- frequent cache misses
- noisy neighbor problem
- lack of indexing and data size growth
A top engineer will run through their own list, and for each, they have some ideas on how to check on that item or to rule out what is unlikely. Some of them are trickier than others. The key is they have more angles that they'll know off-hand to follow up on.
Can someone expand on this and explain this? I'm just starting to build my first API, and it's not clear.
The point here is great engineers make sure the API answers correctly even if the query is messed up.
The simplest example is probably using a Big O approach to view how to walk a large data structure. The math works out that certain approaches to walking it are cheap enough in terms of total operations at small scales, and incredibly expensive at large scales. By doing the math, I mean that you can both know when you can get away with sloppy approach, and when you have to be careful to not touch each object more than once.
I didn't just mean big O, though. You might do calculations like "if every user creates these structures in a database, every 5 minutes, we can expect that this table will have n records by time y. If that happens, one, is the size tractable, does it need an index, and do we want to reconsider this approach in favor of a cleverer design that can avoid this query? Things like that.
A recent example for me was researching max zookeeper writes per second and deciding if it can be used for super high volume writes in a system. The conclusion, based on some napkin math of the system needs, and some benchmarking of ZooKeeper was that a different tool needs to fill that spot in the architecture because it was too write-heavy for zookeeper.
I don't care how good you are, if your personality is hostile and toxic, you're not making the team productive. If you can't take honest feedback during peer review, or QA or even from the client, then you need to evaluate why.
The best engineers I know are humble. They don't freak out when you point out a bug, they look into it and figure it out, then they share with you what they found vs what you found in order to determine if it is indeed a bug. Then there's developers that think all their code is perfect and sacred. Nobody likes working with those kind of developers.
- Let people learn from mistakes by making sure they can be backed out easily. Do more prototyping, etc.
- Figure out how to make mistakes more visible to everyone, not just you.
These are also helpful when you are the one making the mistake.
I've met people who literally believe humility to be a weakness, I think a lot of people misunderstand what it means to be humble these days.
It is another thing to be a legitimate expert and to not voice your opinion to avoid conflict. I have seen smart people who avoid direct conflict at all costs. Even if that means working on something you know is a bad idea.
It opens up two new good paths: 1) the other person actually having important perspective you don't, and you learning that in a civil way, and 2) the other person recognizing the problem, adapting the plan accordingly, and still feeling ownership over the plan.
It also opens up a new bad path: 3) the other person being ashamed of not having this perspective and digging in on the bad plan.
But if you're seeing #3 regularly, that's a job you need to quit ASAP.
2. Are willing to help with getting proper requirements
3. Don't need a JIRA task for everything
4. Don't say they are done if something is untestable
5. Are willing to do stuff other than their skill (eg. one of the graphics required for the project is too big, top engineer opens up gimp, resizes and continue. Bad engineer will report to manager that design team did shitty job, reassign JIRA ticket, write two emails and wait for new a graphic)
6. Top programmers deliver well packed, documented software, keep repository clean with easy setup steps accessible for everyone.
7. Top engineers enjoy what they do, and are making the project enjoyable for everyone, keep high morales and claim responsibility
bad code should be called out. It's reasonable to feel frustrated by bad to terrible decisions that your predecessors made and left you to clean up the consequences of.
when a group of people is presented with a shitty situation verbally commiserating with your co-sufferers is a form of group therapy. Yes, it has to be tempered with hope and humor, but... bitching is good. Cynicism actually helps ward off burnout (barely, but it's one thing that contributes to warding it off).
This! One hundred percent this!
However, sometimes the sign of a senior is the ability to distinguish when something isn't a fire and should go through the proper channels... so maybe I'd also add:
8. Knows when a JIRA task should be necessary before starting work.
This kind of thing can be dangerous in the wrong sort of organization, be careful about stepping on other people's toes. What you call Bad engineering is adaptive behavior in an organization like this.
Bad organizations incentivize bad behavior and punish good behavior.
Genuinely asking: how so? Tickets are not just some bit of bureaucracy, they are also a living log of what has been done and why that thing was done.
And as bonus points it can effectively prevent people from gobbling up only the work they want to do and leaving the shit for everyone else.
If you try for an hour and it's still not done -- make a ticket.
There's nothing more annoying than seeing the same trivial task get shuffled around 4 sprint planning meetings, taking mental overhead from 8 people.
And nothing comes out of sprint planning with less than a quarter day allocated. You've turned a 20-minute task into a whole day of wasted effort.
Sure, but that doesn't mean you can't create a ticket for it. It just means that you have tickets which fall outside your sprint planning flow.
There's no need for a ticket for small tasks, unless you are being evaluated on number of tickets closed (which is a separate problem)
You can create tickets for every small thing, but it might not be very efficient.
The devs on my team are encouraged to move tickets around and plan their time however they want, but JIRA allows our product, design, dev, and qa teams to coordinate without being in the same room every day. It's just another form of async communication that helps us make sure we're shipping everything we promised, and fixing all critical bugs before the end of the week.
For us tickets are most important to track what gets included in releases and patches, and to tell QA to test our branches (and we almost always require QA to test our branches before they're merged).
Huh... you've never had 10 things that are worth doing that take less than an hour? Or did you just line them up and work 10 hours that day?
You need some way to prioritize and work against those priorities, typically measuring impact or some variation of ROI.
Bitching about legacy software is often necessary if you want something to change: pay off technical debt, additional time, migrate third party software to a newer version.
The difference, which I hope the original author was alluding to, is that experienced dev accepts that legacy software is a part of life that has to be accepted, and dealt with, and that fixing it / keeping it on life support is frequently the right answer. Whereas junior's see it and run screaming or say there's no hope and it needs to be rewritten from scratch even though they don't really know the details of what "correct" behavior is.
I hate this sentence so much. "Bad engineer"? Being a good or bad engineer doesn't come down to a single thing, ever.
A lot of organisations, like mine, which is a well known public entity, REQUIRE this sort of operational procedure.
In fact I can think of at least 10 sites in the alexa top 100 that don't have any tests.
For eg. One feature require backend & frontend work, after a week of work engineer says that this is done. So you ask him to demo, and it turns out he is doing a backend demo with some dev tools.
From a product perspective nothing has changed. Top engineers understand, that no one cares that only one part is working, they know the end user and will not bother managers/product owners with partial solutions.
Is this common outside of organizations where people are told to not work on anything that is not a JIRA task? And do people that do that not come mostly from such organizations?
From the book, things experts do more/better/faster/etc than novices.
* Identify patterns faster and successfully predict future events more often.
* Recognize anomalies - especially negative anomalies i.e. something didn't happen that should - quickly and take appropriate actions.
* Identify leverage points within their architecture to solve new problems and deliver new features faster and with less effort.
* Make finer discriminations of events and data, at a level of detail novices don't consider.
* Understand the tradeoffs and consequences of an option.
* (I like this one) Recognize expertise in others and defer as many decisions as possible to that expertise.
* Their ability to "context switch" when describing a situation to other experts vs novices vs non-particpants.
And one that's not explicitly from the book but is contained in its wisdom:
* Skate where the puck is going, not where it is.
If you get lost, and no longer know why something is not working, do not just keep fiddling and changing things.
Simplify the problem. Disable all confounding variables and observe your changes. Open up a repl and try to reproduce the issue in your repl.
Read the source code of your dependencies. I have seen this a lot: People fiddle with dependencies trying to get them to work. Crack the code open and read it.
2. Choose your battles. Not every hill can be the one you die on. You cannot control every part of a code-base when you are working on a team. People are going to move your cheese and you need to learn to not let that affect you.
3. Learn to lose. Similar to the last one. Treat technical discussions as discussions, not a competition. Use neutral language that frames your ideas as NOT your ideas, but just other options. Keep an open mind and let the best idea win.
4. Write tests. There are outliers here, but the majority of talented engineers I have worked with are all on the same page: If you don't have tests, you cannot safely refactor your code[0]. If you cannot safely refactor your code, you cannot improve your codebase. If you cannot improve your codebase, it turns to mush.
5. Simplicity is golden. Keep your projects simple, doing the bare minimum of what you need, and do not refer to your crystal ball for what you might need later. Single responsibility principle. Keep your Modules and your functions simple and small, and combine them to create more complicated behaviour.
6. Quit shitty jobs. If you are not learning at a job, or they are abusing you, you need to get the hell out of there. Burn-out is real. Burn out on something cool that helps YOU, not pointless toil for some corporate overlord.
0: Martin Fowler's Refactoring 2nd edition
* Multidisciplinary
* Quantitative
* Scientific
* Curious, skeptical
* Thorough
* Willing to abandon a bad idea
* Willing to advocate a junior colleague's good idea
As an add-on question: Which of the properties mentioned in this thread do your organization actively drive out of people?
They are strategically lazy, putting a lot of thought into how to simplify at all levels. Great engineers loathe complexity and indulge in it reluctantly. Solutions should never be more complex than what the problem domain demands.
A corollary to above: they use language features and constructs to solve problems, not to show off how smart they are by constructing the most "clever" bit of language gymnastics with which to waste the time of those who have to maintain the code later (including the author!).
They know assembly language and the basics of CPU architecture regardless of what language they use so they understand what is actually happening. They also have a grasp of other aspects of the system like networking and storage even if they do not do much with those directly.
They know the history of computers and computing and how things have been done at various points in the past. This helps them spot fads and rediscoveries of old things that have been tried already as well as generally deepening understanding.
They are skeptical of fads and don't instantly adopt whatever thing is trendy unless it's a genuine improvement.
* Reads the docs of dependencies rather than blindly googling.
2. Take code as documentation. This helps to debug things faster
3. Focus more on problem solving than language/tool priorities
4. Listens more and always towards exploring and experimenting new things. This improves breadth knowledge
Architecture is important, but organizations employing "software architects" tend to be bad at software.
The worst thing I remember is a web API where some call could fail but didn't tell you, it just gave you some kind of plausible looking inert data. The call to query system status was separate, so there was always a time of check / time of use problem. Also there was a transition period when the original call did return an error but the system status API didn't yet. Nobody (I hope) comes up with such a disaster while implementing and testing it.
Time spent getting it right the first time saves significantly more time than dealing with the fallout of not doing the right thing.
They have no long-term perspective about the future cost of support - fixing bugs, adding features, or just understanding what was done.
When something changes and their snowflake solution breaks, their first reaction is to blame the external change for causing their poor solution to fail.
Top engineers don't do this. They understand why their solution works, they know its limitations, and their solutions are understandable and supportable. This significantly lowers the total cost over time and is what helps projects deliver on time with quality.
Most work environments train programmers to act that way. I find this is especially the case at workplaces that (sort of) follow scrum and agile. In most of these places, junior to mid-level programmers are given a task to work on that already has an hours estimate on it. Managers like to optimize efficiency and programmer time, so that estimate is always tight. Asking why we're doing this task is rarely received well and the tight time estimate means there's barely time to make the change work in the first place, let alone understand why it works.
We're literally training our devs not to be top devs.
It is important to test the code base.
The central bank system is making every day people poorer as they print new debt. They got wage inflation imported from cheap globalized goods wrong. This part is annoying to know since most do not understand this flow fully. It’s engineering but financial such. New printed money is flowing to automation and are deflationary. Understanding other engineering areas other than your own field and seeing strength/weaknesses in those.
# Prefer un-opinionated framework. This comment is opinionated. From my own experience, opinionated framework only looks good in the first 5 minutes. You always need to implement a more difficult solution to solve problems afterwards.
# Offload your task to your subordinates People can only gain experience by failure. It won't become a serious problem if you can give guidance and review. The major benefit is you can spend more time on anything else.
Or otherwise put, knowing when to do things by the book, and when it's okay not to.
- knowing how to make the solution "the right level of good", as opposed to "academically perfect" or "unmaintainable hack".
- being able to design and implement a "good enough" solution that is still reasonably maintainable (if needed) much better and/or faster than others
Knowing where on the scale to set that mark requires being able to bridge speed of implementation, execution, maintainable, upgradability and the overarching business goals you're building for.
So really a broad range of understanding that only "excellent" senior developers will possess
Huge in the terminal, huge on the business/product side.
Their ability in troubleshooting, problem solving, risk-assessment, is detail oriented, and seeing the forest for the trees. Being keen-sighted, and maintaining a field of view, and depth of field.
Informing, teaching, educating: They have a pervasive means of being able to explain, walk through a problem and/or solution to a verity of audience types. Some of the best at ELI5.
Understands the Intent, over just the labeled end goal. (As basically what the US Military defines Intent as.)
Ability to give and take critique (also sharing with Tact above). This always seems to infer just the negative, criticize/criticism, and the opposite is often overlooked.
Follow-on question: OK, so how does your company's interview process test for these traits?
It's mind-boggling to me how many organizations understand that the most important traits of a great engineer are "soft skills" (how many answers here are about really understanding big-O complexity or pointer math?)... and yet are content to interview candidates with whiteboard algorithms problems.
Interview for greatness, not for having-brushed-up-on-Djikstra's-ness!
* Happy to ask the uncomfortable questions early.
* Will ask technical questions proactively without caring about maintaining face.
* Has good discipline and communicates well.
Be very organized / systematic..
- in their thoughts and how they articulate their points during discussions
- in building their personal knowledge base
- in how they approach problems, be it during solving a customer problem (adding a new feature) or debugging code
Even Though I am convinced that experience (and thus age) contributes to the segregation between engineers and senior engineers, I also think that it is a special frame of mind which enables some people to truly become the top of the crop. Despite my inflated self image and overly optimistic assessment of my own intelligence, and despite the fact that I am convinced I would have little trouble convincing an f500 company to give me the title of senior engineer, I know that I can never attain the 'seniority' my brother and friend already possess.
Having scanned through the responses, I didn't immediately see the specific behaviors I have found in top engineers. First of all, I disagree with the humility trait. I would not call top engineers humble. I would not call them arrogant either. Top engineers have strong opinions, yet are flexible. They have their ideas about best practices, yet are 100% comfortable adopting something else.
Using a stupid example in software development, tabs or spaces. Number of spaces. Top engineers will have their preference, whichever it is. And they have thought about it, deeply. Not just from their perspective, but from the perspective of all engineers, future, present, and past. You may probe as deeply as you want, they will have looked at it from every possible angle and will be able to explain to you in every detail why they prefer the one over the other. And! This is so, so important... And! They will put their preferences immediately to the side if that's just not how things are done on this particular project, in this particular team. I wish I had better words to convey this thought...
Top engineers will work with what they have to get to where they want to go. And, that's not just the technology, it's also the team. Senior engineers in your team will automatically make everything better. Sometimes their value is educating the entire team on best practices, sometimes it is just driving towards success despite the all of the feces.
Sometimes they might come over as arrogant, because they can speak with confidence on certain topics. That's another thing that sets top engineers apart from junior and regular engineers, whenever they speak with confidence, it is because they are in fact confident. And they are only confident when they have analyzed an issue sufficiently deeply and from everyone's perspective that they are comfortable being probed deeply.
When they do not speak with overzealous confidence, it is because they are still learning about the subject. And they will learn and analyze it deeply if that is necessary.
A single inconsistency, or a single counter example, or even a single ambiguity is sufficient for them to reconsider their position completely.
Also, senior engineers will never assume they know anything. They tend to listen attentively whenever anyone is talking. When someone says something 'stupid' they won't assume that the person is stupid, or said something stupid, they will assume they did not understand something and ask for clarification. They tend to not jump to conclusions... If something is ambiguous, they will identify it and ask for clarification. As a result, they will often ask for follow-ups in the future... such as: "Thanks for the information, can I get back to you if I have more questions?" Now that I think of it, I don't think I have ever seen someone whom I consider to be a top engineer not keep that door open....
I hope this helps.
There are a few top engineers that get bored when they're not coding, rather than getting tired after doing it too long. But that's kind of orthogonal to whether they're a top engineer or not. In over 30 years, I've known one person who was both a top engineer and a "code all day and all night" (or at least all evening) person.
tongue in cheek, yes, but with a grain of truth, I'd argue.