How do you learn that solution curve and their tradeoffs? Swap jobs and hope you see them all in your lifetime? Read articles and hope you pick everything up correctly?
In more detail, the steps go like this.
Find as many solutions to a given problem as you can. Bad engineers run with the first solution that comes to mind, letting confirmation bias drive them.
Evaluate each solution for its costs and benefits. Imagine two steps in the future when the solution is implemented. What pains are there?
Search for creative new solutions that create win-win scenarios. That's riding the solution curve.
Given all viable scenarios, compare the costs and benefits against the quality measures for your specific context. Some projects value speed over precision. Some projects value performance over extensibility. Some solutions are easier to change later than others. This is choosing a specific point on the solution curve that best fits your context.
You can apply that method of problem solving to any problem, large or small. You don't need a ton of experience to practice it.
I’ve repeated it that exercise with junior engineers to great effect. Some catch on over time and start intuitively considering the trade offs of a few reasonable solutions to a problem; some don’t.
I never reflected on what he was doing there; thanks.
The best software engineers know when to solve a problem without using any code at all, like the classic "just do it manually" for a complicated task that is worth more than $x per task.
An intelligent man learns from his own mistakes. A wise man learns from the mistakes of others.
- Working with experts
- Writing and deleting lots of code
- Participating in code reviews
- Reading even more code from others
- Watch & read technical presentations
You can never truly be perfect, you can only approach it. Understanding your limitations is just as important as understanding your capabilities. Depth vs breadth is hard and you only have so much life to go so deep into so many domains.
- Identify and document possible solutions. Think hard about why each solution is good and each decision is bad.
- Within each solution, attempt to find common patterns in that problemspace (when working with Sharded databases, this approach brings these results, etc)
- Try to match your personal and organization requirements against the various patterns you find and the pros/cons you defined previously
During implementation
- Keep a running list of things that seem weird, or things that seem great, or things that turned out to be untrue
- Don't stop implementing, but for each thing you found that wasn't great in step 1, try to find a better way to approach that thing (while moving forward)
Post implementation
- Compile all the notes you made in pre/during into a manageable list of goods/bads.
- Use this to drive A.) iterations of your solution, and B.) future solutions
If you do this enough times, you'll have a pretty decent list of your experiences in a space, and you'll start to notice patterns as you try different things and become exposed to problems, their solutions and their tradeoffs. Eventually you won't even have to look at your previous notes very often, as you'll have built up a pretty decent amount of experience in various problemspaces such that you can predict what goes well/doesn't go well.
Also worth noting, this type of behavior doesn't really have to be applied to software, but will also gain you big points with future employers when interviewing. You'd be surprised to find how many candidates never stopped to reflect on the work they'd done, why it was sub-optimal, or how it could be corrected moving forward. Response like "We did it ____ way because that's how we always did it." Which, from a progress standpoint, is basically a none-answer.
Note: this is obviously just my opinion, and I'm no expert in "Becoming a Master of Stuff", but these are the methods that I have used and seen my peers use (in one form or another) over quite a long time. Sometimes not as obvious (doing these things in your head vs. writing them down), but the shape is always pretty similar. Ultimately I think being reflective is one of the best skills a person can possess (with respect to employment and I guess also relationships). Acting without thinking is reckless, I think.
As they say, pain + reflection = progress. If you don't wanna reflect, good luck enjoying that pain without progress.
Attending retrospectives after things go wrong is a good way for experts to introduce their thought processes to less experienced engineers.
Asking "why?" when senior engineers deliver feedback during design review can help you learn that a trade-off existed when you did not see it.
Only, I didn’t like having two to chose from. Something told me three would be better. So we paired, I wrote mine, then hers, then just made another one up on the spot. Once we agreed they were complete and implementable, I shopped them to the team.
About 2/3rds preferred the made up one. Including me. So that’s what I implemented.
Naturalistic Decision Making - https://en.wikipedia.org/wiki/Naturalistic_decision-making