A student asked how I keep us innovative – I don't
ntietz.com
ntietz.com
Happy to see this mentioned! By involving everyone, the experienced engineers will usually ask the harder questions and test the assumption of your design and the juniors to learn and re-evaluate assumptions they have about building software. Almost a win-win for everyone.
Even though in these discussions there may be a social aspect where certain individuals are trying that much harder to find disadvantages of a proposed design (many reasons which readers are probably familiar with). However, I think this becomes a net positive to the overall engineering as everyone tries to bring their A-game to these discussions.
Anyhow, my experience has been that collectively reasoning about a design (assuming the team feels comfortable with criticism) will always get results quicker.
I had a boss like that once. He could literally turn everything into shit. You had a good plan for an event room, he strikes everything useful out for "minimalism" and aesthetics reasons. Then they ended up with a room that was both expensive and unusable. No wall sockets pure concrete ceiling ("to show the carrying structure"), a reverb time worse than in a church.
For him it was always looks above literally everything else, no matter if it was about architecture, furniture, tools or graphic design. And he had the habbit of impulsively demand changes after months of planning, as soon as things started to take shape. Horrible.
Socializing the design is good advice.
Seeking out contrarians is not.
It costs you much more to defend particular engineering tradeoffs than it does to raise questions about them. So unless you have a good relationship with your "contrarian" and they will operate in good faith, you'll quickly end up exhausted and dispirited after the anklebiters attack.
Architecture astronauts and other armchair engineers will suck up all your energy with half-baked ideas they aren't even invested in, if you let them.
There are people who somehow manage to ruin even the simplest code reviews with their precious input that ultimately goes nowhere. Such people will absolutely destroy your spirit when you invite them to consider your something substantial. Those guys so everything in their power to get themselves into such position.
I want criticism about the designs. Tell me where you think the product is junk and why. There is not one user but many users you are designing for in a single product. Each has an unique take and requirements. Features to sell it, those that write the checks, and those that use it daily must both be met.
I would argue that if you have this problem, it is fundamental, and your team will never be firing on all cylinders until you fix it. Admit it is not easy if the dysfunction is well set in.
Testing takes time, but we factor it into our schedule.
When new people join the team, they make more mistakes than experienced people, which can be very demoralizing for them. There are a couple of things that I find helpful for this:
1, I warn them during training that they will make a lot of mistakes, but it will get better as they get practice, and I ask them to be patient with themselves. You can only learn one thing at a time, so some of the training won't click right away.
2, I tell them that even those of us who are experienced make mistakes; that's whole reason we have this feedback process. I ask them not to shy away from pointing out issues if they notice anything in anyone else's work, even if that person is much more experienced than them.
3, I make sure they see other people's mistakes before they start making their own; once they are out of the training period, their first tasks are to check other people's work, rather than do their own work. After that, they may be tasked with fixing other people's mistakes (which also helps them understand the sorts of mistakes to watch for in their own work later). Only then do they start their own work.
Or vice versa, depending on what side of the fence you are on. Acknowledging incompetence seems to be taboo everywhere, managers don't want to acknowledge that managers are often incompetent, and neither do workers want to acknowledge workers are often incompetent.
And I say that as someone who thinks this forum tends to dunk on management a bit too much. My point is that workers do want to acknowledge that others workers are incompetent, and likewise for managers and their colleagues, but they are structurally and culturally disinclined to do so primarily from an information gap problem.
We have that with one of the teams internally, I don't know what the technical leadership on that team is actually doing but whatever it is, it's not working and hasn't been for a while now.
I'm talking 6+ months of code not working right that I myself could have implemented better in a single day because it's not hard. I'm talking API 1 calling into API 2, API 2 reports a problem with detailed information (per our standards) and API 1 logs it as "see API 2 logs". Then you go look at the API 2 logs and _it's not there_. I've legitimately had this experience.
multiple of our offshore development teams run _circles_ around this team. The developers are not good so they can't be successful despite the poor leadership and the leadership is not good so they can't fix the problems with the development team.
Bad programmers are an organizational failure. You cannot fix that with tools.
You can make it easier for devs to fall into a pit of success, however. But, it’s fine line between facilitating that versus foisting things on users under the guise of being “opinionated.” The reality is it is often taken to be patronizing, such as views of nextauth on storing local usernames and passwords.
I recently learned this because I was asked this precise question and basically had to drudge up all my two-quarters-ago research that was the answer to this question. My answer to their question was "yes we could do the easy thing, but..." and everything trailing the "but" is a very long string of small points that add up to something that tipped the scale (at least in my and several other folks opinions). But without that laid clearly out they thought I was just over-engineering for the sake of job security or something like that.
Letting people know why is two fold. Not only does it keep them from relearning the 1000 edge cases you discovered in the school of hard knocks, sometimes technology changes and something that was a hard limit no longer is.
This form on "collective design" usually uses a confrontational form of rhetoric, but the exercise is very much collaborative.
Ideally, if as others have suggested, you've already worked through that question and documented it, you can just refer to that documentation amd move along.
If you haven't documented it, it's a good reminder others may have the same question and answering it will either lead the to the same conclusion or other questions you did not think of.
"Could you do the easy and obvious thing?" means the same thing, I don't see how it makes a difference.
This way proposing xyz as a solution is positive not a negative.
fast forward 6 months and suddenly they have a lot of problems due to that shared library being a larger risk than they ever anticipated.
sometimes "I've seen this before" is a legitimate concern.
The example I gave above, I definitely did spent months explaining to them how the risk to a library like that grows unexpectedly over time. How it's a small risk at first but if you don't consider the use of that library over the next few years you don't see where the risk profile ends up.
What's interesting is that I had a similar conversation with a VP recently. What I told him is that sometimes people really just flat out can't see it and in that case you have to "rule by fiat" as it were and issue an edict. If you wait for compromise with someone who just can't see it you'll get nothing done, especially if nothing can ever be agreed upon.
and I say this as someone who is very particular about the battles I'll fight, I'm a fan of compromise but only if everyone in that conversation truly understands all sides of the issue or I think it ultimately doesn't matter (yak shaving).
It's important to make sure that folks who are doing the work, agree upon the work. Building a bridge only works if the engineers are building a bridge, not a suspended walkway of dubious quality because they didn't like the design so some wood and rope is what you got.
You can tell it's getting old because it used Phony Stark as a symbol for innovation, but the underlying message is timeless.
Nothing pisses me off more that to maintain two stacks at the same time - the good and the bad. The bad one was enough.
but if, in contrast, what you want to create is a business
then it's easy to argue you actually DO want to create more problems; partially so you can "solve" them in "the next version" (or something); this way your "customers" (which really are looking more like victims) can have a real motivation to upgrade; to buy the next version, to keep on being "valued customers"
this is why making tech products used to be very different from making consumable goods (e.g. food); right up until all businesses turned into subscriptions businesses and other kinds of rent-like schemes
As a random example, look at SpaceX Starship: a key aspect of its design is the use of a specific stainless steel that has ideal properties under cryogenic temperatures. Similarly, the Space Shuttle external tank used a weird lithium-aluminium alloy. The F35 fighters gain some of their performance from high-performance plastics you've probably never heard of. Etc, etc...
Say we have silicon chips now to build computers, which give us say an iPhone experience.
Tomorrow, I have a new material, which can also be used to deliver an iPhone experience.
From the experiencer's output pov, it is the same innovation. They can't tell the difference between Silicon chip iPhone vs new material iPhone.
You may use any number of materials and in permutations/combinations to get a given output. An artifact is termed innovative, when the "whole system" can do something that people consider particularly exciting/useful/interesting/etc.
I am merely questioning the author's notion of what makes an innovation (improved building material view vs whole system improvement view).
They’ll believe that a computer vision company is a dinosaur because they’re using C++ and a glorified CRUD startup is hip and relevant because it uses Rust. It doesn’t help that the anemic-ness of some domains doesn’t set in until 1-2 years later.
There are no silver bullets and the only way to get an order of magnitude improvement is iterated marginal improvements. That something is at best a marginal improvement shouldn't be taken for a lack of innovation.
Often it's all just out of date.
> "As a principal engineer, how do you make sure your company stays at the forefront of innovation?"
This is most likely about using all the shiny new technologies in a rapidly changing "forefront of innovation". The article simply isn't about building innovative products and services.
Of course, choosing boring technologies is one of the ways that you can help save engineering brain-cycles for the actual innovative core product or service you're building.
In general, there are a few assertions people make:
A. The industry always changes
B. The industry always follows the same trend
Both are incorrect, but rather the truth floats somewhere between the two.
A better question is often "what skills will offer long-term fiscal utility?"
The answer to that utility question exposes the dirty side of the tech industry:
1. is the concept an orthogonal theme from purely Academic or Commercial Marketers? If Yes, than the probability it will still be around in 2 years is vanishingly small.
2. is the role occupied by Senior staff over 35 at more than 6 firms? No, than the probability of redundancy increases exponentially with time, as the skill demand is under artificial supply manipulation due to wage suppression, regulatory capture, and or tax incentives refactored into a subsidy.
3. what is the average ratio of staff with the skill still present at firms over 3 years? if less than 5%, than the roles still fall under churn-and-burn economics, and at some point the stock valuation bump will need ritualistic HR sacrifices. if over 90%, than a roles true value is marginal, and will be bid down in pay over time.
4. An "award for the worlds smartest sucker" is not a certificate that will hold value. Anyone that claims they know what will happen in 5 years is a fool selling something like nonsense vendor certifications, media packages, and or political rhetoric (STEM etc.)
5. Does the skill require physical presence? No, than the law of outsourcing bids down skill value over time due to irrational competitors.
6. Integrity in whatever you choose to do is important regardless of the role. For example, someone using an emotionally charged subject in an 72% LLM generated article will antagonize the wrong people eventually.
7. Copying what others do... often means one will land in second place... or worse.
Good luck out there, =)
Everything is a tradeoff. If you want to switch to something newer, make sure you know the benefits and drawbacks of both systems, and then decide if it's worth the effort and risk.
Innovation can be good, but you have to understand the risk/reward profiles.
I think this is the right thing for personal projects. I think throwaway toy projects should be free to go that route too.
I wanted to understand single page applications a bit better and just decided to do it in Vue. Had never done anything bigger than a bit of web tracking in JS before. So it was quite a fun experience. I still sometimes enjoy using it (it is still running and available, so I like to visit it like sn old friend).
Innovation means inventing novel solutions that solve relevant problems better than what has been known before. This could be creating a new tech, and using it, but it could also be applying existing tech in a novel way.
So you could be innovative using old technology, and you could use new technology, but not be innovative.
I guess the company where the author of the post works is neither innovative, nor do they use new technology, but there's space on the market for companies like that. Principal engineers who work there don't even need to know the meaning of innovation.
And I guarantee if we look at this persons tech stack it’s probably slathered with a bunch of shiny new AWS crud that’s “web scale”.
This is so true.
“Choose boring technology” is also used to post-hoc justify whatever choices you’ve already made. To many, golang is boring and Rust is unproven. To others, both are unproven and C/C++ are boring. Who’s right? Everyone and no-one.
I love this part. This is the validation after proof that you indeed, solved the correct problem. Because if you can't shave away the complexity to the underlying problem and "get to the heart of it" then you didn't solve the problem yet. You merely handled its edge-cases.
Any engineer knows intuitively that the answer when it comes to picking new technologies in your architecture is always: “it depends” but that’s not the point.
When it comes to socializing the design, imo, it’s another ideologically motivated waste of time. In order to poke holes you must know the problem well enough to do so, but that’s not what people do on their “sprint planning”. Instead everyone has their own problems, which they must complete by the end of the sprint, or get a ding in their performance review.
With no accountability in their estimates, there’s no pressure for accuracy. That’s why tests are better than code reviews.
Neither of these two need to be innovative IMHO. Technology should enable us to do something useful. Or translated into business terms, it should be a profitable business as profit is a proxy of how useful a product or service is to people.
Lead engs are about bringing order to a chaotic eng process.
> Have a problem with PostgreSQL? Pop it into a search engine and you'll get an answer right away. But have a problem with a vector DB? Comb the GitHub Issues or Discord and hope that you find an answer
> They often have great ecosystems around them
> They use well-known concepts
PostgreSQL issues are generally significantly harder to debug than vector db in my experience. In general, a lot of times I found documentation and concepts to be significantly better for quite a bit of newer technologies like tailwind, caddy, gin, golang than say springboot or react or oracle DB or C++ or cassandra. Comparing PostgreSQL to graph DB or vector DB is very apples to oranges as one can't replace other.
Telling old is better than new in general is both wrong and harmful to software engineering.
To me this sounds like a stogy old grey-hair being defensive about why its okay to be obsolete. Its not
The most effective organizations I've seen have been the ones who are the most willing to both use "non-standard" technologies and develop their own tools in-house. There's a question of causation—are they more effective because they're open-minded, or do they get away with weird choices because they're more effective?—and, from what I've seen, it's a bit of both. But I've absolutely seen, first-hand, that top-down standardization on enterprise "best practices" is absolutely not a recipe for anything beyond passable mediocrity. If anything is a sign of weak engineering leadership, it's that.
I'm not saying that a team should use different technologies for the sake of using different technologies, I'm saying that strong engineering leadership means having a clear technical vision and culture where the popularity of a tool is very low on the list of considerations when choosing what to use.
...is not what the article is saying. It's saying that you should use boring tech when it is sufficient to get the job done that you need done. "Boring" is not the same as "popular"; indeed, "popular" is often skewed by hype about the latest shiny new toy. "Boring", as it is used in the article, just means "does job X and is proven by much experience to work for that purpose".
> I'm not saying that a team should use different technologies for the sake of using different technologies
And this is the same point the article is making. The article is saying you should only use different technologies to the extent that you have to to get the job done that you need to get done.
Popular: 1. Widely liked or appreciated. 3. Of, representing, or carried on by the people at large.
In the phrase, "use popular tech because it is popular" you should think of the latter definition: carried on by the people at large. Hence the comparison to "Nobody got fired for choosing IBM." Instead some people decide to call this "boring" technology and you have a movement of people trying to make boring choices glamorous that the person you are responding to is lamenting. The article is saying to use popular technology (definition 3 above) because of the implications of "at large".
Pragmatically, the folklore isn't helpful. Choosing boring technology and then building an ad hoc layer on top of it compared to using a cutting edge off the shelf solution for the problem at hand shouldn't be an easy decision based on engineering folklore. The difficulty is often in seeing that your actual problem is well adapted (never perfectly) to a particular novel technical choice and compounding that is the difficulty of knowing about the novel technology and what problem it is actually adapted to solving. Hence the need for technical vision and understanding of the problem.
No, he's conflating "popular" as he uses the term with "boring" as the article uses that term, which, as I explained, is not valid. As the GP uses "popular", I agree that tech should not be chosen because it is "popular"; see my response to tikhonj's follow-up post. But as I noted in that response, the article is not arguing for choosing tech because it is "popular" in that sense.
> The article is saying to use popular technology (definition 3 above) because of the implications of "at large".
No, the article is saying that you should not even care whether tech is "popular" in any of your senses; you should choose tech based on whether it does the job that you need done based on much prior experience. I.e., based on technical criteria, not popularity (in any sense).
The article does imply that tech that does the job you need done based on much prior experience will be "popular" in your sense 3, but that does not mean it is telling you to choose the tech based on popularity.
Not the way the article is using the term "boring", or the way the post I responded to was using the term "popular".
"Boring" means choosing the tech on the basis of it being proven to work by much experence. That is what the article is advocating doing unless the boring tech simply won't do the job you need it to do. The "boring" tech might well also be popular, but that is irrelevant to what the article is saying because the tech is not being chosen for its popularity, it's being chosen for its technical capability.
"Popular" means choosing the tech on the basis of its popularity. That means not even looking at whether the tool meets technical requirements. That is what the post I responded to was describing, and arguing against. But the article is not advocating for doing this, so arguing against it is not arguing against the article at all.
> it's still a popularity contest at the end of the day.
Not if the criterion is technical merit. See above.
No it really doesn't.
In this context it means something more like: we know this approach has been successfully shipped hundreds of times for similar use cases. There may be warts, but they are known, and it will work.
Even if the business they're supporting is doing something genuinely new (and IMO that's rare), most of the software written for that business is almost certainly going to do something similar to a bunch of existing software.
ChatGPT's Vision add-on is able to do things like generate code for a web dashboard from a screenshot of a mockup because nearly every web dashboard is extremely similar except for the branding and labels on the data/elements.
Most developers working for a business are writing that kind of code. It's the dev equivalent of hiring an architect and contractor to build a house. The result may very well be unique overall in some way, but nearly all of it would be based on well-tested designs and components. Most people wouldn't want the contractor to fabricate custom roofing panels out of recycled Russian submarine titanium, because even though it might have a "wow" exotic factor and some tangible benefits, having maintenance done on it would be a nightmare, and there are likely to be surprises no one considered because it was the first house to use them.
Software is even worse in some ways. I've seen complex products built on flash-in-the-pan frameworks of the moment that ended up with problems so close to the foundation that fixing them would require a complete rewrite or forking the (now unsupported) framework and making low-level architectural changes there. At least with a titanium roof, the worst-case scenario is having to replace the entire roof.
There's some audience bias here, because people who read Hacker News are more likely to be working for startups that do something genuinely new, and maybe even writing code that does something genuinely new, but IMO it's still not very common.
There is a whole section called "When (and how) to use innovative new tech" talking about this, and another one called "My framework for choosing technologies" describing the author's approach.
If most of the things you are trying to do were never done, you will never achieve anything, with complete certainty. Not even if you are trying to make art.
A app is not a "new technology". A microprocessor is a technology and the code you run on it is just another pattern on the loom.
It's not useful to get pedantic about the definition of a "new technology." What is your point? Take the example cited in the article about deciding whether to store your data in SQL databases, vector databases, or blockchains. Are you saying that this is a purely aesthetic choice and not a choice that bakes in a lot of other patterns and decisions?
When we list humanities technological inventions a century from now, I don't think we will be listing CockroachDB or similar.
That's exactly what the author is arguing against. Decisions about foundational infrastructure (frameworks, databases, programming languages) have consequences for the organizations that decide to use them. Switching costs can be high. Lock-in of all kinds is real.
> When we list humanities technological inventions a century from now, I don't think we will be listing CockroachDB or similar.
That isn't at all what I or the author of the article was saying.
You could pick another word at some arbitrary level, but I'm not sure it's a particularly useful distinction to draw.