Maximizing Developer Effectiveness
martinfowler.com
martinfowler.com
Open up JIRA and pick up a unit of work to produce today -- gotta stay faithful to those story points!
Somehow, a magic team of spherical devops in a vacuum created an environment where things are just green, predictable, and are never broken.
Another theoretical team of angels from a parallel universe has materialized into this one, updated all the docs, and beamed back into their universe of productivity and effectiveness.
Unit of work defined, I then proceed to produce it for a few hours completely uninterrupted, since if another human being were to make contact, I'd simply explode and format my hard drive, losing all my productivity.
It's break time -- I ingest 2 story points of coffee and join a game of table tennis. The ball bounces back and forth without ever touching the ground. My movements are productive, and value-adding, just like my partner's.
Back to producing units of work, only half a story point left (sike! no such thing as half a story point, and no such thing as "left", as it's a measure of effort, not time. Almost got you there!).
Git push! Automatic CI gains consciousness and validates that my work unit has no chance of causing a user to break our software. It gives me a thumbs up and discards its human shell to fade back into the ones and zeros. A QA engineer cries out in the distance.
A metric of my productivity is logged to the OKR database.
Nothing breaks, and nothing unexpected happens, as the universe is completely predictable.
This summarizes and completes perfectly the ThoughWorker bingo. They are always dreaming about the ultimate factory line for knowledge workers, but they don’t get much done other than blog posts.
Are you saying ThoughtWorks bait and switch their customers? Do you mean without their knowledge? Or are they convincing them to accept dev teams from India?
Me saying goodbye to the bus driver before walking into the office.
Thanks for writing, 'lxe.
Why am I on Hacker News at 1 in the morning on a Friday (Saturday?)
And reward them for doing it.
If your company do "360 degree" evaluations where your peers praise you for your quick answers to their questions, you are effectively incentivised to NOT document and instead be interrupted...
And if their question is not something that can be answered that way, you can improve your documentation so that next time the question is asked, you do not have that problem.
On the wider article, I think the litrature is a poor subsitutde for reality. And honestly I question whether a lot of these agile consultancies actually have the required experience to teach us how to be highly effective as opposed to what we can learn directly from orginsations that are highly effective, but I do believe they have enough experience to tell us that effective development teams invest in development tooling, dev focused user experience, and a dev focused culture.
Who and how you'll build this is left as an excercise for the reader, which is unfortunte because we now have a legion of consultants who've never produced a line of code making a career telling us "culture is hard" and then forcing us into scaled agile frameworks and tooling that are contradictory to the ethos.
Doesn't mean the underlying point is bad though. Engineering teams need some time to work on developer experience instead of just being feature factories; bigger org can hire teams dedicated to helping with this. This is a good thing, but many orgs fail at understanding and allowing the investment.
Dev duty takes care of rollouts, crises, rollbacks, keeping an eye on performance and error monitoring. It's a great approach that works for us and allows every developer to get familiar with rollout process eventually, but I don't know if there are downsides for smaller teams or for projects that are split into several independent services instead of one monolith.
e.g.: "The promises that were made to executive leadership about the latest technology are not coming to fruition quickly enough."
It feels like he then proceeds to talk about how management can get the developers "organized" to make sure they get the job done. Having a developer doing support for the day kinda sorta seems like a nice idea until you realize that "Isn't that what the manager is supposed to do? Understand his team and get what they need/negotiate with other teams so that they can just get their work done?" It's kinda like he's telling managers how to outsource their work to someone else. Pass the buck and live in a fairyland.
I know people who idolize Thought Works, Martin Fowler, Uncle Bob, and others in the "consulting set"-- but it seems like they fill a particular niche, and it isn't the phenomenal tech experts, it is more of the "how management views tech from an academic level and how we can get these interchangeable programmer cogs to get our project done w/ the least amount of effort". They rarely give hard advice, it is more of soft exposure to "new tech" and "here's something that works for some people, you might be that person". "We'd love to sell you on new tech that will be a silver bullet for all your problems"
In any case, I learned some interesting stuff from the article, and I do think it is a valuable read.
The lack of hard advice is great for them because they're essentially never wrong but you'll also never fix the problem with the advice alone. Keeps the gravy train going; if they solved our problems with hard facts in blogs, we wouldn't need them.
That said, the advice itself is still pretty good if you have the chops to follow through, but I honestly don't think you can build the chops by hiring in the consultancy to tell you. In fact, it's actually a bad sign that you leadership both need obvious advice, and think they can enact change by hiring the consultancy.
The reality is, you'd probably have better results changing the leadership.
Asking for leadership from a contractor is bad:
1. They don’t understand what you need or how to get it to you, but they sound like they do, and they look professional.
2. There are no great contracting options for something you can’t do yourself and that there aren’t requirements for.
Fixed contacts may start off looking good but they swap resources out and you end up incomplete or halfassed, overtime and maybe over-budget.
A renewable contract may blow really pretty smoke too and look like a high speed train, but that’s not what was needed, they never finish, or when it’s forced to completion, it’s incomplete or halfassed, overtime and over-budget.
If you don’t take care of your health on your own, you can’t expect a doctor to do that for you. Similarly, don’t expect a contractor to solve all of your team’s development problems, though some can give good advice or assistance.
Steps towards making yourself healthy may include exercise, eating better, adequate sleep, regular checkups, etc., while steps towards fixing your development team’s problems may include raising those to your leadership, own problems, foster trust, facilitate, and change your job if problems are not resolved.
Funnily enough I was a contractor providing leadership and helping orgs for the last 5 years. I did my best to varying degrees of success because the leadership themselves would either let you run with it or not. My 1 man band is very different from thoughtworks though.
Now I am leadership and it's way easier to just make decisions.
But if you don’t understand the requirements, choose to pay someone to figure it out, and they finger-paint your business, you may have been better off without it.
Nothing sells better than crap. By that I mean literally nothing, like you could sell emptiness more feasibly.
It works pretty well, our devs are capable of assessing urgency to either bring it up on Slack, wait for the next stand-up or call to arms if it's something critical.
It improves rapport with stakeholders and other teams; it improves morale, it's predictable who will be interrupted during the week, alleviating that from the rest of the engineers. And as a side-effect it helps to force knowledge sharing over time.
Inquire often is followed with more requests, if each developer serves as the weekly-speaker-of-the-team, will the external partner eventually figure out "John will do anything for us and Steve is mean"?
I have had jobs ranging from entry-level phone tech support to Silicon Valley FAANG (E5 equivalent) to "the ops guy" at a New York startup. Like many here, I've been doing "DevOps" since before the term existed. Every job will have both kinds of day as described in the article. The only thing that matters is the ratio of good to bad, and how much you enjoy your work, your boss, and your coworkers.
Leaving aside this funny idealization and its comic interpretation of the blog post, in real life there are good habits and bad habits that can make a team work more efficiently and less efficiently. I have had first-hand experience with developers from ThoughtWorks, the consulting company behind that blog, and I can tell that they are indeed trying to apply those principles and strategies in their projects.
The fact that they are a consulting company gives them an advantage when trying to apply these principles. They only need to worry about the "small picture" of the project they're consulting on. They don't have to deal with all the "big picture" issues that might affect the employees working within the company they're consulting for (e.g. company politics, technical legacies, etc.) So IMHO perhaps some of these techniques and strategies to become a more effective development team are less applicable if you're working in $BIG_CO rather than in $CONSULTING_CO.
I don't subscribe to this, I think the work is ultimately very similar in creativity (just different skill set), both on high and low level. If something seems mundane, it is either a sign that there is lots of hidden entropy that you failed to capture (and thus risk that the high level understanding is wildly incorrect), or it means that we are doing things conceptually wrong (with more effort than required), but we don't know how (and so there is potential for innovation).
And from that duality is derived the idea, that if only we can perfectly specify the mundane work (for instance through "acceptance criteria"), it's just a matter of getting enough bodies to do it according to some grander plan. From this misconception derives the history of software development methodologies.
So originally, people thought, this duality is the same as in building a house, you have an architect, he does the design, and workers (coders) will build it. So the waterfall was born, where the emphasis was on the plan, and it was also the weak point, because in the real world of SW development, it turns out, the actual implementation influences the plan (so the architect cannot just throw it over the wall, as they say).
Then, from the disappointments, Agile movement was born, pretty much out of the idea that all development work is creative, and it cannot be easily formulated/predicted, so we basically need to iterate quickly and hope for the best. Just treat everything like research.
But, as it happens, people promptly misunderstood this (where are my metrics now??). Instead of understanding that the belief in the duality itself is the problem (a kind of wicked problem, really, because it can manifests in many ways), they looked at superficial recommendations of "Agile practices" - Scrum rituals, "embracing change" and foregoing proper planning, etc. (For me, good summary of this discussion is http://steve-yegge.blogspot.com/2006/09/good-agile-bad-agile... and https://simpixelated.com/two-year-work-retrospective)
One can draw an analogy to this duality in other industries, it's a problem older than software. It seems that with technological and process automation, other industries are actually becoming more like SW development (aka "software eats the world"). So elsewhere, this belief in duality gave rise to Taylorism (https://en.wikipedia.org/wiki/Scientific_management), and the rejection of the duality gave rise to Demming's philosophy (https://en.wikipedia.org/wiki/W._Edwards_Deming) and the related management methods (which were, again, misunderstood, I am sure). There the fundamental notion coming from the duality is that it's the management (or anybody else not doing the actual production job), not the workers themselves, to decide how to effectively organize it.
And still, to this day, some people intuitively cling to this duality, and try to apply Taylorism instead of something like Kaizen or Kanban to SW development (and certainly not only that). So you have JIRA tickets and people beancounting story points..
The upper castes have to be responsible for the work environment, otherwise it degenerates: The best people in the lower castes switch jobs to join a higher caste or a company with a better caste system. Or they unionize, either formally (like nurses) or informally by forming silos. Thus the castes become self fulfilling.
As an aside, one thing that HN has taught me, reinforced by this thread, is that I would not have survived in a high throughput coding environment, and am glad I didn't steer my career in that direction, though I love programming. I don't have the self discipline, and I'd have burned out quickly.
- high degree of trust and emotional safety between team members. The team can safely share feedback and risk sharing our crazy ideas
- high degree of care for the craft. We hold each other accountable to quality
- ships, regularly, to real customers
- little status seeking - goes with emotional safety - few individuals on the team need to be “in charge” or hold arbitrary titles.
- the individuals are more than fairly compensated and the company shows their love for the team in big and small ways
- the team talks to their customer and has a lot of empathy for their problems. They want to be accountable to them
- the team has a lot of empathy for new team members and works hard to make on boarding easier
- some willingness to “get in trouble” with the broader org because you know what you’re doing is right ultimately for the customer
- high self starters: instead of complaining, people feel empowered to solve problems or prototype ideas without a permission structure
- not too much catering to “super stars”. 1-2 heros does not a team make, the senior people make it their job to lift everyone up. The team doesn’t obsess over their high performers.
There’s a lot more emphasis on emotional intelligence and empathy in an effective team than on any specific process.
- teams no longer have a real customer, such as “big rewrites” that will ship in 3 years
- teams accept low quality and slack off. One PR isn’t called out and that gives a permission structure for lower quality
- a bad egg gets on the team the wrecks the feelings of emotional safety. The brilliant narcissist the company feels they need to let his/her abusiveness slide
- we throw new hires “into the deep end” because they have to “take their lumps” like we did
- people get territorial over code because they’re insecure and maybe not great developers.
- the customer is ignored to chase the latest cool new thing
- hiring standards are lowered and anybody with a pulse that walked by the tech section in the bookstore is hired
- good developers stop seeing peers, and realize they do all the work, so leave for a healthier team
- tech illiteracy outside the team trumps tech competency in the team. When the manager says “stop writing unit tests, just get the feature done”
- any rumor “hire cheaper labor” causes your best to flee and your least qualified to jockey for status
- Bureaucrats cannot code, therefore they needlessly churn around and associate themselves with other people's creations. People objecting get called territorial and lose the political battle to the mediocrity.
As you see, it all depends on the exact circumstances. Sometimes your points are correct, sometimes mine.
>good developers stop seeing peers, and realize they do all the work, so leave for a healthier team
These seem like alternative framings of or reactions to the same situation.
>teams no longer have a real customer, such as “big rewrites” that will ship in 3 years
"Real customer" is a luxury. In consumer facing tech, the core stuff was all written years ago, and most teams are responsible for some features of / enhancements to the overall product. UXR can put together surveys and focus groups with people in your treatment group, but you are lucky if they even clearly remember the part that your team changed. Most of their opinions will have nothing to do with you.
It only takes one that gets away with it to start a race to the bottom.
Amen.
A team is only as fast as its slower member.
h/t The Goal, theory of constraints
I disagree. Advice is contradictory, practices and processes are often orthogonal, and tools quite literally don't exist. I say this as someone that's worked in large companies and saw how lengthy not only process feedback loops were (especially developer ↔ product team), but also engineering feedback loops (testing, deployment, dev environment setup, etc.)
And I also say this as someone that works on side-projects by myself. The tooling simply isn't there. Setting up fast engineering feedback loops (multi-tiered deployment, test, dev, local, prod, etc. environments, heck even hot reloading or debugging) is needlessly difficult, or janky, or simply impossible. This goes for just about any language/framework out there: from Java, to Javascript, to Go.
It's not surprising that FAANG often designs their own internal systems that handle this sort of thing (from DevOps, to automated testing, to A/B testing, to deployment). I'm a believer that this is also a space that's ripe for disruption. I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now.
This is one reason why I've loved using Notion. I maintain a few open source repos, and our team's process has grown organically over the past few months. Being able to use a Kanban board for one set of information, then decide that it's just better as a bulleted list; storing meeting notes from quick bullet points up to detailed step-by-step guides on how to fix some issue people had that week...
The low friction to high flexibility combo is great!
Jesus wept. It's sooo fiddly to set up everything. The prod and non-prod environments need dozens of parameters that are all slightly different. Some of these are exposed as resource properties, making them obvious, some are environment variables that are nearly undocumented. Everything is off by default and has to be turned on. Logging. Monitoring. High availability. Secure parameter storage. Staging. On and on....
Oh, you want high availability? Sure, if you pay an extra $3K per month and switch to internal only networking, build Azure DevOps "agent" VMs in a zone-redundant set, and register a bunch of private DNS zones. Good luck automating this. There are template samples available, but they've been broken for years.
Regional disaster recovery? Just duplicate everything! Okay, but not like that! Slightly differently because replication is asymmetric and you may not want active-active. Then you have to layer stuff on top to actually fail over. But not one thing at a time! That's a performance disaster. All or nothing, but that's not an Azure feature, so good luck with that...
This is what I thought the public cloud provides: Some place to upload a ZIP file of code that "just runs" without me having to baby-site the infrastructure.
What's actually provided is a bunch of infrastructure parts that the public cloud vendor will patch for you. Everything else is your problem.
For every ticket they've come back with: It is like that by design, go complain on the feedback forums.
Like the other 10 people that also had the same issue and did complain? Their "feedback" was marked as rejected...
ARM system in particular is deeply broken in so many ways.
All cloud providers could have been a lot better but Azure is worse than average.
(Yes I have to usw Azure a lot)
The implementation on the server-side is a bit messy. Things like read-only properties being mixed in with read-write properties, and internal code-names used for final production systems.
Then the actual client-side tooling is a disaster. The ARM template language is just raw JSON, but isn't case sensitive and includes comments and a bizarre expression language. It's very weakly typed and then even ignores the weak typing that it does have. I've lost count of the times I've been forced to put a number into a string because... reasons.
They're working on a replacement language called BICEP, but IMHO they should have gone down the same path that Pulumi did.
It sounds confusing and contradictory. And it's likely that 1 of those ten things is much better than the others.
However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best.
IOW, just go ahead and implement one of the potentially contradictory improvements that people on HN, for example, have suggested. You will already be way better off than before.
And once you start doing this often enough you will get to the point where you're able to identify the best 2 or 3 of the 10 options presented almost immediately. And the gap then between #3 and #1 is small enough to possibly not even matter.
I can't really get behind this sentiment, since in my experience the worst of those ten things is frequently not only ineffective (which would be bad enough since switching costs are very real), but actively harmful compared to whatever you're doing. Chances are you didn't arrive at your current processes entirely in a vacuum, and it's already a mishmash of ideas from the zeitgeist, filtered to not being the worst on all dimensions by the fact that you're still around to observe it.
You really need to be able to pick out the good ideas from the bad ideas, and that doesn't come readily to everyone.
Or maybe I've become experienced enough to distinguish between the good and the bad ideas without realizing it and don't really remember the earlier days when I had mentors who would help me navigate those decisions.
Because we are always shifting frameworks and approaches, there are few consistent targets for tools like that. What you might call the "Developer Experience" tooling ecosystem, is woefully immature for this fact.
I would way rather use old approaches with mature developer experience tooling than new approaches with constantly breaking environments and pipelines. But we constantly start all over again with new approaches, and never quite make it to the mature tooling phase...
Terraform is trying to solve this but it's still a far cry from "Heroku for infra."
Sometimes I conjure up the inspiration to actually implement an idea that's been on my mind. Then I spend half a day trying to setup an environment that just fucking compiles.
I'll spend several days kicking at an open door before I actually write any code. In most cases I just give up before this point.
My experience has been very different (was able to validate ideas and mvps extremely fast, sometimes even in a day/night - but I am very product driven which I think makes the whole difference (meaning that I care to see if the idea works from an end-user perspective first, the rest comes after and only after the project has grown for a while))
Only mature tools can be complex and still work. But many projects and ecosystems increase complexity much faster than their maturity can set it. I think this loss of balance between maturity and complexity is the root cause of most of the brokenness we experience as developers.
I'm very spike on fast-compiling, statically typed monoliths for orgs of most sizes - especially for product-shaped companies versus infrastructure.
Once you go distributed, so much becomes somuch harder, and the amount of dev effectiveness focused engineers required to keep the effectiveness high becomes prohibitive until you hit a much larger scale. Logging, debugging, scaling, keeping dependencies up to date, A/B testing and feature flags, and perhaps the biggest one: developer environments.
From both scientific studies on the matter and anecdotal observation of the same, I can unequivocally state that the top three priorities are:
#1) The high-level goals of the organisation must be rapidly and clearly passed down the management hierarchy so that everyone can "row in the same direction". Ambiguity about goals, or uncertainty in whether one's work is even worthwhile is an astonishingly effective destroyer of productivity. Never treat employees on a "need to know" basis, feeding them the bare minimum that you think they need.
#2) There is no known method more effective at improving overall product quality than mandatory peer review of all code. Not only does this have the obvious direct benefit, but it also helps cross-train senior staff, bring junior staff up to speed, and helps improve consistency.
#3) There is no known method more effective than checklists at improving the quality of manual processes. Sure, if you plan to do something a hundred times, automate it. But you know what is really easy to turn into automation? A checklist. For non-automated processes, a checklist allows junior staff to confidently produce quality. Checklists are a training aid as much as they are a tool for quality improvement.
Of course, there's a very long tail of things one can do on top of those basics to improve productivity, especially of junior staff. Use an IDE. Use a code linter. Use automatic tests on checkin. Etc, etc... But none of those hold a candle to the above three in terms of raw effectiveness.
To put things in perspective: I'd rather have those three things and no source control than source control and none of those three things. You can meticulously track every version of a piece of garbage and it'll still be garbage...
#1 - on the flip side, you need employees who give a shit about your company mission and your clients. Just like a good organization is eager to share these goals, a good employee is eager to receive them. Whenever someone doesn't care, it's a big indicator to write them off.
#2 - Likewise, people have to give a shit and be critical. I used to run code reviews on my team like this - person A reviews person B's code (together, in person) and I sit behind them, not to judge B's code but to judge A's review. It's shocking to me how people want to give code "the benefit of the doubt" and don't flag things unless they are truly egregious. It doesn't help anyone when people hold back like this.
I’ve seen countless times people treat reviews “an approval” by both author and reviewer and for various reasons. People don’t consider it worthwhile, or they don’t want to offend, or they feel offended.
There is countless amounts of advice out there on how to write good code. But there is precious little on how to review code, and even less on how to write “reviewable” code, which is also a thing.
But if people understand the PR review process as more of a conversation, meant to align people on one direction, PRs are incredibly awesome.
I did manage to track down this New Yorker article: https://www.newyorker.com/magazine/2007/12/10/the-checklist
I know it's not exactly Nature, but the point made stands, and I did track down some papers related to error rates and different methodologies used to reduce them. There's a reason surgeons and pilots both use checklists. It's because a failure of quality in their profession kills people.
Lives don't have to be at risk for quality to matter!
I have a simple style that is easy to fill out:
[.] Commenced high-level task.
[o] Completed sub-task
[.] Incomplete sub-task
[ ] Not yet started
[-] Cancelled or not-applicable
[?] May not need doing at all, or pending advice
[!] Issue or error
[ ] Include command-lines under the tasks:
wget https://test.com/foo
[o] Record output after the tasks if needed -- "Task ID: 929123"
In my experience, it's almost impossible to correctly automate a complex, high-level task given no other information that its English description, such as "build a new web server for a developer". However, if you've built 5-10 such servers manually, diligently recording the tasks in a checklist each time, then the script practically "jumps out at you"...
As a practical example, most engineers will say that to install a new physical server (not including the OS) is about 5-10 steps. I counted 50-100 after developing a checklist!
E.g.: Did you remember to:
Check that the firmware version matches the version of the other servers in the same cluster?
If not, did you update it?
Check the NIC firmware too?
RAID firmware?
System time, date, and time zone?
Secure erase / factory reset the BIOS?
Upload an SSL certificate for the management port?
Register it in DNS?
Register it in the CMDB?
Register it in system-management software?
Burn-in test the memory?
Configure LACP on the NIC?
On the switch too?
I have personally witnessed major issues, even outages, caused by the above being skipped by careless engineers.
"Oops."
It's amazing this fad of microservices still hasn't subsided yet. It is the bane of every company that isn't a FAANG but has management who wants to pretend they might someday have FAANG-level scaling/development problems.
I don't fucking get it. It's like one layer of abstraction on top of folders. Instead of breaking up ownership by folders or whatever bullshit - they decide to breakup ownership by github repo.
I'd love to be sold on it otherwise but that's what I've been told directly as the reason for the move to microservices by many companies and what I've seen at my own. Can't do a good job because your business practices suck? Fuck it - microservices, baby!
For less centralized organizations, I think it can be a useful forcing function.
I believe people see ORMs such as Django's as a way to perform one big query as opposed to many small queries. The big query spans 3 apps and 8 models. You have these things sprinkled throughout the codebase and you end up with a big ball of mud.
This situation is a necessary conceptual step before decoupled micro services and far far easier (because it introduces none of the ops problems of microservices).
But that never seems to happen. I think it’s because codebase discipline rules are harder for management to ‘see’ than API rules.
In fact I would say it’s MORE difficult to prevent circular microservice dependency programmatically than in code (where you can hook into module level imports).
1. One person or group needs to decide on what the rule should be, what the boundaries are and how to enforce those boundaries. This is usually not an easy process, ymmv.
2. After a decision is made, you'll have to decide on how to put the rule in place. Are you going to stop everything and make sure all code paths are working according to the new rule? Are you going to do it incrementally? New features = new rule, old features = common sense? Another variation?
3. You then start running your new rule and maybe you enforce it with some custom static checks, and when that fails you hope your code reviewers will always catch this every time.
4. You still need that one person or group to make sure this actually gets done and to assess if it actually has a positive impact on velocity and reliability.
Or you can put in an implicit "rule" that new code happens in a separate repository, where you don't need to care. A developer that has to do an API call because they simply cannot "import" another module or "query" someone else's data has no choice but to do the right thing - though many opportunities to do a lot of other wrong things too, this is not a panacea. But when we're talking about more than ~5-6 teams, the coordination effort of making sure everyone is aligned on a monolith can be much tougher than to go to multiple services (hopefully not micro).
Microservices have problems but the way they’re often portrayed doesn’t match my experience.
I agree with the sibling commenter who said that a microservices architecture exchanges an organizational expense for a technical expense. For certain organizations, it can be the right call.
Probably some truth to that.
You can have 100 micro services from a single repo, and the services are just different folders.
Microservices help people have bounded context when they are develop and they leak their abstractions less into the next one. Monoliths (admittedly I've only seen them in Python, Ruby and .Net) start reaching into each others pieces and adding to tech debt in subtle ways.
You are of course right, discipline is required in both architectures, but there are different emphasis in both.
- Who’s at the Helm? [0] - Microservices, hard as 1, 2, 3 - You can move it but complexity is still here [1] - Microservices — architecture nihilism in minimalism's clothes [2] - Do I Really Need Kubernetes? [3] - Your team might not need Kubernetes [4]
I doubt that any of those articles could convince higher management. I don't know what a "respected source" actually means. I general I believe that advice from consulting agency and cloud providers should be taken with a grain of salt.
If you drew the same conclusion from the "Who’s at the Helm?" article than I, there is only one way out which is a curated container catalog. VMware's Tanzu Application Catalog [5] is the only product in that space I'm aware of. I have quoted the price tag on HN, I think it was removed since that, so I won't repeat it here again. In any way it was magnitudes higher than the EKS base price which already seen as a showstopper by many small shops. Bottom line is: it is really hard and really expensive to operate containers in a secure manner. Probably too hard and too expensive that it would be a good idea to start with it before you actually need it.
0: https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 1: https://medium.com/@danielpetisme/microservices-hard-as-1-2-... 2: https://vlfig.me/posts/microservices 3: https://thenewstack.io/do-i-really-need-kubernetes/ 4: https://medium.com/faun/your-team-might-not-need-kubernetes-... 5: https://tanzu.vmware.com/application-catalog
Sure, you can run them on the same box, but it's very easy not to, and I imagine that this will absolutely crater performance (at least in the DS/ML stuff I do).
Typically speaking you're adding submillisecond latency per external call, or you've done something wrong.
Something wrong can include incorrectly organizing your system. Two separate services that constantly chat with one another should probably be considered as candidates to merge into one, for example.
Deployment matters a lot. I see people mocking systems like Kubernetes and tools like service meshes, probably because they don't understand this critical and hugely beneficial capability they bring which is to organize communication on the bases of (amongst other things) locality and system topology automatically.
In the end, most people seem to totally miss the real reason that "micro" services (the "micro" part should really be ignored today) are beneficial. Hint: it's not a technical problem being solved. It's a people problem. Service oriented architectures give very clear boundaries around which a development organization can structure in a way that allows parallel delivery.
The biggest problem any growing organization will face is: they don't pay attention is contention (yes like as in concurrency/locks/etc.) across the development team(s).
I have personally witnessed and worked with a 400+ person monolithic development organization. This was essentially a "unicorn" and an exception that proves the rule. The one and only reason this organization was successful was because they had top to bottom extreme coordination managed by a very small group that worked effectively together. Without that tight central orchestration it would've been a nightmare.
Most organizations can't orchestrate a 7 person team effectively, good luck getting 200 to share a monolith well. So you break up the org and the architecture follows, see: Conway's Law. It goes both ways, too. You break up the architecture and the org can follow.
With that in mind, it's extremely important to get your boundaries right and for the love of all that is sane in a daily job, don't break it up too much. So many organizations would triple or quadruple their productivity breaking a monolith into literally 2 parts. If you can do 3-5 that's cool, amazing! Just because you can see the dotted lines between 25 different potential services doesn't mean your organization will tolerate that.
Anyway that was a diatribe.
TL;DR: "micro" services solve PEOPLE problems, moreso than technical ones, and the price you pay is additional architectural and delivery challenges which you must balance well and plan for
Modularisation is an efficient tool to address growing complexity of software. The code is organised in closed subsystems - modules - that work together via communication with each other over defined interfaces: functions, REST endpoints, messaging system.
Modules can be done in monolith by enforcing boundaries and policies: namespaces, packages, agreements on communicating only via "service" classes, isolating database entities, no foreign keys between entities belonging to different modules, etc. It is so called "modular monolith", and I believe it's an efficient way to progress for organisations before jumping on microservices train.
Microservices are the next step of modularisation and isolation. By introducing independent deployment of the modules and isolated runtime for each module (different machine, VM, containers) they enable independent CI/CD for the teams that helps to scale up the organisation and they help with isolating different pieces of software that have different technical requirements (traditional Web API with database calls, stream and batch data processing, etc.), different stability and change frequency.
While bringing a lot of additional concerns, requiring education and discipline, microservices are critical enablers for many companies to further grow their products and scale up the technical department.
Let's be straight, both have pros and cons, but I value the following pros of microservices a lot more than any of the cons:
- Enforces separations that are important for: - Maximum parallelization of work. - Scalability of each service. Never share database. - Reusability of the service from other services.
The trick is to learn how to cut microservices though. Establish an interface that it exposes, like GraphQL or RPC. Don't try to split too fine-grained, but find a balance that works for you.
I promise you, when it comes to scalability, you'll be glad you don't have to scale one massive database just because one part of your application is seeing more activity.
This also allows you to pivot quicker to a different database suddenly, or a different programming language, if requirements change. E.g. Node.js -> Rust if your service is found to be on the hotpath of everything, and you need correctness and performance.
I fundamentally view the difference the same way I would choose Rust over Clojure:
- Monoliths: Rely on discipline of the team to do it right (like Clojure won't catch your dynamic mistakes) - Microservices: The most important parts you need are built-in and enforced by the concept itself (like the Rust compiler guides you towards the correct approach).
- Tech debt is for others to fix. While others cleanup after me, I will be completing my next task and paving my way to promotion.
- If it works, it's good enough. It does not matter if I can't explain why it works.
- Everything is an obstacle. Documentation? obstacle. Coding standards? obstacle. I just want to complete tasks.
- I do exactly what I am asked for. It does not matter if it is insecure, it does not scale, or if the entire system crashes. All of that will be someone else's problem later.
- Code does not have to make sense, as long as it runs it is good enough.
- I worship project management.
B) Sustainable mindset developer:
- The cost of tech debt compounds over time. I should fix tech debt before more code depends on it and becomes more expensive to fix.
- I need to understand how and why things work. Is this really working? Do I really understand what I am doing?
- Reviewing relevant documentation for the technologies I use is a good idea. Making sure I understand and follow coding standards is a good idea. Well crafted coding standards can save work.
- I understand that not all stakeholders are technical. I need to understand the technical implications of what I am asked to do, and push back if necessary.
- Code is written once, read many times. Readability makes everyone more productive.
- I acknowledge that project management documentation and tools are a form of documentation, and as such, it is not a source of truth. When it comes to things to do, the implementation is the source of truth, not tickets.
Bad companies promote A), good companies promote B).
Someone that follows approach B learns more in a shorter amount of time. Because learning (advancing the frontier of your knowledge) happens when you encounter that limit, recognize it and try to move forward. Rather than dismissing it, which is what A) does.
B) always thinks: what is a better way of doing this? A) never does. Therefore, over time, B) becomes more effective than A).
In mind sports it's the same. You learn more in slow games when you take longer to think your moves. Every pro will recommend you to prefer slow games over fast games.
I used to work with a guy who was representative of this dogma. He'd give people a hard time if they didn't comment every line of code, and his own comments would be shit like:
if (country == "Japan) // If the country is Japan
total_cost = price // set total_cost equal to
price
else
total_cost = price + tax // Otherwise, set total_cost to price+tax
People like this heard somewhere that comments are good, and thus good people write comments, and if I am a good person then by gosh I am writing comments. They don't think about utility at all.
A better example of B) in this context would be understanding that IEEE 754 floating point numbers cannot be used for currency, and perhaps adding a comment about that.
Meaning, really understanding what you are doing all the way, and understanding when it works and when it doesn't.
Dear developer colleagues, do not fall for this trap. Hold your managers responsible. Let them make the difficult decisions. Real leadership should lift some burden from the people being lead.
Resource and release conflicts don't appear if you have the right infrastructure.
But things get really messy when your software is distributed in heterogeneous environments. Say you ship a database yourselves. Or a multi-platform app, or a game, or the control software for medical devices, or...
Software development is not just running the next SaaS product.
Higher infrastructure bills for spinning up new hardware is often cheaper than fixing resource conflicts
I sometimes wonder who these articles are written for. Am I supposed to share this with the CTO to show them how poorly the organization is being run? No, obviously not, that'd just fast track my firing. After all - if you're in an environment of psychological safety where you can air these issues then you're probably going to not have these issues very much.
If you're a Leader (big L) in an org, this gives you a way to asses your organization. If you're a leader (small l) it may give you a way to concretize your thinking around problems so you can discuss them better.
>> Am I supposed to share this with the CTO to show them how poorly the organization is being run?
That would be the worst way?
Why don't you pick whichever problems on the list resonate the most, come up with a good case as to why it's a real impediment to your team, ideally quantify its impact, perhaps come up with some solutions, and then go have the conversation?
Ultimately, it's not important whether your org looks good or bad against some checklist, the "meat" is whether you have an opportunity to be more effective against your goals. If so, people generally want to have that conversation. Especially if you go in with a realistic understanding of why things are this way and the tradeoffs.
>> and even then - it will still IPO
Sounds like they know what they're doing then?
I bet there are way more failed businesses that had followed meticulous software practices than there are 'rocketships' that took off despite(?) what may appear to be bad practices.
It sounds like you want to gripe about the problems on HN but not actually try to solve?
And then still have nothing happen because the people above are mostly paying lip service to improvement, but not actually changing their ways.
It’s like needing to expend 200% effort because others are expending only 10%.
Edit: But yeah, I also gripe about it on HN because it’s carthartic ;)
For what it's worth, my strategy has always been to grapple with these situations. Sometimes you get to have impact and learn in the process: eg how do I communicate about this problem in a way that gets traction? If you can do it, you will "level up" in your career.
Alternatively you may learn why things being as they are is a reasonable trade-off for the situation you are in. This may also level you up by helping you think on "what makes sense for the business" rathe than simply "I don't like it" level.
At worst, you'll have a story for your next interview about how you tried and failed to make a big improvement like this. I have a similar story and it resonates really well for senior roles.
This attitude is likely contributing to whatever problems you are encountering, and they are potentially trying to fix.
He claims 5-15 seconds to validate that a local code change works is highly effective.
But in practice, that amount of time makes for one of the most unfriendly environments to be in.
For example having to wait 5-7 seconds for an AWS SAM (Serverless) Lambda function to be built and locally invoked is so much worse than making a code change in a Flask, Rails, Django, Laravel, Node, Phoenix, etc. app and seeing the change as fast as it takes you to focus your browser and reload, or running something in a REPL.
When you do such a thing 100 times in a few hours those 5+ second pauses are deadly for motivation and productivity. It's a constant reminder at how crappy the development experience is and it's also not long enough to do something else while it completes so you're stuck wasting your life away on a tool.
Is anyone really happy when they need to wait 10 seconds to see a change when developing a web app?
I think the web context is important because you can relate to it from past experiences. In 2001 there was a near instant feedback loop with making a change to a PHP page and reloading the browser. Anything less than that 20 years later seems like a direct downgrade, especially considering computers are probably 1,000x faster today.
> notes that a previous feature has been approved by reviewers, she moves it into another branch that kicks off a long nightly E2E test suite that is almost always red, managed by a siloed QA team.
This is not a low effective environment. A low effective environment does not have a large e2e test suite. It does not have a siloed QA team, and people write little to no unit tests, never mind integration tests.
If you are in a "low effective environment", a "siloed QA team" is a godsend. At least someone is responsible solely for quality, as opposed to everyone churning out features as fast as they can. Disposing of the siloed QA team in such environments rarely helps matters.
Relative to the highly effective teams the article is describing, the points on the spectrum you discuss here are “low” and “lower”. Sure, low is better than lower. The article is making the case for “high” though.
Reading through Fowlers "Good Environment"/"Bad Environment" lists... I can use my own critical thinking to see why one is good, or bad. Fowler while he doesn't really get his hands dirty with these problems he is very good at rounding up the current zeitgeist and common sense in ThoughtWorks and publishing it to the world. He's rarely been flat out wrong. (I don't mean this as an appeal to authority, just how I look at this article, and Fowlers publications in general).
1) I encourage people to use JavaSE on the server and hot-deploy to the servers directly with async. non-blocking: http://github.com/tinspin/rupy
2) For clientside: mostly C syntax (compiled with C++ compiler) is your best option and I recently made a in-app debugger for Windows: http://move.rupy.se/file/stack.html (this delivers something much worse than a Java stacktrace but as good as it gets without a VM).
I also hot-deploy the C/C++ app code with a .dll to my .exe and the debugger works for that hot-deployed .dll too!
On linux the .so hot-deploy works as well, the only reason I have not taken the addr2line source to port the in-app debugger is that I know which hardware I'm working on as I only plan to ship linux on ARM! Fight features where you can, in this case limit your porting and exposure to unknown unknowns!
Potentially I can hot-deploy the client .dll/.so over my hot-deployed server pipeline, making the platform a distributed real-time system where you can patch the native code in real-time remotely while users are using your app! Mostly usable for development I guess (¯\_(ツ)_/¯), but still really exiting!
Code design is a huge bottle-neck. I've been a part of 10+ engineering teams that have all the right tools and stuff, everything around the code is really frictionless and runs well (CI, tests etc.).
But the code is so badly written and designed that adding new features needs edits of dozens of files and sometimes even copy-pasting files (because of circular dependencies and singletons everywhere).
I guess existing developers can be effective in this kind of environment but getting new ones to contribute will still be a chore.
Although, even existing ones, when they forget what they did, start getting difficulties creating new functionality.
For example, the biggest issue I've seen is "dependency injection" mindset. It's a mindset that dependency injection library solves the dependency issue and you can just inject anything you need anywhere.
The entangled mess that this creates is horrible. Yeah, all of the objects initialize correctly but not thinking about separation of concern, proper encapsulation or what really needs the whole dependency will create a mess.
I have seen developpers "coding to the test". By that I mean they are modifying a piece of code they do not know well, and assume that if test pass, that must be good. Without understanding that test will never cover all possible inputs/states of the system. This "coding to the test" can appear very fast with a quick feedback loop, making possible to "monkey code" something until it passes. If you do not have careful review, by people knowing the system, this will end badly, with race conditions only appearing in production, integration failing randomly.
The reality is that their customers have slapped a developer title on anyone willing to take their 80k/year job. And then they hire "product managers" who fill out schedules with arbitrary dates. Then when "dates are missed" meeting are added and developer training is added. That makes productivity go down even more. In come the consultants who will fix everything with a change to processes instead of being honest and saying that they need to fire everyone and start over from scratch.
Small and light grey text on a white background really doesn't work for my aging Mk I eyeballs.
I have worked in small teams of highly-effective teams generally. But I am about to embark on a much larger team consisting of a large range of skill levels. It would be nice to lay some effective groundwork from the start that isn't overly cumbersome.
Assuming this is true, you're still fucked, because nobody can force your executives to then force their employees to do X. If the executive doesn't tell people to do it, nobody will. So knowing what makes for good productivity is still worthless unless a handful of chuckleheads at the top actually make it happen. And they won't, for the same reason nobody below them is doing it.
Yes, until they realize how much of a crock it all is, and go back to actually getting work done.
Backstage looks interesting though, so maybe I'll get something good out of the article.
This is not a “highly effective environment”, it’s the bare minimum to have a functional environment. Might have been good advice in the late 90s though.