On Software Tooling
marcel.is
marcel.is
DAMMIT THEY WILL BE MADE TO READ THIS LINK
Specifically, reactionary conservatism about tools and services can be a competitive disadvantage.
In several workplaces I have seen this attitude co-mingle with pride in original code, and result in a counter-productive NIH (Not Invented Here)[0] attitude (reinventing wheels). Macho/heroism makes it worse.
While trend-following can be inefficient, if the engineers are both excited and humble and ready to weigh many trade-offs, this inefficiency might be "cheaper" than reinventing wheels for all problems.
Cue platitude: it takes balance :)
[0]: https://en.wikipedia.org/wiki/Not_invented_here
---------------------------------------------------------
EDIT:
To clarify I'm talking especially about _uninteresting problems,_ for which there may be an obvious off-the-shelf solution, such as when I've seen engineers invent a build system from scratch (and then cost _lots_ over years as that custom thing needs maintenance) ... when their needs could have been met by the language's standard tooling and using its plugin architecture.
Or, when "just use sticky notes and a whiteboard" gets so stubborn that a growing team stays in chaos for lack of organization.
On the other hand, it is quite bad when trend-following causes not just churn (switching tools), but overkill that introduces tons of cost and risk. i.e. distributed systems where you didn't need one. Relevant on this point: You Are Not Google by Ozan Onay https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
I don't think the point is, "no need for so much software tooling" but rather, "software tooling by itself isn't sufficient."
(And that "underlying hard-work, the focus on stuff that matters, and courage to make a decision" are much more significant.)
You do not become a photographer by buying a lot of gear. You train your eye, study, and maybe buy gear eventually when you know why you need it.
You do not become a software developer buy buying a bunch of supercomputers. You study, you start with smallest things, and maybe you buy hardware eventually, or deploy a lot of cloud instances, when you know why you need it.
You do not become a successful company by quickly raising tons of capital. You study, learn about the markets and the needs of your prospective customers, start with building rather simple things, and maybe eventually you take hefty investments, when you know why you need it.
You can go on and on. The old adage is to use the right tool for the job. This begins with understanding the job, and understanding the need of tools. Then you can choose the tools. Starting with tools, especially expensive tools, and believing that the use of these tools will automatically help achieve success, is a mistake.
When you're a telecom, and you have more than 40 million subscribers, a whiteboard may be a bit limiting.
Investing in a tool without first doing the task by hand is wasteful. No I'm not saying you should do it with all 40 million subscribers, just start with getting one or two and a whiteboard, then a few dozen or hundred in an excel sheet.
The idea that your problem is too big for a whiteboard shows that you're trying to solve too much of the problem at once.
It's fair to say you can't process 40 million bits of data on a whiteboard. On the other hand, if you have 40 million bits of data, but can't work out how to answer the question you're asking of it (or in many cases, what question you're trying to ask!), then it's useless.
This is where whiteboards, paper, and other physical solutions really shine. You can make notes for yourself or communicate with those in your shared physical space in a very immediate, speed-of-thought way.
In general, software (even BI software) is limited to doing what it is programmed to do. We necessarily adapt our thinking to the tools we have to express the ideas we have. Dancing about architecture is difficult, as Steve Martin pointed out.
Put another way: Commitment and accountability to a process must precede the tool that implements said process.
They might bring a consultant in, and nod away at his analysis. They may write some memos, talk about it in the manager meetings. But that's it. Nothing changes.
Management defines culture. Culture is a function of personality. Personality rarely changes. That is why leadership matters so much. That is why we have seen companies stuck in a rut for years, but then radically transformed when a new CEO is brought in. Or vice versa, companies that are doing well, and then tank under new leadership.
But hey, if you can talk them into flying a middle-market CIO in to give them a come-to-jesus speech, I'll be happy to take the money.
What I have seen is process is the attempted cure to bad employees or incoherent culture.
Good employees just do the right things.
When someone checks code into your master branch, do you let anyone do it? Do you do peer reviews (of some sort)? Do you first run tests? Those are your process. If you don't record them (via documentation or automation) then they're learned by word-of-mouth and can easily be miscommunicated.
Do you have a process (and this is more critical for larger organizations) for purchasing? A lot of tracking here has to do with legal and regulatory compliance, not just tracking for the sake of tracking (at least, for the company, the things being complied with may be worthless to you). Is this written down anywhere? Is it automated in some fashion? Who would you contact if you need to buy a new <something> for your office? What if you need more office space, how would you go about getting it? How does your office track reimbursement for work travel?
The good SEs that I have worked with knew when to get reviews and when to run what tests. The good SEs tried their best not to harm others and knew when they were exceeding their bounds of authority. This was communicated via cultural transfer and mentorship.
The reason process tends to be bad is that tends to be cumbersome, inhuman, and ridged. "The first value in the Agile Manifesto is “Individuals and interactions over processes and tools.""
But processes happen everywhere, and even good developers aren't going to be uniform in what they do or have experienced. This is why I still find myself teaching 20-year programming veterans about make or *nix, because they've never used it in the past. With many of our more recent projects shifting to git (versus SVN or TFS for our historical projects), we've also introduced new workflows that people aren't used to. We can either have dozens of inconistently styled branches on the main repo, or we can establish a protocol/workflow. Once it's established we have to communicate it. The choice is: by "culture", which is loose, ad hoc, and often ineffective as people move in and out of teams; by process, document what that workflow is.
The follow up to that: If that process becomes Process (seemingly fixed for all time), you've lost. You should reexamine your processes as time goes on to make sure they're still correct, or that they actually accomplish what you wanted. Maybe a part of the workflow was to deal with a particular technological limitation that's since been removed, then change that workflow. But letting it go by word-of-mouth is not going to scale beyond small teams and small projects.
The only thing I would say is that the word of mouth can work quite well still even in large spaces due to hierarchy. If someone on your team doesnt know how to setup a repo and there is no process It is likely the leads responsibility to convey this information.
Everyone follows a process - the only difference is that if you write it down then you have a hope of everyone following a similar process. If you don't write it down then everyone just follows their own slightly different process.
This doesn't even touch on why process is good for scaling a company. It's easy to have everyone work to a standard process that isn't written down when there's five developers that sit in a room together. But at a certain point you're going to have to hire new employees and you'll need to communicate to them what the expectations are. You'll eventually have to hire "cogs" who aren't rockstars who just know what to do. You'll have people working in different offices and across timezones. Again, these people will all follow A process. By writing it down you can help ensure that they follow the process you want them to follow.
So even still why not simply just use process despite its drawbacks of rigidity since it will still at least solve your problems.
The answer is that just like corruption no finite set of laws will ever be enough to curb an underlying behavior. This is why things like mandatory code reviews are pointless unless people take them seriously
Whereas without process what basis would you have for declaring person A out of band?
People, process, technology. They are 3 pillars. You cannot build without all 3 working in sync.
Good employees can only do the right things with the right technologies - which may be a whiteboard and a notepad - and the right processes so that they know what the "right thing" is.
My company has good people. We have defined programs in place to make sure we have good people, or that the people we have are in the right seats and are engaged in their work. When we have toxic players who cannot be happy and engaged no matter where we put them, we move them out.
But we also had rapid growth, and so with good people came a scattered process - every project team did things differently. It has been steadily improving, and we are building the tools to make the process as effortless as possible. But before we did that, we spent months mapping out existing processes and standardizing them. It was painful. But it's also now fueling growth. More work gets done with the same amount of effort.
Management here tried to throw tools at it early on. It's the natural reaction. But thankfully, they learned quickly. We have a good management team.
First of all, lets define good people: Happy employees, who are engaged with their work, have the skills and the tools to do their job, and have a management team invested in their success. These are not clichés.
Very broken companies have bad people and bad process controls, or no documented process at all. If such a company is small, it may last for a while, but it is inevitably doomed.
Slightly better companies have bad people and good processes. By that, I mean they have reasonable process controls and accountability, but no program in place to ensure that they have the right butts in the right seats, and nothing in place to keep their employees happy and engaged. They are proactive as regards to process, but reactive as regards to people. This is probably the center of the bell curve.
Better companies have programs and hiring practices to ensure that they have and keep good people. But they may not yet have well-defined and enforced processes. If the company is small, they will probably do well. Process controls often come with growth.
Better still are companies have good people and well defined and enforced processes. But they don't have a culture of continuous improvement which is always looking for ways to improve, be more efficient, etc.
But the best companies of all have good people, and good process, but are constantly working to keep their people happy and engaged, and are constantly optimizing their processes to do more work for the same effort. Such companies are not exactly unicorns, but they are not common either.
Important companies don't do data science with pen and paper.
Important data-science teams don't work on whiteboards.
Important studies aren't presented in notepad.
What counts is perception, the focus on image, and the courage to spend big to prove value.
Maybe one way to think about it: some tools try to model and streamline the things you're already doing, and some tools expand the set of things you're able to do. I think the first type is the one to be more skeptical of. Face-to-face conversations, whiteboards, and Excel sheets are all extremely flexible and powerful. It's possible for tools to improve on these, but it's a high bar.
A ticketing system, for example, is how you create a unified view of what matters and what should be worked on. Otherwise it becomes a game of politics and social cunning down to the IC level which is not how I want to spend 8 hours of my day.