Systems Engineering, or "why james gets paid"
cohost.org
cohost.org
I think my org is pretty good at the informal SE process described in TFA. We used to be more strict about it, and I definitely remember hearing the term "model-based systems engineering" when I was interviewing, and after I started. Fancy UML generator tools, the works. But we've gradually become more informal and I don't particularly feel like we've lost anything in the process.
One thing we are bad at, though, is onboarding. We have some very basic first-day-type documents, to get newbies started on domain knowledge. But the standards we hold employees to and how our work is done on a daily basis... is pretty much not written down at all! (actually some of it may be, but scattered throughout Wiki pages, probably out of date, and definitely no central document with all the necessary links...)
Incidental to TFA, my org generally considers people (like James) who can do both the software work and the SE work unicorns. I personally started on the SE side with almost no software ability, but gradually clawed my way up to a decent level of technical competency, and it continues to be my competitive advantage. I don't code on a daily basis, but I do fix bugs & work on small features every once in a while. I dream about setting up a crash course to get my colleagues on the same path I took (currently I'll take a few hours here and there to help individuals pick up the skills)
Kind of curious what other peoples' experience is on this point. Is it normal in other orgs for SEs to not also be good coders?
BTW, I like the term "unicorn". I'd never heard it in this context before.
That was a lot of fun for a while, but it was pretty clear that I needed to get in front of projects, rather than behind them, to have any useful impact on new products. Also, I hit a couple of home runs: Creating the architectures for a couple of successful products because nobody else wanted to learn the theory. So I was invited to have a more front-end role.
This led to a new informal definition of "systems engineer." I've become the person who understands how products work in their entirety, when it requires synthesizing knowledge from multiple disciplines. I create and test the theory of operation. This is a good use for physicists in industry, by the way. Physics research, especially in small labs, is multidisciplinary. We don't really have any of our own techniques, so we constantly borrow from others. My graduate research involved optics, electronics, programming, mechanics, and data analysis. In fact, I've dropped the "systems engineer" title, and am now a "scientist."
Today, I'm at a level where I am much more focused on onboarding and development of junior colleagues. I'm also finding and developing talent within the organization for people who might like to have more of a systems role. It is a hard role to fill, because engineers tend to specialize -- perhaps for good reason based on job markets. And if traditional engineers get good enough at coding to do it for money, they get sucked up by the software industry.
How is that an informal definition of it? It's almost identical to the Wikipedia description of it:
> Systems engineering focuses on analyzing and eliciting customer needs and required functionality early in the development cycle, documenting requirements, then proceeding with design synthesis and system validation while considering the complete problem, the system lifecycle.
I see you have no contact info on your bio, but you can find mine in my bio.
This is exactly my experience. I'm SE since i finished my school. I always feel frustrated by the SE global vision. Some projects do allow me to go toward more technical stuff (sysadmin, network, dev, signal processing, CAD...) and it's quite enjoyable.
I've also started with no SW skill, but i've learn and it has been a great help in my daily job. My SE collegues only do the SE part and don't want to deal or see the more technical parts. In the end, i'm always offered to work on new projects because of this ability to switch from global vision to technical stuff, but in my org, this is not really encouraged nor rewarded.
Systems engineering is a huge thing for us, so every single project involves one or more systems engineers. This is a more senior role - the systems engineers usually start out in a specific discipline, and once they have more experience they become systems engineers.
So for us, no, most SE's aren't particularly good coders, but every SE is good in some discipline. We staff projects with systems engineers based on their specific strengths.
Definitely started as a software engineer, but in a safety critical field (avionics), at a company/team that in my opinion has an incredible "systems culture" (Garmin Avionics). During onboarding, we were taught WHY we needed to do so many things for the safety critical process.
After leaving safety critical, and going to much more start-uppy companies, and realizing that SE knowledge was not widespread, and how to put the "easy but good bits" in place quickly, is pretty much how I ended up where I am.
Im a Systems Engineer by title, but have no idea where my place is (its all confusing). I'm basically an 'super IC' on a team - and try to play with other groups my group interacts with... So there is no power to set the design. Basically just answer 'will this work with your part of the system'?
I basically end up just finding issues, POC end-to-end (well outside of my discipline: I'm a HW guy to I'll do a FW to Cloud security thing).. and well, why business does the HW Systems person have suggesting we use PKI for authentication?
For this stuff to work well, you need real leadership. Leadership should not build consensus, take polls, or let structure grow slowly and organically. You know what structures grow organically? Mold. Know what doesn't? Steel. Don't grow mold; build steel. Use the precise amount of material, at the right temperature and pressure, follow the right workflow, and inspect the result, then ship it with confidence. Don't hope it gets done; ensure it.
Leadership needs to force everyone to use the one magic doc, to follow the one magic process, and take training to be able to immediately and effectively work with them, without wandering around wondering how to do what's expected of them. Otherwise your teams/org will forever be slow, inconsistent, and buggy. Systems are a result of humans doing work. The better organized the work, the better organized the system.
systems engineering is much less about the deliverables than about how you collaborate to build them, how you ensure consistency, how you walk that talk.
The pdf is in the public domain, I sent it to a printer, had it bound as hardcover, ribbon bookmarks, I love it.
The role of the systems engineer is to manage all this complexity and construct a sound model of the system(s) in question that can be communicated to others and used in development. Both of the final system and of the many intermediary prototypes and "spikes" (to borrow the XP term) needed to fully understand the system.
Naur's Programming as Theory Building is a useful software-centric take on this topic: https://pages.cs.wisc.edu/~remzi/Naur.pdf (I tried to submit it, but apparently another submission a few months back made it a dupe).
Except for one time, when the United States Army Aviation Safety Office took the reins and corralled all the panicked cats and dead-eyed MBA drones. The integrator that was in charge is still one of my favorite engineers of all time. "The hell this has forty parameters? It's fuel. Am I outta gas? Yes. No. That's the parameter."
tosses paper down table, picks up next paper.
Hammered out a doc system for that guy on the fly, something that everyone could see without paying 50k in license costs, and worked my ass off but felt better than I had in years. That kind of magic doesn't happen often, but you gotta save it in your mental cabinet of treasures when it does.
Funny enough, this also (over-regimented documentation and too many requirements) also happens in small organizations when they don't have the expertise to know doing systems engineering isn't to produce a pile of documents. Unless it's driven by someone from higher up, it's an uphill battle to say that we can get rid of 90% of requirements that's just a repeat of some section from different standards.
What usually happens is that consultants are selling time, not outcomes. So it is in their interest to waste time and minimize risk to themselves.
I’ve had experiences like yours a few times, most recently during the nadir of COVID. It felt great.
Is there a follow-up? This is a crucial piece. You clearly need all of these: current state, plans for the future, past states, past plans.
Fine, sounds nice! A team that can work without a communication overflow, that's wonderful!
A few lines later:
> Requirements should be a living document.
He lost me here.
On one side, he preaches for a "source of truth" that everybody can read and take autonomous decision accordingly; on the other side, he recognizes that the "truth" has the horrible tendency to drift with time. He just misses the elephant in the room: what do you do after you updated the document?
How do you track all the assumptions that every team member has made based on that item that you have just changed? What useful is an accurate document if you can't trust it won't change?
Requirement/specification change is the bane of every complex engineering challenge, and acknowledging it while proposing to ignore it cannot be a solution.
Requirements changing are a reality. In safety critical fields, these changes are noticed by using traceability tools.
I haven't written this far yet: but you should have a process for changing things like system requirements. It should be discussed, reviewed, and approved by the relevant teams. The "downstream" documents, like software or hardware reqs, should be updated as well.
Hard to fit all the details in a single summary, but happy to answer questions
What replaced it? Agile. Wait, don’t throw rotten vegetables—I’m not talking about the overwrought excuse for micromanagement people call “Agile” nowadays, but the truly innovative thinking that was happening in the late 90s and early 2000s. Stuff like the Poppendiecks’ Lean Software Development (based on their experience at 3M) and Cockburn’s Agile Software Development (based on research he originally conducted at IBM). And of course Beck’s Extreme Programming, based in part on his work with Ward Cunningham at Tektronix.
These processes and thinking started with the reality that it’s impossible, given a large, complex system, to correctly specify everything in advance. They recognized that documents are a particularly poor communication medium—one that’s time-consuming to create and consume, and very difficult to do in a way that isn’t misunderstood.
Instead, they focused on people as more important than process, and adaptation as more important than prediction. Traditional SDLCs fell by the wayside, and for good reason: they didn’t work well. Their document-heavy approach resulted in poor visibility for management and software that met the spec but not customer needs.
I talk about this more in my book. The first chapter talks about how and why Agile supplanted traditional processes. It’s free online (along with a lot of other material) here: https://www.jamesshore.com/v2/books/aoad2/what_is_agile
You can choose an accurate document that doesn't change often, or an accurate-ish "dynamic" document that is managed by a single entity, or a document editable by more people. You can't have a perfectly accurate document that's also changing constantly and is managed by multiple parties.
As a side note, thank you for not coming up with silly acronyms for everything.
and in that tracking you also capture the "known deviations", like 'this need isn't fulfilled yet, at all'. thus you document the "state of the union" of the whole thing, which for a living collaborative work is all you can wish for.
This is the idea behind "requirements management" and "requirements tracing". Alas, there are few tools out there doing this with a low barrier to entry, cli like.
here is one I'm aware of: https://github.com/itsallcode/openfasttrace
Having a constant flow of these data can make a pretty accurate "end-product" document.
Perhaps the missing sauce was a good systems engineer advising the bosses? How do successful projects avoid this sort of problem? Is it common? It was really frustrating to be a contributor knowing that the whole thing was a disaster but not knowing if your ideas to make things better were useful or a waste of time, even if you could get anyone to listen.
E.g. ISO/IEC/IEEE 24765:2017 defines systems engineering (SE) as "interdisciplinary approach governing the total technical and managerial effort required to transform a set of stakeholder needs, expectations, and constraints into a solution, and to support that solution throughout its life".
Or - if you prefer - the definition provided by INCOSE (https://www.incose.org/systems-engineering): "Systems Engineering is a transdisciplinary and integrative approach to enable the successful realization, use, and retirement of engineered systems, using systems principles and concepts, and scientific, technological, and management methods."
In either case, fixing bugs would fall under the support aspect of what they listed. The full design, development, deploy stages would also fall under that and may be used in various amounts of formality when dealing with bugs.
There is an engineering design process [1] that anyone can learn about and take some useful ideas from. Obviously adapt what's relevant to your needs and discard the overly process or paperwork heavy details that may not be suitable.
There are SDLCs [2] and other software specific IEEE standards that have been developed [3]. If you've been in tech long enough, you've probably figured these out on your own, or some approximation to them, and you may have even come up with your own name for aspects of it. I've seen "metaprogramming" used a number of times on HN to describe aspects of the planning, analysis, and design stages of engineering design.
There definitely are things to be learned by looking at other fields, and I don't think software can claim absolutely zero overlap with other fields. It's not 100% and there are unique constraints, but it's not zero and throw the baby out with the bathwater so to speak.
[1] https://en.wikipedia.org/wiki/Engineering_design_process
[2] https://en.wikipedia.org/wiki/Systems_development_life_cycle
If the CSS is part of a system or system element, and the system is considered large enough to involve systems engineering, then yes, there is both a development and logistics support system which are analyzed, designed and implemented; fixing the CSS will either be under the development system or the logistics support system; but keep in mind that SE does not replace, but integrate the different disciplines; so it won't be a systems engineer fixing the CSS, but there was a systems engineer who planned and help implement the corresponding life cycle system where the maintenance happens.
Unless you want to prove that the job involves mastery of stilted corporate jargon :)
True, but it describes very well system engineering.
> Unless you want to prove that the job involves mastery of stilted corporate jargon :)
That seems to be the new language. "We do not know what the system does, but look how beautifuly is described."
Thanks to people doing DevOps burning out, it's like an ongoing joke/question as to what 'DevOps' is vs. 'SRE' when in fact it was all work that a 'Systems Engineer' performed prior to someone in middle management working with HR to justify their roles changed my title despite the work not changing a bit during their bi-annual "reorganization".
As someone who is on the hunt for a new place to land, it drives me crazy how the title doesn't match the role description and often when you interview there are times when it doesn't even match the expectations. And the depressing part is that it's difficult to even find "Systems Engineering" jobs unless it's with another industry with different skill sets all together.
But as managers are likely finding out now, most developers don't want to do Systems Engineering as it's a cross-team discipline to pull all the resources together and operate complex systems with greater proficiency (in many different metrics) and maintain service delivery (in multiple other metrics).
What has been interesting is this resurgence in people trying to figure out what "Ops" and "SE" did for developers that they have been thrown into doing and learning how little they actually knew. Partly our own fault, of course, as nobody really wants to know how the sausage is made. But they also didn't envision where our sights were set in the future once we got people doing more self-service resource and code deployment. That just getting the apps launch were just the beginning, that to deliver robust code requires metric gathering, SLA, Chaos Engineering, how to get the right metrics tracked and to look at all the signals as a organism to teach machines how to heal themselves.
But it always comes back to respecting the process while pushing forward education of the people taking part in it. Tearing down the silos while also demonstrating often the value of engineers who know everything but their app/service/code - yet is able to learn it and architect solutions.
HR.
Designing the organization is system engineering. You are building a complex machine with human, where flow of human, their rewards, and various "processes" are the knobs. And if I am to wager, HR is also the department people got issues with the most.
Some of you may say HR aren't the only one involved in designing the orgainization and you are right. Many in the management also do.
> Be explicit about what you are building
This. When you are in position to form a team, be conscious about what problem you are to solve. Organization wise, take business unit for example (forget HR, finance, compliance and legal for a moment), are you forming a team to PoC a new product line, to handle part of the customer journey, to build a service or to build a feature. How are the team carving up the boundary with other existing teams. When "refactoring" is needed. Companies at different stages should form team accordingly.
One way for me to judge one's acumen is whether they question the chesterton's fence - why a team exist and what problem they are solving. The same applies for engineers, why a component exist and what problem they are solving. There are a lot of similarities in designing a software component and an organization. One with code, one with people. More often than not, your code is also written by people.
The staff architecture is decided by the management and department heads according to the hiring budget they received from the bean counters.
HR just handles basic hiring and firing paperwork and organizing meetings and Interviews, that's it.
You might like my follow up, where I touch on this: https://cohost.org/jamesmunns/post/178752-systems-engineerin...
I have a lot more to say about how ignoring the people aspect of things is wrong for a lot of reasons.
The problem is "good vs bad" is a little domain specific. I can show something that works for embedded, but it won't be perfect for some other domain.
I have not yet run across a prototype or a product that does not have a producer and a consumer. Nor have I seen one yet that works for everyone on this planet.
Chapanis 1996 might be a good read.
It might be for your annoying nephew, the one who changes technology like underwear. It's not for you, the one who keeps the same phone for 10 years. :)
(worse, it is for advertisers, or stockholders)
This is the source of most problems: System defines something while HW and SW each come with their own implementation which contradicts each other.
System engineering has evolved into a sort of abstractive art, when only those who went to art school can underestand the abstractive painting.
Of course, elaborating in further detail is essential.
Of course, the contrarian position includes the fact that true system engineering organizations are actually publishing houses, producing hundreds or thousands of formalized documents. The actual systems created are byproducts. /s
IF so, why is James not getting paid?
If James works on an IOT widget the company will probably manufacturer a few hundred thousand devices and hopefully profit a few million.
If Steve is working for Spotify, with 130 million MAU, whatever Steve produces is not linearly scaled with the physical manufacturing of a widget.
Arguably operational costs go up with number of users but the cost is much more invariant in it's relation to total MAU.
This means that whatever Steve decides to charge Spotify, will always be a literal drop in a bucket for their budget.
It's the same reason VC's pour money into SaaS and social media apps but tread very carefully before investing in companies making physical things.
Fundamental paradigm of capitalism is to have a huge class of individual mass consumer. Prisoner of “individualism” to turn people in perfect consummer.
The side effect is that, the concept of complexity, complex system, complex system dynamics, are totally stranger to most peoples.
Mainstream rationalism is based on direct causal relation between two elements. But this framework of though and analysis is very limited and has very low fertility in term of complex system. Mostly cause most interesting features or beneficial properties of collective dynamic aren’t possible not understandable with mainstream rationality.
Let’s take an exemple: water. Water is matter, and matter in physics has three common state, either solid, either liquid, either gaz (we not gonna talk about plasma).
Can a single molecule of water be solid, or liquid, or gaz ? None of them. A single molecule is in undefined state of matter.
If so, can two isolated molecules of water can be in those states ? Nope again.
For physics, phase properties turning a set of water molecule into matter in one of those three states appears when the minimal set of molecule of water is 21.
So matter properties aren’t existing in a set of water molecules lower than having 21 ones of them.
If you aren’t aware of the works of Micheal Levin in morphogenesis and cellular self organisation, collective intelligence in tissue formation, I really push you to listen to the few interview of him on youtube.
How a set of cells are starting to communicate to collectively self organise and produce a complex living organism with demonstrating special intelligence, reproduction and adaptation is a real miracle.
None of those individual cells are able to do that alone, only when they start to communicate and build a cohesive group.
Why do I tell those things ? Cause from my perspective, there isn’t any difference between those cells and human, there isn’t any difference between those cells and company.
And each time I look back to my 20years in IT working in complex platform (finance or payment industry) I can only state that system engineering, should be renamed complex system engineering.
The final product (either a platform providing service, or a physical item) is the result of a collective intelligence trying to self organise and expand itself. So the product is fully part of this collective intelligence as a growth bénéfice. And is fully part of this collective intelligence.
It’s like quantum mechanic: the witness of the experiment is part of the experiment. He is not indépendant and neutral.
Therefor the system engineer designing a system is not neutral, as far as the set of all individuals in a company aren’t neutral and indépendant of synthesis of their own collective intelligence whose téléologie is to give birth into a product (either service or goods)
So in conclusion, the human environment is part of the complex dynamical system, and the complexity (information level) in the outcome (product) is often deeply tied of the complexity of the collective intelligence producing it.
So system engineering should not only focus on the outcome only. Like a system engineer designing a platform for a service, but consider that the company, the product, and customer, and the world outside all those element are a super organism far more complex than the product itself, and therefore should ensure that the superstructure has all the condition for the emergence of this super organism and make it viable.