Becoming a Systems Architect
wojtekmandrysz.com
wojtekmandrysz.com
I find this distinction harmful. Are we going back to the days where there were programmers who did the grunt work and architects who told them what to do?
I prefer how my company career ladder, where there are no architect titles. Each level is 'fractal', containing almost the same responsibilities but at different scales and scope.
So for example, as a Junior Engineer you will define the design (architecture) of a feature within a service, while as a Staff Engineer you will oversee the architecture of a system spanning several teams. This way architecture is everyone's responsibility, and people have a natural path to grow their skills.
There's just a lot of other activities that Systems Engineer does (and still consults them with the team, but is at the end responsible for doing them). (all the bullet points in "The realization")
Large organizations working on grand projects benefit from slicing up roles so that experienced specialists can anticipate and prepare solutions for the uncommon challenges that arise in those projects (and so that the bureaucracy of large orgs can be managed at all).
Meanwhile, smaller organizations, or groups working on more modestly scoped projects, benefit from generalists who can hold much of the project in their head and develop highly cohesive solutions that can be delivered efficiently.
A skyscraper requires seasoned architects or its going to crumble under its own weight, but even an extensive residential home renovation can be planned by a general contractor and done way cheaper than if many specialists were brought in.
It sounds like there are some organizations/projects you wouldn't want to be involved with, but that doesn't mean that those organizations/projects are doing something harmful or regressive. They just don't suit the way you want to approach work.
The largest tech companies out there (usually? anecdotal) do not have this distinction, it's the smaller and older companies that continue to enforce this arbitrary divide.
Great Architects work closely with the dev teams to give them reference diagrams, proof of concept code, and the art of the possible. Bad architects drop a big requirements doc and walk away. Almost every large organization has architects (enterprise, solutions, software, and cloud).
I’ve worked as a software engineer at places where everyone did architecture and it always ended up in a hairball mess that some senior/principal would recommend rewriting after 2 years.
How did you reach this conclusion? Both Amazon and Google don't structure their software engineering teams around an 'Architect' title.
I also searched the Google Carrera page for Architect positions and the great majority of them are customer facing roles with very few Architect positions assigned to specialized fields.
This is how you end up with antipattern-laden monstrosities. And I don't mean that in the dogmatic sense, I mean juniors will find "clever" ways of getting the job done, often with far too much abstraction and reinventing core technologies like message-passing, task schedulers, databases, ORMs... It can be a good learning experience but they need a active mentor/architect overseeing every major decision.
If you are going to reinvent something you should at least be aware of it. Juniors just don't have this broad perspective yet, almost by definition.
OTOH, one of the my pet peeves is an "architect" who gives this advice I am giving but doesn't actually have the requisite technical background, i.e. saying "don't reinvent the wheel" while looking at a teacup.
It’s not like SDE 3 (senior) makes you immune to architecture over-complication.
In your example, a mid level dev would oversee the architecture of the service. This dev would work to the junior to ensure their approach fits the rest of the service, and review their work. In turn a senior eng would oversee the architecture of the system owned by the team (possibly involving several services), and keep tabs with each one.
Delegation doesn't mean lack of accountability.
ah haha I just stepped away from my work computer after creating a new branch and writing a "short term fix" in a particular module instead of fixing the root cause because that would involve talking to multiple teams. Instead, I can just do some simple string manipulation just for the one bug I have...
I am trying to talk myself into or out of creating this pull request...
Building some theoretical construct and having other people build it is by itself bad architecture, resulting in bad team culture and bad systems.
However, there is a scale where that does not work anymore. Once you plan at the scale of 20 or more teams, "working closely" is very relative.
If nothing else it's a shift in the responsibly of architecture. In the most rancid organizations there are silos where devs only code and architects only design. The joke of the Ivory Tower Architect is real. On the other hand as a dev II (between junior and senior) I was asked to design and implement a large scale green field distributed system that interacts with other teams. Our ladder and job titles not only allow this, but it's encouraged and sometimes expected.
Agreed on fractal model with healthy room for mistakes that should be carved in for everyone - it's the only model that can enable healthy growth. Non-practicing architects are 0, and non-architectign engineerign is 0 too.
This said, as much as I hated the architect idea earlier in my career (because I was architecting regardless of my title) - the concern and ability to architect things is a very real one. It can only be learned by taking responsibility and having room for mistsakes.
So literally that's "hiding" an actual distinction that it's already there.
I agree, every dev should be able to design systems. With software development becoming a highly desirable career due to money, this is just not true and possible anymore.
The world is full of productive developers that are not skilled enough to design systems. Designing systems is also incredibly hard, I have rarely seen (in my very limited sample) more than a few devs able to do it in a company.
So we have a category of people who put a lot of effort in becoming good at something, but we don't recognize them because "everybody should know that too". I think that's not positive.
People who want to grow in the design skills are free to do it and resources should be available, but the reality is already that people at higher levels tell you what to do. Isn't that what you would expect, somebody with more experience driving you?
This is one of those arguments that looks good on paper, but causes untold pain in practice.
On the whole, lanes are created by companies to create a promotion flow so that people can always keep trying to climb the ladder. We see more flat organizations today, but as an organization grows to become huge, hierarchy is almost necessary.
It is about culture. Which is very hard because many junior architects have a power trip over their fellow engineers.
lanes are created for specialization but when you do that it's a clear indicator that if you're not specialized in that area you should be deferring to those who are.
You wouldn't want a devops person telling you how to organize the controllers in your application, and that devops person wouldn't want you telling them how to organize the management of the pipelines. That's not culture, that's necessity due to scale.
The point being made is that should be put off as long as possible because when you put it into place suddenly the communication becomes a lot harder and it's not obvious it's always a benefit until there's real pain from not doing so.
---
I would not consider understanding the large context of things to be stepping outside your lane and I suspect most people wouldn't either.
Now obviously to design such a PaaS successfully you should know a thing or two about ops, but the outcome of that expertise is a reusable software platform, not per-product (or worse, per-release) labor.
I don't know any companies that use that word. Normally we are called software engineers. Amazon is the closest, with "software development engineer."
Nor do we use the word devops. Which is funny, because as I understand it, software engineers working on infrastructure automation in the same way that that we work on products are the closest thing to what "devops" envisions. The companies that are doing it don't use the word, and the companies that use the word are the most incredulous that anyone could be getting by without dev and (dev)ops as substantively separate roles. It would be almost as if the people who most liked to talk about Agile were the ones most invested in very long range estimation and planning. Oh wait..
In regards to culture and scale and lanes...what I was referring to was having a culture of people who are accepting when others ask if they can help, rather than telling those people to go mind their own business. If employees want to do what they are specialized in...totally fine...but if I am interested in a topic that someone else specializes in, I would like to have an opportunity to explore that(without being a PITA obviously).
It's pretty common even in senior developers to not be experienced in system architecture, so the problem is not resolved with this approach (based again, uniquely on my experience. I didn't run any statistics)
I saw that all the time, it's really hard to identify the right partitions in a system. It requires a lot of research and experience. Whenever one of those partitions is missed/overlooked, the software gets muddy and hard to work with around that area.
When brought to the extreme where very little boundaries were identified, this is common in CRUD apps where entities are confused with partitions the entire time, every change to the code ripples from the database all the way to the frontend.
In Rails world, this is a highly common problem: a change to database schema affects the UI directly, so every time something is changed in the schema, the following has to change: - JSON renderers - React or HTML views - db schema
Even for additive changes
If you are in a smaller company, totally agree. Get way stronger foundational developers.
Anything that harms those feedback loops is going to create pain.
I would expect an Architect team to deal directly with the product requirements, stakeholders, etc. and produce a design that they hand off to others to execute, while they move on to the next set of requirements. Architecture is part of the project lifecycle for everyone. This process is simply the architecture complement of code review, only slightly more formal.
Regarding the Architecture team, that's literally what caused problems in so many companies, that's the definition of how to have bad architects: the architects are so disengaged with the system that they cannot make recommendations.
An architect team should be highly integrated, participate in the coding aspect and only once something is shipped, moves on.
>An architect team should be highly integrated, participate in the coding aspect and only once something is shipped, moves on.
At that point, why are they a separate role or team? That sounds pretty close to architecture being one of the responsibilities of the main development team.
Having an integrated architecture team serves for training purposes and for overlooking the mistakes.
At the same time, it helps keeping the architecture team up to date with the codebase.
Why wouldn't you do that, if it's highly beneficial in both directions?
The fact that a separate team does design work has been proven to be wrong, of course we shouldn't do that again
Yes I'd agree it's a similar role, I'm just making a point of recognizing the specialization.
Put another way, wouldn't you want the Postgres expert to design your Postgres schemas? Or the Rails expert to lay out your Rails application?
It shouldn't be a person out of the software loop, it wouldn't make sense.
I never intended to create a low bar for the title, nor a separate role from "software developer". I'd expect the architect to still be a software developer able to contribute to the existing stack.
The prevailing sentiment here seems to me to be that we should do away with the title architect and allow senior developers to perform that function. I believe I saw a reference to Hohpe's "The Architect Elevator" as an example of the old school of thought in the comments here. Check out this guest blog of his https://martinfowler.com/articles/architect-elevator.html where he makes the case that senior developers should set the architecture for services and systems but that there is also the need (in larger organizations) for architects to advise on software portfolios and technology radars.
The job title may say Staff Engineer, JavaScript Ninja, Principal Engineer, Senior Architect or any combination of the above, but in the end someone has to do the job.
When your company didn’t use the specific title of architect the role of architect was still distributed between different engineering levels. For example you had the role of X Engineer + Architect… this can work well if the project is mostly of technical nature and doesn’t require a lot of technical requirements elicitation, negotiation, synchronization and other people and organization focused work. Because if it does contain such work, you’ll be spending so much time in meetings that you won’t have time to write code any more.
Other companies assign the architect role to a specific person, or have multiple kinds of architects. An important duty of architects is alignment within and outside the organization and mediation between business and engineering.
In large-scale projects such work is especially important and can’t be done by someone with the title of developer, because they would probably get bored out of their mind and also not actually develop anything any more.
Neither of these approaches is wrong. I’ve worked with dev teams that expect to receive a design/spec that they will implement and at the other end of the spectrum with teams that love to design their own architecture. I will provide that spec for the former and I will make sure that their architecture remains cohesive with the platform architecture for the latter.
And it did happen to me that teams would insist on designing their own architecture and then create an over-engineered heap of code. Too bad for them, because they have to fix the bugs, but still ok for the platform, since all the interfaces and platform requirements mesh as they should.
Whatever level the architect is operating at, it’s crucial that they keep their technical skills sharp through building small prototypes, reading and analyzing code, code-review, etc. I can say from my own experience that whenever I neglected that, I started to become detached from the reality of the project and my decisions were poorer as my problem-solving approach became too abstract.
Sure, but at some point "spanning several teams" means you will not be close to the code anymore. You'll spend most of your time in meetings and it increasingly becomes difficult to stay close to the code. Doing "grunt work" is increasingly a waste of your time and a distraction from doing things that have impact at your level. At that point you end up telling people what to build and having them build it.
Doesn't this distinction exist in every profession? Experienced people start doing less and less of the day to day tasks, and start doing more strategic work.
Only in software do we think this is a terrible thing, and that an acceptable use of someones 10 or 20 years experiences is for them to be down in the trenches centering divs with the new grads.
Hence terms like "staff engineer", which is a modern architect but with a name that pays lip-service to egalitarian fantasies.
And I've seen what happens when a whole team of devs get invited into a zoom meeting like that - they're bored out of their skull, most likely sitting there browsing the social media du jour, and I can't blame them.
it's like throwing Eisenhower into the trenches. Sure, he's a good soldier, but it's time and experience wasted.
Yes because aligning goals, putting together a system,scoping work, and synching business goals with technical milestones adds no value at all. It's the nail-hammerer who believes that JIRA tickets spring from the JIRA ticket tree magically aligned that caries the whole org on their shoulders.
I did architecture work when I was a junior. It's what made me a senior. I got asked if I wanted to do architecture work recently. I told them that's what we hire juniors for.
Architecture is piss easy. Any senior thinking juniors can't do it are full of themselves.
How anyone could actually BE an architect without a lot of actual developer experience is something I'd like to know. Are you sure you're not just being unfair to the architects around you for disagreeing with you on things?
I cannot fathom an architect who wouldn't be a strong developer if slotted into that role.
I think the disconnect here is that the majority of architects I've worked with haven't been "good" in a technical sense from a "actually building software" perspective. I'm assuming this is true for a lot of folks here, based on people's responses in this thread.
The day to day of some of these roles has been:
* sit in meetings with high-up product managers and biz leaders
* come up with pretty diagrams of how to totally re-build (sorry, re-architect) the system. Often this is jargon-heavy - protobuf this, envoy that, circuit-breakers the other thing.
* talk engineering management into this plan
What's been meaningfully missing is the details of point A -> point B to achieve their plan. None of the jargon is necessarily wrong but it's not grounded in the reality of the current software systems at all.
This comes about when the technical VPs aren't hands-on with the system anymore either. If non-technical leadership is saying something like "improve uptime" and you go get hire an architect into the company and you'll get a lot of presentations about things the devs on the team may already know about but haven't been able to get prioritized, but it's still on the existing devs to actually build it.
"Paying someone fancy to convince non-technical business leaders to let the devs build what's needed" isn't necessarily a bad decision from a CTO or SVP of Engineering, even! But the devs who have been saying the same shit this whole time only to now see the credit go to the higher-ups and the new architect, after getting the blame for the old status quo, don't exactly have a positive view of the architect's actual engineering competency.
Oddly enough, I'm having this discussion right now with a few other people at a new company. They want new documents to describe the ideal and I want it to describe how things work currently and evolve it over time.
OTOH, I've never gotten the impression that any of them are technically weak, I think they just view their role differently than I do. Much like how a TDD proponent will start imagining unit tests are the point, I think these architects have started to think these documents and diagrams are the point.
Just in the last few months, I've found it difficult to be personally productive writing much code. Between catching COVID, then a string of other viruses, and having a young daughter and a baby on the way, I just haven't had a lot of quality focus time. Instead to feel like I'm earning my paycheck, I've had success in writing out high level architecture documents and diagrams, then relying on my coworkers to take over ownership and execute on the details. It's actually seemingly working out well, since I'll be going out on paternity leave toward the end of the year.
> "The SystemArchitect responsible for designing your system hasn't written a line of code in two years. But they've produced quite a lot of ISO9001-compliant documentation and are quite proud of it."
As long as Conway's law holds, system design is going to be at the mercy of the person who designs the team organization. It doesn't really matter what title you give things.
I can tell you from experience that powering through a conways law mismatch doesnt yield good results. "communications overhead" feels abstract, so let me make it more concrete. When the org structure has no mapping to the software structure, the following bad things happen:
- there is no clear rule for who's job it is to fix a bug. You have to always inspect the code and guess.
- same issue for code review, and your code quality standard ends up beholden to the team with the tightest deadlines
- nobody knows which manager to escalate issues to, and the only effective strategy is to haul a bunch of PMs and senior engineers into a conf call to analyze and decide
- the testing and release strategy becomes one size fits all, and cannot make any assumptions about independent subcomponents of software -- you have to test and release the entire product in one go
This is the form communication overhead takes.
You're fooling yourself if you believe that that's not the norm already.
In fact, it boggles the mind how anyone believes that inexperienced people who are taking their first steps writing code and don't know what a software architecture is have as much say in the daily workings of a project than the grey bearded guys with a couple of decades of experience under their belt.
No I don't think so. Where in the article did the author said he is telling other people what to do? What he said is
> proposed software architecture consulted it with key developers, and made sure everyone is onboard
Being an architect, or staff engineer in modern term, means that the person will need to use his experience and expertise to help the team to find appropriate solutions, present to the team, list out pros and cons, let them pick, and drive the change with other stakeholders.
We - unaffected bystanders - don't really need to have opinions.
It always seemed to me that people with more of a managerial background would be better managers - is software/system development the only field where masters of their craft ultimately become directors?
I don't know enough to say whether this is a common pattern or not, but if it is, it could perhaps be that tinkerers tend to gather a vast breadth of knowledge that can be very useful when making strategic decisions. They reach the point where they know enough to understand what questions to ask on many topics, even if they are an expert in only a few (or none) of them.
I don't know how else you pick up the skills to be an effective technical manager
Because those tinkerers get to a point where they want to be the ones making the decisions, controlling the culture, technology, and direction of the company. Without position you have no power and without power you can't affect change.
I'd love to be able to find folks that can do what I do -- it's probably our #1 issue holding us back -- and direct them to meet some business and technical goals, but I've yet to work at an organization that supports that mode of management.
It's a lot easier to spot developers or contractors bullshitting when you've been in their shoes.
People with managerial backgrounds have....what? Nothing really. They have to guess at any plan presented by technical people, are always suspicious they are being screwed over on estimations and real problem areas, and are unable to correctly identify when people are doing good work -- thus also being unable to set a healthy engineering culture for success. That's why most managers are demoralizing for engineers. They just don't get it.
It's interesting to note that alot of the most successful startups in SV are not from MBA's but engineers with masters or PhDs....it's not a coincidence I think. They have the practical experience to lead a real world venture to success.
Managers are good at managing departments like insurance claim processing or bad debt collections, which any human can learn in a few weeks fundamentally.
People with managerial backgrounds can become quite adept at helping you:
- Identify blindspots in your biases and behavior that keep you from peak performance
- Avoid working on stuff that's not valuable to your team
- Settle disputes within a group
- Motivate you and keep you engaged/fulfilled with your work
- Get unstuck with personal problems
This is not an exhaustive list and you don't have to have a 'managerial background' to master stuff like this. I am an engineer who has had to learn management as a startup founder. I used to distrust the whole management thing but that kept me from growing as a teammate. Management is not only useful in 'non technical' jobs, it's useful in all human endeavors it's why we study it so much and why it has so much leverage.
Managers in the middle of the food-chain are all about power-struggle and social games they play to out-compete each other.
That is especially the case for non-technical managers who don’t have a sense for the underlying technical challenges. They have all their bandwidth available for political positioning and social games.
It would be great if all managers where “serving the team” in the sense of your bullet points. But alas many don’t see it that way.
IMHO many startups are successful because they have a technical person at the top. Who is capable to understand, evaluate and positively reward technical work within their organization.
In tech and pretty much every company I've worked at, you need to get into a managerial position to be able to have a say in what get built and how much you get paid. Programming fatigue and frustration with being told what to do also sets in after a while. The two together convinces people who are terrible with their people management skills to chase a manager-path career.
Essentially, you're diagnosing the symptom to be the cause.
- Cannot identify blind spots that keep you at peak performance, only another engineer with more experience (tech lead or senior) can do so. They can only identify behaviors that make them look bad or are inconvenient when viewed from the point of view of their peers -- other managers.
- A non technical manager has literally no idea what is valuable to delivering complex technical work. They can only guess, and often guess badly. Again only a senior or tech lead with experience could do this.
- How could a non technical manager motivate any engineer, without an understanding of their difficulties, problems and ways to solve these problems practically? I just don't buy it. "Let's do overtime on the weekend guys..."
- Managers should not be involved with people's personal problems. I've met so many managers that are extroverted and managing sensitive introverted teams, that all they ultimately do is the equivalent of hammer on the aquarium glass. Remember that sign in the pet store: "Don't tap on the glass"? It's true of technical teams that are of a totally different temperament than managers.
Thanks for the detailed breakdown and no name calling :D
> Cannot identify blind spots that keep you at peak performance, only another engineer with more experience (tech lead or senior) can do so. They can only identify behaviors that make them look bad or are inconvenient when viewed from the point of view of their peers -- other managers.
Given you have different levels of skill in the team, a good manager would convince you and the other teammate to help each other out with the learning. Lubricating these interactions given everyone has responsibilities is not always trivial. I'm less experienced on the politics but I believe what you're saying about politics distorting incentives. I'm conveniently side-stepping this issue bc it applies to all positions in a corporate structure.
> A non technical manager has literally no idea what is valuable to delivering complex technical work. They can only guess, and often guess badly. Again only a senior or tech lead with experience could do this.
A non technical manager can know very well what's valuable to the product/company. That they don't believe their team on the value of a specific piece of technical work to enable that seems like something else is at play here (lack of trust).
> How could a non technical manager motivate any engineer, without an understanding of their difficulties, problems and ways to solve these problems practically? I just don't buy it. "Let's do overtime on the weekend guys..."
By reminding/reframing/convincing re:impact their work has on their team, personal growth, customers, society or personal preferences. Doesn't have to be only technical; they can help you deal with any self-inflicted discomfort regardless of the subject matter.
> Managers should not be involved with people's personal problems. I've met so many managers that are extroverted and managing sensitive introverted teams, that all they ultimately do is the equivalent of hammer on the aquarium glass. Remember that sign in the pet store: "Don't tap on the glass"? It's true of technical teams that are of a totally different temperament than managers.
Agree to disagree. I've had great conversations with peers when/if we're open to talking about non-work stuff - both ways not just me 'giving advice'. It's not binary and depends on the relationship. A skilled manager can care for reports beyond work, create genuine bonds and be respectful when they have not been given an opening to engage in these subjects.
Trust is a difficult commodity to build, a lot of company culture issues stem from lack of trust. It's particularly key to the manager/team relationship.
When you have a non-technical manager directly over technical teams it's particularly difficult to build trust. People, emotionally, want to have someone really understand them. Someone who doesn't, at a fundamental level, understand the actual work you're doing is going to be at a disadvantage as the work is crux of the purpose of the interaction.
Not to say that it is impossible, someone with well above average people reading and listening skills can still build that trust and get it. But it's definitely going to be more difficult than someone who really knows the turf.
I’ve seen multiple times my careers were dozens of people no will work for months on something that just isn’t important.
Being able to put things in terms like this feature will cost us $1 million in developer time. We can expect a return of $50,000 over the life of the product. Or vice versa.
Stuff that many developers don’t think about.
Reminds me of “What would you say you do here?” https://youtu.be/m4OvQIGDg4I
Probably ambition. At a certain level of experience, you realize you cannot create your technological dreamscape by yourself.
Becoming lord of a technology company and directing its resources is like programming the ultimate computer.
I didn't get the impression that he's your typical CTO and completely hands off with code and development. If anything, it sounded like he's still very much in the trenches but has learned how to delegate work well and pick which problems are worth the time investment.
When it came to his work on Oculus and the work he's about to do in the field of AGI it sounded like he'll definitely be making direct contributions. It's entirely possible that I misunderstood his stance throughout the interview though and he's a more hands off guy now.
After that, you have a set of "programming tools" (I don't want it to sound mechanistic, because it is the opposite of that), which are your team and reports, and they will be able to fill in the details (and the details here can be significant pieces of design and architecture by themselves). And your role is to choose the right tools, and allow them to work to their full potential. This means clearing obstacles, clear communication, technical help at times, mentorship, aligning expectations and giving them clear paths for growth.
All these things can be considered engineering at a larger scale. You want to get a really big system shipped and productive? This is the work, these are the skills you need.
The distinction dates back to the industrial revolution, where you had manufacturing line workers and line managers. A manager would usually be an owner's relative or someone they trusted – more loyal to the company than the unions.
This distinction perpetuated well into current age, just notice how much implicit bias there is about "programmers don't have people skills" to keep workers accepting a career ceiling. Most managers aren't skilled either, and not respected by the workers due to it, but companies won't keep them from managing because they need someone to be responsible for plans, estimates, OKRs, etc.
I guess in software it's "more" common due to survivorship bias – the business of software is so messed up and people have so little idea how to manage it, that companies without experienced leaders have a smaller chance. Strong companies and teams in the field have leaders with enough hands-on experience to have natural authority.
Theatre and Cinema are two other fields where folks have been known to work their way up to director/producer/megalomaniac...
I think Weinberg in "The Psychology of Computer Programming" talks about the performative nature of writing code.
I disagree.
In my experience hiring the right people limits the need for organization and process. Obviously you can't eliminate those completely but if you hire the right people your org will generally succeed in spite of a lack of organization and process.
I find the larger problem is how most engineers define "smart people" - they typically only mean someone with a high technical aptitude when really things like communication skills, collaboration, motivation, etc. are better indicators success on a software team.
For 'simple' systems, Systems Engineer is not needed. Typically Software Engineers share architectural responsibilities.
I recommend listening to this https://youtu.be/pSfZutP9H-U?t=386 from 6:30 to 10min. I find it very valuable.
> I find the larger problem is how most engineers define "smart people" - they typically only mean someone with a high technical aptitude when really things like communication skills, collaboration, motivation, etc. are better indicators success on a software team.
I agree 100%.
But I agree if you have a lot of smart people you can go for a lean process, and let everyone mostly guide themselves. But you do have to provide that freedom.
Most technical people view disagreement as a negative for the person doing the disagreeing, I've seen it play out too many times over my career.
This is true. I like Marty Cagan's take on it in his article "Scaling with Process vs. People"[1]
Netflix has a presentation about this somewhere...
I highly respect the approach of architecture in the team/agile environment and seniority in engineering. But at a given scale, someone is there to orchestrate all of it and does the overall plan. That person is a solution architect.
Personally I noticed, that there's always something to analyze more in-depth, something to document better, yet another thing to teach others, however other engineers that I work with don't really care about creating coherent view of solutions they create and they don't care about providing clear description/instructions/guidelines for their solutions.
So do you depend on developers to prepare complete solutions and only verify, if they do everything that you would expect or do you actually have to guide them through details and support them in design, so you can together create something that works according to architecture?
We expect engineers to refine and review the architecture. They are engineers. However, they are busy delivering something while the architects talks with business and define the solution for the next iteration. Solution design takes time, soft skills and expertise. You cannot do that and deliver something.
One of our best solution architects is not capable of programming. He holds a degree on physics. But he is good with finding solutions, managing information and also excellent in communication.
I've been a solution architect embedded with a team of 4 devs, but was constantly dragged into meetings by more senior architects collaborating on the wider platform we were re-writing. That collaboration was important, but it consumed too much time. At the time I lost connectivity with the developers and they were to a degree twiddling their thumbs while a design was nailed out.
Since then I've come to see the benefit of design and architecture stages including the actual developers as much as possible/as much as they want to be included. By that I mean specifically things like getting developers to pair with you on proof of concept work, where you need some form of discovery that will lead to a design decision; pairing with developers on pieces of analysis and documentation around the current state of the system/application, or business/user workflows (in teams where there has been no business analyst).
With conceptual/high level design you might not need much developer input, but when you get into the detailed design/logical design dev input can be crucial (especially if you don't know a lot about the current state of a system). When I'm putting together a design I like to go as wide as possible first with the scope, and then fill out the scale of each area with details after further analysis. Getting your developers (if they're available), to help fill in some of those gaps can be beneficial.
Getting that early engagement with the actual developers is key, and in my opinion taking a more agile and scrum approach to the architecture is much better than doing all the big design up front with zero developer input and then throwing it over the fence. That said, your development team needs to be geared for this, and some might not be, if all devs are busy delivering feature A while the architect looks ahead at what's next. But if I can get time from at least 1 developer who has a good handle on the current code and tech stack, then that's definitely something I capitalize on.
I've spent most of my career fixing bugs or improving existing apps/systems in C++. Every time I interview somewhere they want me to design a (usually offline) program or service and I don't know how best to do that. At this rate I'll be stuck at Senior Engineer for the rest of my career. (I'm already 14 years in).
* RPC = function call
* API design = function signature
* data schema = simple structs
* message queue = list
* load balancer = simple facade object
* database = hash table / in memory SQLite
Once you have a rough sketch in place, you can run your prototype and test different behaviours.
It is a bit "tricky" to tackle asynchronous and concurrency, which you should do from the start. I prefer to do everything based on callbacks, and run my simulation from a static input list of events, ticking every component with a tick() method. This way I can generate different scenarios, save them to JSON, replay them.
Here's some examples:
- https://github.com/wesen/summer-pasture-netflix
cell.py is a kubernetes prototype (blackboard system), while app.py is the start of the netflix event notification system.
I am thinking of writing a course / book about the subject, and have a bunch of (wild) notes here: https://publish.obsidian.md/manuel/ZK/Structure+Notes/2+-+So...
I also played with pymc3 where you can just place plain python inside your probabilistic model and MCMC the heck out of it. It's slow but you can use some real world input and see what distributions arise at each point of the graph.
I don't know about the book, but I think MATLAB Tech Talk about systems engineering might be a good start for you: https://www.youtube.com/playlist?list=PLn8PRpmsu08owzDpgnQr7...
Part 3 is about designing a residential toaster :)
You can spend years learning all about great design patterns that make sense for an IoT or embedded system but have completely backwards ideas about how to organize code for a game that needs to process 1000 unit movements per frame.
But once you figure out some narrower domain that you like, you should be able to find books about it , (if you're good at reading code) can start studying open source approaches to it, and can start prototyping small projects to gain some hands-on experience and muscle memory.
Systems design is for me quintessentially embedded system architecture for a microcontroller, a SoC or combination. It could be a smartphone or console, or medical device, etc. This is a very broad domain… in my case I work exclusively on non safety-critical SW, so I’ve focused on learning Linux internals and system programming which I combine with an architecture framework. Typically I design middleware and daemons which work together to cover a specific area of a product. I work with different kinds of specialists (HW, kernel, performance, etc) and a lead SW architect for each specific platform. The system architect is a different person that oversees both SW and HW and requires skills in both.
For Linux I cand recommend the classics: Stevens, Kerrisk, Love. I found Chris Simmonds’ book to be a good general intro to embedded programming; it has much info which also helps with embedded system design. I’ve also picked up ideas from Making Embedded Systems by White or Embedded Systems Architecture by Noergaard.
Application design refers to individual applications running on an OS. These are the typical MVC or layered architecture styles. The official OS UI framework will likely define a recommended architecture while headless services are quite flexible and support several architectural styles.
Vol 1 of the Pattern Oriented Software architecture is a good intro. I have Software Architecture Patterns by Richards on my reading list, since it covers more recent styles like microservices.
Often, decomposition is based on areas of domain or functionality. There are some downsides to doing it this way. The method in the book proposes it be based on areas of volatility instead. As an example, imagine writing a piece of software for FedEx shipping. What would change to use that same software for UPS instead? That may identify some areas of volatility. If you can plan that something might change, you can limit the ripple effect of those changes in future. Just an example of one benefit of doing it this way. The book explains some of how to approach designing this way.
Another point: Architecture is not just about designing the software. It is also designing the project to build that software. This involves things like critical paths, staffing, risk management, etc. Understanding these is important if you want to be able to deliver on time and on budget.
I suppose it is another tool in the box, however in the video it was about to be introduced as nothing less than a magic bullet.
Regarding the second point, I’m not sure I agree. An architect could get involved in staffing decisions and any project has a list of risks, for example I’ve maintained such lists as part of the architecture description. But I would define these these are side-tasks and not as core tasks.
One worry I’ve had with taking jobs with that title is that the salary range seems significantly lower on average. Even if the specific job paid well, I worry about pigeonholing myself into those roles.