Software designers, not engineers: An interview from alternative universe
tomasp.net
tomasp.net
> You fixate on some technical solution and then completely forget what is the context in which you're building it. Good software designers basically treat all problems they face as ill-defined, even if they are fairly specific.
Oh my god, I nearly fell out of my chair. YES.
What we need are tools for software development that are highly flexible, easy to use, and safe. We need to put those tools in the hands of someone like a software designer from this blog post: A domain expert (and/or intuitive listener) who deeply understands the business, understands their customers' needs, and can hook up whatever logic is required themselves.
(And uhh, shameless plug, I'm actually working on a product right now that attempts to realize this dream. We're hiring platform engineers. See my profile for contact info/links.)
But the problem is, no one has put them all together in quite the right way to satisfy our needs.
- We needed a platform that would let our non-programmer staff create complex integration-related features for customers. We needed an accessible VPL that was easy to get started in, but powerful if power was needed.
- We needed a lot of integrations to not-particularly-popular business software (which means, we needed the ability to write and/or extend them ourselves); we needed the integration editor to be accessible and easy to use by junior programmers.
- We needed data pulled from integrations to be stored and kept around for a while; we need to look up individuals, see their history and a history of triggered VPL logic; we needed to be able to update and bulk-execute VPL scripts against the data store; we needed full text search capability as well.
- We needed workflows/data organized into tenants with a user auth scheme that would let us invite business customers to see just their data and no-one else's, with view-only permission as well.
- We needed something that could scale up to handle a lot of work, and still be at a competitive price.
There isn't a platform that has all of these things. One or two, sure. But not all of them. My hope is that in 5 years b2b saas companies won't have to build much integration software themselves; there will be us, or someone like us, that'll handle it for them, and they can focus on whatever it is they actually made the company to do.
We _have_ product design experts and processes for software products, but that's not a reason to not have engineers. And as to why they're not celebrities typically ... I think that's also in part because the companies that produce software want to control their brands. This isn't a threat for architects; BMW doesn't lose anything by us knowing that Zaha Hadid designed a building for them. If she had then designed a building Toyota, that wouldn't have been a problem. Their value proposition isn't about the building they work from. But if you knew and cared about the name of the specific designer behind your favorite software product, and they left the company ... that would be a problem for that company. Part of the prestige associated with their core offering is able to walk away.
> Not really, no. When you're solving a problem, even as you get to a more technical level, you always keep in mind why you are solving it.
I have mixed feelings about this. On the one hand this pretty much describes many applications built with a lisp. Everything is built to solve the original problem. Every construct made ad-hoc only handling the cases that are relevant, even making up effect partial languages suitable for the subject at hand.
On the other hand, we don't end up with globally shared libraries that have 'completely' solved a particular technical problem space. We actually lose the 'science' of computer science and everyone is an application developer.
Based on the popularity and effectiveness of libraries and ecosystems, I'd say our universe is winning. I do however enjoy quickly making partial implementations that perfectly satisfy an application's needs up and running quickly. Sometimes the complete solution would be unapproachable but the needed one clean and simple.
Perl is an exception--it has a vast ecosystem of partial implementations.
Consider a serialization library, for reading and writing objects from memory. This library could aim to have a minimal memory footprint, and may fit into the design of an embedded device. This library could aim to have maximal throughput, and would best fit into the design of a server application. This library might aim for usability by new users, and handle its own memory allocation for the serialized object. But, that would imply serializing the entire object at once, rather than streaming it to disk or to a network socket, so that impacts what designs can use it.
Libraries tend to be implemented such that they fit neatly into some original design. Figure out what that original design is, and that gives you a better sense for how it will fit into the design you're making.
There are many software designers, they are just not famous because no one would understand what they do.
IME the more general a library tries to be usually the worse it is to actually use. Opinionated AKA designed libraries that have some specific intention behind them are more immediately usable and it's much more straightforward to see the pitfalls in using them as they tend to typically be intentional.
and yet, the most popular libraries tend to be very general, and libraries that are popular tend to try to get more and more general.
What's this about? Why are we chasing a different direction than the quality that comes from focused, opionated libraries?
That choice of components still requires keeping the purpose and context in mind, of course. But if I want to get anything done for the user, at some point I hit bottom and return my attention to getting something done and out there.
Once the data layout is properly designed, the data transformations required become clear and the rest of grunt work.
~ Fred Brooks
> I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.
~ Linus Torvalds
Algorithms are technical debt.
Once you modeled your problem into some data structures you now need to make the decision of whether or not the transformation between input and output is easily achievable with an algorithm, if not then you need to further break down the problem into more data structures or your design is flawed.
Is that practical today? No. The effort required to get a full program created at that level of detail would be enormous. Technical debt is in itself not a bad thing, and I'm not saying anyone in this realities goal should be no algorithms.
My experience is that good teams start by asking lots of questions. The design may not be entirely formal, but good developers do go through the process of figuring out their internal list of requirements needed to satisfy the requirements of the system as a whole, and try to make sure that the decisions made are well thought out and don't box the codebase into a problem space in the future.
In other words: this article was written on a computer built in our universe. In their universe they'd be stuck on electric typewriters.
Also what specific things does an OS let you do. Think about the end user from that perspective. What do they want from an OS? It’s hopefully quickly apparent that you can consider these things in the way the article suggests. It’s just the emphasis is slightly different.
I certainly think that emphasis difference would result in different OS designs but by no means would you be stuck with electric typewriters. That feels like a very uncharitable reading.
For example:
I still feel like this way of thinking can work only up to a certain point. Don't you need to get to a more precise problem definition at a lower, more technical level?
Not really, no. When you're solving a problem, even as you get to a more technical level, you always keep in mind why you are solving it. I noticed that software engineers in your universe often ignore this. You fixate on some technical solution and then completely forget what is the context in which you're building it.
As you can see, this isn't the sort of mentality that gets you a general purpose tool. You might get JPEG, but you won't get ZIP.
But it is true that most of the existing books probably don't include some of the types of case studies and lessons that you would get from a job. But I think that textbooks should simulate those things with exercises. Of course a textbook can't include real dynamics or interactions but it's not true that it's impossible for some of those lessons to be in a book or at least touched on.
I mean, a textbook could actually emphasize things like digging in and iterating to really understand the problem in depth. The criticality and difficulty of understanding the problem and finding the right framing for the problem, realizing that is something that should be deliberately shaped to enable the solution.
To me the most challenging thing has always been the power imbalance between myself and the managers or users who are incorrectly framing the problems, and the struggle to negotiate an adequate problem definition that will enable a good outcome.
Ditto for hardline requirements from project management. It's either a CYOA strategy or a holier-than-thou mentality.
Imagine if software was held to the same level of scrutiny..? Software is carpentered, some is designed, and little is actually engineered.
Engineers don't fail forward, early, fast, or often. They don't move fast and break things. Not engineering software is great, because if you had to pull permits to build your software, the web would still look like 1996 and my phone would have a keypad.
We aren't developing anything with direct life-safety consequences, but we are dealing with PII and other people's money. I see no reason to relax due diligence on anywhere along this spectrum. It's either correct or its not, and the consequences for getting something wrong are universally bad for all involved.
Now, I also strongly believe in the idea that software development is inherently an art form. The nuance is that within this art form, there are still very rigid conceptual frameworks you need to work with. For instance, designing the UI for any application is more-or-less a pure art form. Developing the SQL schema backing that UI is arguably something that is 100% deterministic and can have mathematical proofs established which prove its correctness one way or another.
The key to the success of any software project is blending these two worlds together in a sustainable and responsible way.
Maybe I'm misinterpreting your point here, but the way you create the different models, compose the modules which will use them and represent the abstract entities and processes that execute when that UI is interacted with can be done in many ways, both well and very badly, depending on the measurements you take to evaluate it. All the different terms to consider 'clean', 'maintainable', 'readable', etc. code, I consider to also be somewhat abstract to define in an art-type of way.
The reason I mention this is that the 'visual design' of the UI is not so much a software work as much as a UI/UX designer work. Bringing that design to life in an application using actual code and not a design tool is where the same principles I mentioned above come into play, even if the actual 'visual design' was completed by a different person which might not code, or the same dev.
If you get a super clean domain model in 3NF, you can be assured that writing business logic on top of this will be a breeze. Getting relationships between domain types incorrect, or having facts on the wrong types, is how you get a vicious cycle complexity layering on top of itself as you try to compensate for bad foundations.
One thing that burns a lot of developers is the need for circular dependencies. You cannot say that circular dependencies are disallowed when trying to model the real world. An example in banking would be Accounts can have many Customers related to them, and Customers can have many Accounts related to them. Attempting to "choose sides" on this is folly. The most correct path is to create 3 collections - Customers, Accounts, and AccountCustomers (or CustomerAccounts). Getting this one thing correct can save you years of developer hours on the bigger projects.
If you are using something like .NET, you can abuse LINQ to produce any arbitrary projection you desire in a few lines of code. If you want your business people to have a hand, you can expose SQL as a scripting language against your properly-modeled domain. There are so many good reasons to slow down and pay attention to what you are calling things (and what boxes you put the things in).
To put it more briefly - The argument I am essentially trying to make above is that there is actually one truly correct way to model a specific problem domain and you can test for many degrees of that correctness. I will admit it is never absolute, but I feel it gets damn close if you are thorough in applying single responsibility principle to the logical business facts.
Implementating BCNF is a science.
Choosing whether to use a normalised relational database, a star schema, or a non relational eventually consistent database, that's design.
Maybe you can prove what software does, but "proving" that what it does is the right thing is necessarily subjective and grounded in reality; mathematical proofs need to get dirty.
I think bot good engineering and good desin practise is the opposite of developing a picture. It is really iterative process, trying to factor in as many of the important aspects as possible and try to get the one solution where all of these aspects don't fight each other, but strengthen each other.
Many software developers I met are just focused at what framework/language/design pattern/etc they are going to throw at the next problem. Of course that is all exiting, but what if the thing you are tackeling could be solved so much better, if you just killed your darlings and tried to find the best solution?
Sure you cannot learn $framework you were curious about on the way, but finding elegant and good solutions is itself a skill that needs training.
This is why I never approve criminalizing software bugs.
Imagine if you committed some codes, then turnout codes in that commit can be exploited by hacker 10 years later, are you willing to be imprisoned for creating codes that "fails to achieve a prescribed consensus of acceptability" ?
I can't find it now! Which is frustrating me. Anyone else seeing htis have any ideas?
It was enlightening, and challenged some things we assume about differences. "real" engineering is also pretty diverse, of course, not just civil engineers making bridges and highways.
A world where design flair trumps actually, you know, working, meeting needs.
This is software turned into the movies, or architecture, where the valued work doesn't have to improve anybody's life, and accidents of birth determine rewards. Photographers flock to red carpets.
We already have too much of that here. Fashions dictate web page designs to the near exclusion of the actual purpose of any given site or page. Needed information is artfully concealed behind long downloads and meaningless animation. It is a plague.
Any attention to the visual appeal of these objects is wasted effort. Some things need to just do a job.
Often the particular choices made for "visual appeal" actively detract from correct function. Designers notoriously insist on sans-serif typefaces, despite their objectively lower legibility. Most phone browsers and messagers make it impossible to select a face where lower-case L and upper-case i can be distinguished, thus: "lIIlllIlII". (Maybe they even differ slightly in your view, but which is which?)
It amounts to contempt for users.
Stewart Brand reported that Architectural Digest readers almost unanimously rejected a suggestion that their annual awards incorporate a measure of users' satisfaction with the building. Contempt for victims of designer malpractice is not just common, it is built into institutions.
If the situation is not novel, then this step is only useful for documentation. If this is uncharted territory, any spec with this level of specificity will be thrown out in seconds.
Software does have lower commercial expectations in the areas of reliability and quality, and that does seep into the practices you see in design and implementation. But I don't think it produces a difference in kind.
There are tech jobs which are not creative, of course, but they are not "engineering", whatever title they may have.
Even though I take care, I still end up with non-software things that:
* quickly break
* have bizarre unintuitive quirks of usability
* visual form has been given preference over function
* Low quality even though not cheap
* require hacks to fix them (glue, labels, training, tape, modifications), and often cannot be fixed.
I would love to see some metric of reliability and quality be applied to our everyday things and compare it to our everyday software.
The typical mechanical engineer is calculating HVAC loads for commercial buildings, not working on robotics. In contrast Google is probably only hiring a mechanical engineer for work that’s the tip of the spear.
(In fairness this also goes the other way. The software engineers working on yet another internal corporate CRUD app are largely invisible to the typical physical engineer.)
I guess many devs dream of expressing themselves with programming, just like many film makers, painters, artists dream of expressing themselves with their craft. But what many don't realise, is that after a certain degree of mastery, you know that certain topics demand a certain handling. It becomes less about you, it becomes more about that fascinating aspect of reality you are dealing with.
Whether that fascinating aspect is a certain mood you wanna express, a certain story you wanna tell or a certain problem you wanna solve and create a beautiful elegant solution for is not that different to me.
However: people who always make it about themselves instead of looking at the topics they deal with will rarely create things of lasting value, neither in art or design, nor in engineering. If you chose that framework because you thought it was cool, not because the problem you are solving demanded it, you are still learning and not creating the solution the problem deserves.
Starting off from the beginning: The last thing the world needs is more hero-worship. The idea of a "softwarenova" is somehow even more self-absorbed than "rockstar" or "ninja". It is completely antithetical to open source principles of collaboration, meritocracy, and community. We should strive to include as many people, and as many differing viewpoints, to the creation of software as possible. This is both in the name of diversity and inclusion as well as correctness and reliability. Creating people who are "above the common masses" can only hurt this goal. To the (hypothetical) interviewee's credit, she doesn't like the term.
Secondly, this idea that things in our universe are somehow prematurely over constrained, while in the design universe you work with design briefs and things are looser. The problem in the real world is that things are too often underspecified - Either in terms of latency, or availability, or security, or correctness, or even in what problem you're trying to solve. I find this extremely confusing to be coming from someone who has worked on F# and functional programming. Functional programming has such a strong emphasis on types, and eliminating classes of errors via types. I think if the author went to work in a software shop using C# or Java or any non-functional language he would instantly recognize how this loosey-goosey approach leads to software that is utterly unmaintainable.
Thirdly, while I do agree that a great number of problems are ill-defined, that is not because they are wicked. Wicked problems imply novelty and uniqueness, but the vast majority of software is simply executing some CRUD operation on a set of data. It isn't unique or novel, there are clearly correct and incorrect answers, there are alternative solutions available, etc.
My imagined universe is almost the complete opposite of this one, that is, instead of thinking of software engineers as designers, we should think of them as mathematicians. (Or at least, that is the mental model that would help the world the most). Software should be formally verified, and be proven to be correct rather than aesthetically divined to be correct. We should seek to maximally specify software, much further beyond the way the industry is currently working. We should overwhelmingly reject the notion that certain people have a gift to work on this, and others do not, and instead seek to bring everyone we can up to a level of technological literacy. We should standardize as much as possible to make software easier to use, rather than having this boutique approach. We should avoid thinking that age has anything to do with how well someone can program, young or old. In other words, the field of software development needs to grow up, badly. If the field could live up to the title of "Engineer" things would improve dramatically.
Excellent, you’ve just described a horrible engineer.
An article poor in both concept and execution. A painful read.