Hacker News Reader

One aspect of the problem is that whenever I need to build a particular system, the parts of that future system that are well understood system, with well understood tooling take almost no work - those either are available as part of a well-known framework or can be trivially included from a well-known library.

All the work that remains, all the work that consumes most of the time of programmers is the work where either that parts of the system is not yet well understood, or where well understood tooling is not available.

I'm not seeing the "factory" or "bricklaying" work anywhere, it has been automated or converted to reusable components or frameworks. Whenever I see people spending a lot of time on systems that could be "factory" work, the situation is that it could be "factory" work if and only if the system requirements and details were properly understood, and these people were mostly working and re-working on understanding and misunderstanding what's actually needed - and there's no way around that. Once they've understood it, any "factory" work part is quick and trivial to the point of being irrelevant in metrics.

So my point is everything that we need from software engineering practices is tools and practices to manage the part that isn't "factory" work, the process of understanding the details of the systems that we haven't built yet.

There is a third class of problems in coding that, at least for me, takes most of the time - definitely more than exploratory work. That class is what I call code bureaucracy - the part where you write code that shuffles and transforms data between components, or manages components. The components themselves may be well understood and already built, but they're probably nowhere near each other and have incompatible interfaces.

Note it's not the same as "glue code". Writing glue code is a small subset of code bureaucracy, that happens only at the boundaries. True code bureaucracy begins when you discover that two components that need to communicate are on opposite sides of the system, and you need to carefully route a new path between them through half a project.

You are using "factory produced" code every time you use any optimizing compiler for typical general purpose language.

Compiler optimization phase ordering is NP-hard for every program being optimized and compiler writers cannot abstract that away to reusable components or frameworks.

> Most companies don't want "software artists" anymore than they want "artistic bricklayers"

I recently had brick laid in front of my house. When the bricks arrived, they were not cut with the precision that the landscaper had expected. They were mostly rectangular, but with enough variation in size that laying them naively (assuming they are interchangeable) would result in an ugly, uneven walkway.

The landscaper worked out what order to sort + lay them in to minimize the impact of the uneven sizes. I'm very happy he did; my walkway looks stunning for it.

If you are looking to get landscaping done, live in western Massachusetts, and appreciate someone who takes pride in his work and will take the time to do things right, I have a glowing recommendation for you. (Do be prepared to chase him down about the paperwork on both ends of the transaction; the business side of the business was not his forte). My email is in my profile.

---

I am not sure if that exactly contradicts your point. You said that's not what companies want. As a human, though, I very much appreciate work well done, when the real world inevitably defies our abstractions.

Strongly disagree, software engineering is nothing like factory work, or rather, companies who view it as such and only have "predictable, maintainable error free software" as their objective will have their lunch eaten by disruptive companies who understand software is a creative tool to deliver value and should be treated as such.

You should read

https://iism.org/article/why-are-ceos-failing-software-engin...

> Applying replication management techniques to creative work is a systemic mistake that actually disadvantages companies that are developing software – this is because the skills, workflows and founding principles for discovering new value are almost entirely different from those needed to efficiently replicate value.

> It should be clear that not having an effective creative management system is a huge gap in the value fulfillment cycle. If this gap remains unaddressed, the market will solve the problem by investors, over time, moving their money to entities that are discovering new value at higher rates. So, if you have this kind of systemic void, it’s really in your best interest to address the gap as soon as possible, and I would encourage you to give some serious consideration to empowering an effective creative management system.

How are you going to sell unpredictable software to a customer, and if you do manage that, how are you going to keep that customer without maintainable software?

Ask Facebook? "Move fast and break things" is in a literal sense orthogonal to predictable, maintainable software.

If you are adding value and focusing on your customer's needs, they will put up with a little more unpredictability and extra maintenance.

Again, you should really read that article, it says it better than I can summarize in an HN comment.

Part of its definition of value is that you're addressing unmet need at a rate higher than you are introducing chaos.

So the levels of predictability and maintenance in your software are tradeoffs of other ways to add value (insert "always has been" meme here). They are neither necessary nor sufficient for delivering value.

Facebook changed that in 2014 because it didn't work.

They didn't say anything about unpredictable software. There are things you can do to systemize the software development process but if you go too far you reduce the ability of your programmers to find the optimal solutions. Creating and managing complex systems is an art.

You strongly disagree based on feelings and conjecture?

As engineers, we should know to only use strong words one way or another when overwhelming evidence supports our claims, which is clearly not the case here.

Sorry, where did my feelings come into play here?

I cited an article with a fairly persuasive argument around the value fulfillment cycle of creative work such as software engineering and quoted a bit from it.

> Most companies don't want "software artists" anymore than they want "artistic bricklayers" or "artistic aerospace engineers". What they want is predictable, maintainable error free software that still works when the "artist" moves on.

Well that's a bloody shame because they have to hire actual humans, and we're a smelly, imaginative lot. If they want an iron they should go to Target.

Exactly I'll keep doing the creative work and let the brick laying happen on UpWork. If they confuse me for one of those then it's their loss and I'm moving on

The iron is the shitty no-code, low-code solutions and I imagine it'll follow the same path the iron has. It hasn't changed much, it does the job, but if you want your stuff to look really good you take it to a specialist, AKA a dry cleaner who follows on the facade a similar process that is in reality entirely different.

There are so many falsehoods in this, I don’t know where to start.

1. Adding missing specs is less creative than designing a software system. It’s weird that adding specs to a product requirement is your example of creativity.

2. The correct solution is the boring one? What on earth are you basing this assertion on. Is your worldview so reductionist that you actually think this kind’ve statement contains any true or valuable information?

3. Predictable, maintainable, error-free software is more dependent on choosing/formulating the correct abstractions (which is very much a creative process) than following a checklist of “bricklaying” steps.

4. You seem to be confusing boring with correct and fun/pet with creative. Neither of these are equivalent.

It’s a shame because I assume you hold these opinions out of ignorance, which means you’ve never had the chance to build software in a creative, challenging environment where following a checklist would certainly result in failure. And I sympathize with anyone who hasn’t gotten to experience what’s truly special about this profession: the creative process.

Companies don't want employees at all.

But there is what you want, and there is what you can get.

> It just means that a professional should choose the accepted method over the fun one when given the chance.

Er, but that's rather what's being argued about, what the accepted method is. Funny that everyone making that argument tries to smuggle it in as an assumption.

> What they want is predictable, maintainable error free software that still works when the "artist" moves on.

Producing such software is the essence of the art! Managing the complexity of large software systems as they grow demands the ability to creatively invent elegant abstractions that factor that complexity into manageable chunks. Without that factoring, the complexity grows until no one understands the code -- with entirely predictable, and sadly familiar, effects on maintainability.

> software engineering IS "factory" work because its the grunt work of building what should be a well understood system

So what are you implying here? That we should go the Waterfall route and have 100% clear and finished specs? Good luck with that buddy. You're just shifting the real problem outside of your "software engineering" space.

In my 20 year career, I've never built a "well understood system".

Some things can be built 10 times faster with the right idea. Some features can be supported by a single, simple one. Some screw-ups can take forever to fix. 100% code coverage won't give you bug free software.

Building software is all about the skills of people (developers, analysts, managers, testers, ...). The better the people, and the more power you give them, the better your project will go. If you hire bad people and have to put processes in place so they don't screw up, you're in trouble.

Your post fixates on "artistic or craft endeavour" defined as something frivolous (i.e. "fun" solutions). I disagree with the use of the word artistic (and craft), but the word only appears once. Intuitively, the point is development requires a lot of "design" work that is creative, an logically so; in the same way medicine is.

In your bricklaying example, how does a bricklayer operate without an architect? The equivalent of laying bricks is typing characters.

A developer has to decide what to type, not just type it.

"What they want is predictable, maintainable error free software that still works when the "artist" moves on."

How about fast software?

To get fast software today. You're going to need some artists.

I think the evidence shows that "fast" software is not actually desired by the majority of companies, otherwise they would put more emphasis on using compiled languages, optimisation and reducing technical debt. Sure, everybody will take a speed improvement if it's free but most companies will go for predictable, maintainable and error free software over speed every time.

What matters for companies is whether they can accomplish tasks faster than they can do them today. And it's rare that the difference between executing Python or C code is the bottleneck in that process.

A lot of that maintainability and lack of errors will come from proper design, having a good design where the complex problem is partitioned into many really simple problems, resulting in easily testable code. And that code is easy to optimize, because it's easy to make changes when you have a decent test support.

Now this software won't be as fast as what's possible, if that is needed the optimizations will turn it ugly again in many cases, but it will be decently fast. This all starts with good design, proper intuition about which algorithms to use, and a good dose of creative problem solving.

I spun up a very simple rest API thatreturned an input parameter, and ran it under load, using asp.net and express js. There wasn't any architecture or design, it was one function. Node/express had 10x less throughput than the asp.net version, with 99% being multiple orders of magnitude larger than the asp.net version.

No amount of clever design will outrun that gap.

Of course you can hide the biggest complexity behind a single function.

What I was getting at is that if I see code with a lot of duplication, often it's not only the code that's duplicated, but the runtime as well. Then you have people using nested for loops where a dictionary would do to look it up.

Things like this, in the same language and same framework, make things on average faster, and simpler.

And if that piece of badly architected code is fragile and breaks on every change, you will stop trying to find even the low hanging fruit of optimizations that usually pay off, because every change affects multiple different files and you have no tests to see quickly if it still works.

Thinking about your reply again, it's actually the perfect example of this. Since it was a one-liner, it was so simple that it was easy to exchange it for a different one liner with a different framework just to see if it's faster. If you had to spend weeks on replacing frameworks you wouldn't have been able to do this low-hanging fruit of an optimization.

You've totally missed my point here - the fact is that no amount of low hanging fruit optimisations in the node app, or architectural improvements, or clean code will _ever_ close that 10x gap. Almost all of the decisions you make after the fact (e.g. DRY, DI, composition, and even in many cases data structures) are meaningless on both a macro and micro level compared to the very basic decisions you make - what does it run on.

[0] https://stackoverflow.com/questions/40070631/v8-javascript-p...

"To get fast software today. You're going to need some artists." I don't think that's been true for a couple of decades. Performance issue resolution are now less "The Story of Mel" and more "throw processors, memory and bandwidth at it until it has what we need." Security requirements are more likely to be your limiting factor than raw speed. There are certainly some niche areas where you're working within tight resource limitations (I work in embedded stuff), but Moore's law means we can get MS software to run in an SoC, which means we stopped needing artists a while back.

Sounds like the software systems you are talking about are so mainstream and well understood that it might be simpler to buy something off the shelf. Now that more and more pieces are available as SaaS solution my intuition is that organizations should only focus on those pieces that are unique to the problem they are trying to solve. Rest everything should be acquired as a SaaS solution.

And yet it is exactly in the most, 'boring', 'tool constraint', 'factorized' and 'process goverened' environment of enterprise software projects that the failure rates reach spectacular heights.

The factory work is the sort of things that have been written into framework these days. Learn to use the appropriate one well and it should take away much of the grunt work.

> What they want is predictable, maintainable error free software that still works when the "artist" moves on.

That is where the craft of software engineering comes into play. It's relatively easy to make code that works. It takes a lot of experience to consistently write code that is intuitive for the next developer to understand quickly. Keeping the complexity low while keeping the same functionality is an art.

Because, the vast majority of software engineering IS "factory" work because its the grunt work of building what should be a well understood system, with well understood tooling.

I'm not sure it is, depending on which majority we're talking about.

We size in complexity because that's where the time goes, the extra testing, the rework, the long reviews, the risk.

If there are cookie cutter lessons to be learned software tends to abstract those problems away to the point they're no longer problems.

While I agree with some of your arguments, I think your main mistake is calling it "factory" work - because IMO it isn't a valid analogy (and analogies are powerful! :) ).

"Factory" work implies a far more constrained and monotone environment.

Craft work is more apt, I think.

I mean, to illustrate: Most of the work might be seen as just connecting pipes, but I'll be damned if a highly skilled and experienced plumber isn't worth their weight in gold. :)

Your right, I should have been clearer, both factory and craft imply the wrong thing IMHO.

I think a better term would have been "skilled". Like the bricklayer, or plumber you don't want "creative" so much as reproducible quality which comes from experience. So in that regard craftsman is partially right too. Traditional factory work OTOH had painters, machinist, welders, etc. All of which were highly skilled labor, but their jobs like a blacksmith making nails or plumber brazing pipe frequently were just repetitive application of that skill. One might even say that these painting reproduction shops are in this camp as well despite being "art".

So its the same with software, yes you have a framework, but the act of building out forms using that framework to CRUD up some data somewhere is frequently quite repetitive despite differing colors and button placement.

It seems you have a pet peeve about pet projects. The parent comment didn't say anything about pet projects.

If you pay software engineers to code up a well understood system with well understood tooling you're doing something wrong imo. You should instead pay for a COTS solution. Software engineers solve the parts of your business problems that don't have well understood solutions yet.