Hacker News Reader

Comment by caseyross | Hacker News Reader

Software takes a long time to build because it's always new.

That is, software is trivially copy-able, so there is no reason to spend effort duplicating any software that already exists. (Legal reasons and "not-invented-here" syndrome notwithstanding.) This is a huge different from how the "real" world works, where almost all of the work involved in building, say, a car, is actually just the work of assembling identical copies of that product so that it can be physically sold to more than one person.

The insane amounts of repeated effort involved in building any physical product at scale have a silver lining, in that the repetition enables a very good and stable estimate of how long another repetition will take. Software is the opposite situation --- we only need to build anything once, but not having built that thing before, we don't really know how long it will take.

View on HN

I think focusing on this comparison to the real world assemblies and constructions doesn't yield much insight. The equivalent of a car being assembled is a compiler building an executable from source. You can consider factories to be just very, very long compiles, and forget about it.

The question of why software takes a long time to build pertains to the coding phase, to which real life's analogue is drafting/design phase. It is my impression that the drafting/design phase in "hard" industries can be just as messy and hard to schedule as software development is - we just don't pay attention, because for physical products, drafting/design is nearly free (capex & opex of the assembly phase dominates), whereas in software, the construction is nearly free (compiler time is too cheap to meter).

It's true that, on the surface, software is "always new". But after doing it for a while, I have this sinking feeling that a lot of it is old and repetitive work that we hadn't yet automated - there are deep similarities between pieces of code we're writing, but we don't have a language to fully capture them and abstract them away.

I thought the original article was very poor. There's no real data, no references (except Fred Brooks), and the graphs are just nice shaded opinions based on nothing much.

It fails to ask why the complexity of tools and systems grows over time, and whether this is matched by improvements in UX.

It also fails to note that one of the goals of abstraction is to hide inherent complexity. And in fact this happens time and time again in engineering and product design - an insight that starts off requiring near-genius levels of original mathematical creativity is packaged into commodified tools that make it easy to do a certain job.

Sometimes the tools are aimed at engineers, and sometimes they're built for the public. But because no one expects to literally reinvent the wheel, they all package genius in their different ways.

Except software. In software teams keep working to reinvent wheels completely every few years. Some are arguably more refined than existing wheels. But they're all wheels.

They're not a new kind of thing - like an engine. Or wings.

So I agree. Software has mediocre lumpy wheels They're mostly back-references to existing software, and not so much attempts to commodify and simplify standard problems - like security, reliability, scalability, and UX.

There are packages for all of the above, but they're rarely clean and elegant - in the sense that someone has really understood the problem and designed a maximally effective but minimally complex automated solution.

Worse, there are no processes for efficiently abstracting a domain. And that's what's missing.

I agree. Software is "always new" but it often shouldn't be.

I work for an engineering services firm that does mostly embedded systems. I push hard for reuse, basing our work on standard platforms, etc. Unfortunately, client needs vary so much -- one client might need a wearable, another wants a large system that is mounted on a skid, another needs something to sit on a desktop, all with hugely different feature sets -- that it's difficult to find a small set of devices that we can reuse. So one of our big problems is that we're very often bidding on a project that uses a device, or a technology we've never seen before and that's where the uncertainty comes in.

Patterns certainly help, higher-level languages would definitely help (I'm the only one in the organization programming small processors in C++, everyone else is only comfortable with C), but that's a slow, risky change.

> for physical products, drafting/design is nearly free (capex & opex of the assembly phase dominates)

It really depends on the industry. It's particularly bad in heavily regulated industries (pharma, aerospace, ...). On the other hand, creating a blog is probably the equivalent of designing a new t-shirt.

Fair enough. I haven't considered regulatory impact. On top of that, I wasn't thinking about pharma, or a lot of formulated products.

That said, setting aside the regulatory burden, from the little insight into the internals of chemical processing I had, there may be larger costs in designing and tuning the "build pipeline" (all the coolers, valves, reactors, etc.) than in designing the formula itself. That's just a consequence of scaling up being expensive where your production involves manipulating matter.

> The equivalent of a car being assembled is a compiler building an executable from source

I disagree. The equivalent of a car being assembled is copying the binary executable from one storage medium to another; building an executable from source would be the equivalent of building the production process for a single car model, while coding is the equivalent of designing and testing that model.

This is where the analogy breaks down somewhat - so I propose you imagine a universe in which `cp` doesn't exist and you have to build each copy of an executable from scratch :).

(Perhaps you're signing each copy with a different key, or something.)

I insist on connecting compilation to construction of individual cars, because the source code corresponds to the engineering plans of a car, and not to the plans of the production process.

Sure, every car has a unique VIN and every running process is a different constellation of electrons (or just the one, if one-electron universe is assumed).

But looking at getting functionality to users, looking at the business problem, the time/effort/risk is spent on car design and software development, and much less time and risk is in the copying business. (Yes, Tesla recently had car copying problems, but they brute forced their problem, and now they are pretty good at factory copying too.)

This analogy is flawed, you can't capture insight from the flaws in it.

Best comparison (car/machine manufacturing vs software development) I've ever heard on this topic is as following;

While producing a machine, there are two steps;

1) designing the blueprint (takes a lot of time, unpredictable)

2) mass production (takes a lot of time, predictable)

Those steps in software development;

1) building the software (takes a lot of time, unpredictable)

2) deploying the code (takes virtually no time, predictable)

So the real difference is this: software is almost all about doing something new, each and every time. It's all flesh and has no bone: e.g. devoid of a long, very stable and predictable mass production stage. It's almost all about R&D, all of it's length. Hence 98% of the process is novel work therefore takes time and hard to predict.

To note. All engineering has this problem.

Yes, building cars in a factory yields repetitive results, but what about all the different engineering projects. The truth is that this is not a new problem, we just lack discipline of fields that have been refined over thousands of years.

Because of the quick idea to prototype rate, product research and refinement takes a back seat. All engineering projects have massive research before the first stone is placed. Nobody mid way through building a sky scraper decides "oh, do we need to redo the foundation?"

I would also like to note, nobody is gonna give a self-taught architect who's 19 the ability to build that skyscraper foundation. Each building technique is developed, tested, and then once determined good, is actually used by others, and supervised by an industry veteran with underlings who learn and eventually themselves supervise such projects. Meanwhile in software it is the opposite.

Not to say that there isn't a reason software grows so fast. Things change daily. If software worked like architecture we'd be licensing even the right to use ruby on rails-style modeling. But software works differently, thus faster iteration, thus more loosey-goosey, thus harder to estimate.

> I would also like to note, nobody is gonna give a self-taught architect who's 19 the ability to build that skyscraper foundation. Each building technique is developed, tested, and then once determined good, is actually used by others, and supervised by an industry veteran with underlings who learn and eventually themselves supervise such projects. Meanwhile in software it is the opposite.

It's a bit less black and white: Any 6+ year old might build his own tree house, while in reality no one will let a self-taught kid write the control software for a nuclear power plant.

> Software takes a long time to build because it's always new.

And yet it feels like authentication and authorization code need to be rewritten for every application.

Not in my experience; basic session auth is easy enough, and oauth has been around for a while now.

What keeps giving though is CRUD. Naming things, forms, APIs, persistence, every time, rinse and repeat.

That's because it is mostly integration that is done, not so much construction. AA needs to permeate the whole project, from the details of a function call to high level design like making sure an actor can call another part of the software stack to perform an otherwise unauthorized operation.

"Building" software can mean any number of things. When you get down to the specifics, there is no getting around using specific language.

I don’t rewrite bcrypt, and authorisation is intrinsically linked to your data model.

Only vendor-specific code, so it's not that bad. What business do most people have rewriting auth stuff?

If that were true, it wouldn't take long to build from scratch a functionally identical clone of an existing application. Yet it does.

The conclusion is that the tools to build software from a known specification to working code are sub-optimal.

That's like saying a Ford F150 is functionally identical to a Ford Model A because they are both cars. The specific implementation under the hood requires a lot of work. And very rarely do people ever make functionally identical clones without massive refactoring to allow for extensibility.

"implementation under the hood requires a lot of work"

"people don't make functionally identical clones"

"massive refactoring to allow for extensibility"

You're enumerating the factors that make my point true. Those are some of the reasons why building software takes a lot even if you know exactly what it is supposed to do.

You can compound that to the problem of figuring out what the software should do, which classic IDEs are also bad at. Online collaboration and prototyping tools that put the program in front of stakeholders early; they're like car makers' focus groups that were used to design cars tailored to a target demographic group.

Building the first F150 is an act of creation and design. Assembling the second one is finding the same parts and doing the same assembly steps, the thinking is already sorted out.

Building the first Model A means going back to the drawing board because it's not the same. Even at a smaller scope building a F150 with an updated exhaust means going back to the drawing board for that part, and making sure it keeps working correctly with all the other parts.

It turns out that with software, once you have one fully assembled "F150", a second identical one can be created out of thin air (cp / git clone / run compiler / whatevs). So the effort is ~100% on thinking and design.

Now why would you want 2..n F150s? Because you want to have them in the wild doing something useful. That's making them available and distributing them and having more of these as needs increase. That's cp'ing your code onto servers and deploying and running it and making it reachable. You do that with DigitalOcean, Chef, K8S, or whatever AWS service du jour, and hopefully you automate as much of it as possible just like we have car factories with robots to not build these with human hands.

Deploying and spinning up more of that same code so that it keeps up with the scale of whatever it needs to do... that's ops, and that's basically the only part of software that remotely resembles industrial work. The part before that is development, as in "research & development", and that's 100% not "industrial" by any stretch of the imagination.

Thinking that developing software is like building a house or a car and "industrialising" things with waterfall or Kanban on SCRUM and pulling bogus time estimates to completion is the biggest lie ever, just like pulling out the next vaccine or devising a new maths theorem cannot have any reliable time to completion because you're constantly up against unknown unknowns. This field just flat out doesn't work that way.

> The conclusion is that the tools to build software from a known specification to working code are sub-optimal.

That may be true, but I don't know that your example supports it.

Cloning an app doesn't mean you had a specification for the app. The tools to build a known specification from an existing app are sub-optimal, as are the tools to build a known specification from scratch.

I've only rarely in my career had the benefit of a written specification.

> The tools to build a known specification from an existing app are sub-optimal, as are the tools to build a known specification from scratch.

That actually reinforces my point ;-) All the steps in the toolchain could benefit from more agile interactions, allowing the programmer to spend less time fiddling with syntax errors and recalling which functions need to be applied in what order, and more time evaluating and fixing errors in the current logic as written. That would expose what the build program is doing and how it differs from the expected intent.

The online notebooks used in data analysis (Jupyter, Apache Zeppelin) are a step in the right direction; IMHO their approach of 'data is always readable besides the code processing it' makes for a better introspection infrastructure than the REPL loops of old.

Basically we're going back to what delphi had 30 years ago. I understand that scalability concerns warrant a different approach, but a lot of the software we write will not deal with google or netflix scale.

I keep going back to the example of elixir/erlang (mostly because I'm getting into it now) but the tooling that is available in erlang is much more powerful than any newrelic or what ever monitoring/introspection tool we use now. And we've had it for years, but we've just ignored it.

While your second sentence is definitely true, there are some subtleties that also matter.

Car specs are (certainly as opposed to software specs) very exact and constrained by dimensions, chemistry and physics. None of those have version numbers.

Creating a particular model of a car is decidedly single-paradigm. The same cannot be said for a functionally identical clone. You could make a functionally identical clone of an app in OOP or FP, nodejs, Rust or C++, MVC, MVVM or MVU, as a polyglot SPA + backend combination...

All this to say: software as an industry also lacks standardisation, process documentation and documented best practices. Software "best practices" are more like "accepted dogma at the time" or "least likely to bite us later", not "will work 99.999% of the time". Software as a product is immaterial to start with.

I don’t want to sound smartass but building an identical clone of an existing application is very fast and often totally automated. That’s what your compiler chain does.

What does the compiler do if you start "from scratch", i.e., without any source code?

Change the criteria that you should rewrite it in another framework and the point still stands. You already have a perfect specification in the existing code, no unknowns at all, just implement it. Yet it still takes months or years.

A good physical analogy is the historical Soviet recreation of USA B-29 Superfortress as Tu-4, based on copying all the parts (based on 4 captured aircraft), but needing to rework to the metric standards (i.e. so wherever you have, for example, 1/4 inch i.e. 6.35mm sheet metal or fasteners or whatever, you have to pick 6 or 7mm instead (because your "different framework" doesn't supply 1/4 inch stuff), which changes the weight distribution and structural integrity, which requires additional tweaks). You're making a functional copy, still took more a year just for the designs. The same with other physical objects - as long as the underlying framework or supply chain changes, it tends to require re-engineering.

A very interesting post on this topic: https://vvvvalvalval.github.io/posts/2018-07-23-datascript-a...

At the risk of sounding like another, it can be simpler than that - just install the already compiled software. In a sense, the thing about composing software using "microservices" is about "install it and get going".

Parent wrote "functionally identical", not "identical".

And add to that the number of custom software are many times more than most things in real life. Nobody will asks for their BMW to transport a cow, or for their BMW to connect / communicate with other's Mercedes.

But we often asked for accounting app to able to handle employee's payroll and paid leaves.

Then with that many demand for custom software, we also got many times software manufacturer than car manufacturer. Inexperienced manufacturer (programmer) will make development even slower. Not to mention that the client even don't know what they want.

Every once in a while when I see an example of an absurdity that wouldn't happen, I head off to Google to see what I can find.

> Nobody will asks for their BMW to transport a cow

How about a cow transporting a BMW?

https://carnewschina.com/2013/02/22/bmw-owner-in-china-is-an...

Reply on news.ycombinator.com