Why does software cost so much? | Hacker News Reader

Why does software cost so much?

insights.sei.cmu.edu

56 pointsBillPollak67 comments

Was able to get the site to load after several tries, which has led to a humorous juxtaposition with the comments here. Other commenters are saying that they know what drives software costs, and while I agree with some of the comments, there's also stark irony as the linked research is explicitly aimed at trying to debunk "gut feeling" about correlated data that leads to higher costs and discover causation data:

>> "If we want to proactively control outcomes, we would be far more effective if we knew which organizational, program, training, process, tool, and personnel factors actually caused the outcomes we care about. To know this information with any certainty, we must move beyond correlation and regression to the subject of causation. We also want to establish causation without the expense and challenge of conducting controlled experiments, the traditional approach to learning about causation within a domain. Establishing causation with observational data remains a vital need and a key technical challenge."

>> "Armed with this new approach, we are exploring causal factors driving software cost within DoD programs and systems. For example, vendor tools might identify anywhere between 25 and 45 inputs that are used to estimate software development. We believe that the great majority of these factors are not causal and thus distract program managers from identifying factors that should be controlled."

Since down, didn't read: Don't assume you know the answer. The point of the research is to prove what's actually a causal driver of costs, not just an unrelated correlation winking suggestively at it.

It does look like a really interesting avenue of research. I see a lot of statistical "evidence" presented as justifying interventions without even hesitating to consider if they've got it the wrong way around in terms of causality.

It seems it will be focused on large scale products out of necessity because of the availability of structured data and public disclosure.

There is one preliminary paper linked in the article, which I haven't read: https://dl.acm.org/citation.cfm?id=3172876

It is really interesting to see ways to establish causality without a randomized controlled trial, since in a lot of circumstances an RCT is impossible or unethical, and because for many decision making purposes you need to evaluate interventions, not just relationships.

Alternative: https://outline.com/EV6FPg

http://archive.is/https://insights.sei.cmu.edu/sei_blog/2018...

Internet Archive doesn't yet have the page, though eventually it should show here: https://web.archive.org/web/*/https://insights.sei.cmu.edu/s...

I've been experimenting with what one may call "CRUD shorthand", which is a way to define typical CRUD activities in as short a way as possible. However, changes in UI/UX expectations and/or fads generally muck up the consistency. The industry keeps reinventing the wheel such that we can't stop to master and simplify one kind of wheel: we throw it out for the Next Great Wheel and invent different ways to do the same CRUD things. Some changes are good, others are a waste and distraction. People do judge books by covers, and that costs us.

Another "problem" is that software tries to model the messy "organic" external and internal office politics, making the software messy. Either make the organization more logical, or accept messy software that tries to mirror messy human and organization relations.

CRUD alone doesn’t provide much value to an average company, the real value comes from managing and automating the workflow, which is of curse the expensive part of developing software because it requires full knowledge transfer from a company that likely has never stopped to understand their process to developers that have varying grades of familiarity with the domain

I generally consider "workflow" as part of CRUD, although there are more literal interpretations of "CRUD". But changing the workflow shouldn't be that hard if the stack/tool has a standard way of configuring & managing workflows and user roles.

Workflow is usually so much more than an object state. You might have required documents or a proof of work that go in attach to the state change, some state change roles might be asymmetrical (i.e. only this role can roll back a state transfer) so permission need to be dynamic depending on object states, and then you have good ole transformative batches on top which are squisitely custom.

I mean, I know the problem quite a lot, sice one of my many backburner ideas is a workflow centric self service crud builder :D

If by “software” you mean bespoke custom software developed especially for you, the reason it’s expensive is probably mostly because of bad development practices.

If by “software” you mean off-the-shelf proprietary products, the price you pay for it is entirely an arbitrary construct chosen by checking what other market players charge for similar software and seeing what buyers are prepared to pay. Absolutely no other considerations go into pricing a software product, since the software is literally free to produce, plus the cost of packaging.

(I could, for completeness, as a clarification to the latter point, mention that the cost of hiring programmers etc. does enter into the equation at the start of a proprietary software project, but once the cost of development has been paid, it’s a sunk cost and no longer relevant for making any pricing decisions.)

Ignoring the huge cost of R&D (the sunk cost you refer to) is disingenuous. It isn't even free to produce the copy of the software. There is marketing, bandwidth, upkeep. Maintaing backend servers and so on. I agree, for the most part the price of the software doesn't have a lot to do with the cost to develope it, but you cant ignore the R&D costs.

I’m not ignoring the R&D costs. But a sunk cost is a sunk cost¹; wanting to charge more for software because you worked hard on it is a sweat-of-the-brow argument, and it doesn’t work. Nobody cares how much money was spent on the R&D which went into some software. People care about what they can use it for.

1. https://en.wikipedia.org/wiki/Sunk_cost

I mostly agree with you, but to the extent that your software competes with other, similar software built by companies with similar cost structures and addressable market, the cost of that sweat factors into their costs as well, so does have a (loose) connection to the viable costs for that software in the market.

Why does software cost so little?

When you look at the value created by tech, the productivity gains from buying solutions instead of rolling your own, the possibility to focus on your own competencies vs getting distracted by maintaining your own re-invention of the wheel...

Or maybe, just maybe, it is priced just right.

The buy-versus-build decision is not clear cut. Pre-packaged solutions often have a lot of "buttons" and layers that are distractions and/or extra steps that are not otherwise needed by a particular customer, and thus slow things down.

For example, a given org I once worked for had a 1-to-1 relationship between subscription user and payer (bill-to). But the pre-packaged software implemented it as a many-to-many relationship to be more flexible for different shops. One can use this to emulate a 1-to-1, but it's more layers, screens, and data tables than a dedicated 1-to-1 design. (My job was to build reports based on those data tables.)

Good home-grown software beats bad purchased software, and good purchased software beats bad home-grown software.

And what if it's a decision between both goods? Does good purchased beat good home-grown, or is that a matter of which costs more?

My point was that neither type is inherently better just by being a member of that type. The devil is in the details and both can be good or bad for a given organization or product need.

This. The original question doesn't make sense, unless rephrased as

Why software costs so much compared to not having said software.

Edit: and now that I can read the article, DoD says software costs a lot. DoD still pays for software. Supply and demand. If DoD pays said prices, the software is worth that much.

Site is down, but there a lot of things to contribute to the cost of software:

1. Business doesn't know what they want, or can't communicate it well. 2. Business reluctant to go with startups and rather pay out the nose for large, entrenched and very expensive companies (Oracle Applications, SAP, rent chargers, etc) 3. Scope creep, scope changes mid development. 4. Negative productivity developers (who make things worse) 5. Absolutely poor hiring practices industry wide. 6. Vendor lock in. 7. Poor development tools and languages (weak-typed js makes refactoring 10x harder). 8. Overburdening development processes. 9. Inter-departmental friction for developing software. 10. Hands off, ivory tower management. 11. Trying to make software fit everybody. 12. Security concerns.

These are just off the top of my head. There are many many more. Having said all that, that's why software costs what it does. As far as it being expensive, if it's more expensive than the manual way of doing it, don't buy it. If your company is competent enough to roll your own, do it. It's a hell of a lot cheaper than folders, paper and filing cabinets (just ask the VA).

13. It's a market for lemons [1]. Schneier wrote about it in the context of security related software, but it equally applies to bespoke stuff like CRM or SCM.

[1] https://www.schneier.com/blog/archives/2007/04/a_security_ma...

I would add to the list: 13. CV driven development 14. developers with no interest in the underlying domain they are trying to solve making bad design decisions 15. excessive bureaucracy of IT department & rigid policies

"CV" as in Constant-Velocity? Sounds like an Agile concept ;)

You're probably being sarcastic but I think the parent meant Curriculum Vitae. Making technical decisions based on what will look good on your resume instead of what's the best way to solve the problem at hand.

Chicken and egg perhaps; as a Java contractor back in the day, I noticed the eventual creep of Ruby on Rails stealing some of it's limelight, then JavaScript or whatever the new hotness became (not saying JavaScript is in any way hot there).

As a contractor, I was compelled to learn those technologies because the companies were demanding these skills, not because I had a desire to learn another scripting language or whatever. Of course, it's often (senior) developers within the company that push the new, less-well-travelled technologies and the cycle repeats.

I guess the lesson is, don't let frontend developers dictate that JavaScript should be on the server, or is in fact a good language for desktop apps and so on.

Not sarcastic - just going for a low-hanging pun.

Answers:

Because it costs so much to make.

Because it has, in some cases, a measurable benefit in productivity, and is being priced accordingly.

Because it's something that's still relatively new, and like anything else, things slowly get cheaper as time goes on, but start out priced decently high.

Because 'big ball of mud' is the most common system architecture.

I love PG's description of development in the Python Paradox: "You push blobs of source code around the way a sculptor does blobs of clay."

http://www.paulgraham.com/pypar.html

> people don't learn Python because it will get them a job

Maybe not in 2004.

We have a piece of software for handling child cases in the public sector in Denmark. It cost around 100 million Danish KR to develop, it’s up for renewal, and because the old system was build on mainframe tech that nobody wants, it’s up for a do over. The new bid was for something around 120 million DKR. Software is called DUBU if anyone wants to google the exact figures or want to know more.

Around 30 municipalities run a record keeping ESDH software called SBSYS. It’s not open source but it’s owned by the muniplacities who hire private companies to maintain and develop the code.

A couple of years ago they decided to build a DUBU like module, which cost them around half a million DKR, and it operates so well that it’s managed to replace DUBU without any ill effects in our third largest city.

Now, the base of SBSYS provided a nice base to save documents, but I simply can’t imagine what those 119 million are paying for. If you need 119 million to classify and save documents on cases in a database you probably shouldn’t be developing software - yet one of our most prestigious software houses won the contract.

Honestly it looks like theft of tax payer money that is possible because the deciding politicians aren’t tech savvy enough to know when they are being ripped off. Especially because it’s likely to be both delayed and have it’s costs increased before it’s implemented, at least if it follows the trend of most our major software projects (including SBSYS).

Obviously I can’t be as blunt in my professional life, and I haven’t actually seen the deal and I am speculating, but it just seem off.

I have worked on software in domains including DARPA, defense, and robotics, among others. While I agree with many of the things said here, from my experience, the single biggest factor affecting the cost of software is complexity.

When I say complexity, I don’t mean the kind that is inherent to the problem being solved. What I mean is synthetic complexity; that is, complexity that is artificially and unintentionally introduced by the solution.

This kind of complexity is common in software and usually stems from 1) poorly defined or rapidly changing requirements, 2) technical incompetence, 3) lack of technical oversight, code review and feedback loops, or 4) mismanagement of technical debt.

The larger problem is that the aforementioned problems, in concert, are corrosive and toxic to the culture of the work, and that in turn becomes its own cost factor in terms of the care and feeding of teams.

The largest problem is the realization that the factors I’ve enumerated here are clearly observable while being difficult if not impossible to measure. Cyclomatic complexity scores will tell you that you've got complexity issues, but not that you've reimplemented half of the standard library badly. Thus, these factors are among the reasons that software remains as much art as science.

Too much complexity, too many coders, too much code.

Too little restraint.

Hm - software costs a lot because programmers are hard to find, and it takes a long time even once you've found them. Even now, more than 30 years after it was pretty commonplace for an individual to have their own computer, programming continues to be difficult to learn and master - there's a whole cottage industry of people who insist that this is some form of "gatekeeping" on the part of the programmers and that programming is really easy, but programmers are conspiring to make it seem harder than it really is. Their attempts to "simplify" programming (usually with something along the lines of a drag-and-drop GUI flowchart builder) end up being unmaintainable, unworkable, hideously expensive, and extraordinarily popular with the people who believe that they stand to profit if programming can just be reduced to (as they're positive it really is) a minimum-wage burger-flipper type McJob. But even when they finally get around to swallowing the bitter pill that programming needs to be done by programmers if you want the programs to work, it continues to take a long time to produce because the same people who are absolutely convinced that programming computers is as simple as tying one's shoelaces make a career of putting up roadblocks in the name of "efficiency" that have the exact opposite effect, slowing down the whole process to a snail's pace as the individual programmers are expected to predict the future and state precisely what they're going to do and how long it's going to take, sometimes months in advance. The result is that nothing gets done _except_ for the stuff that's trivially predictable.

Not always, but often, brought about by unclear, underspecified, ambiguous, constantly changing requirements.

> Too little restraint.

100% this. Always, always favoring the ease of the individual developer or meeting short term goals.

An approach I'm finding of interest to goods and services with long-term price trends (increasing or decreasing) is to focus not merely on the supply-side dynamics, SLOC or COCOMO models, for example, but on demand side.

Software costs are increasing because the market (or government contract purchases) exhibit a willingness to pay. If markets are efficient, rational, and accurate,[1] then this suggests that prices rise due to use value available to be captured by the expense.

And if not, this suggests specific market pathologies, particularly around the realm of information, systems, complexity, and risk.

A discussion of which I'd be very interested in seeing out of CMU and SEI, who seem not to have moved much from when I was first researching software development methodologies and failures in the early 1990s.

There's also the success (and failures) of nonproprietary software development, particularly the Free Software world. Excellent for infrastructure, less so for GUI applications and specific high-touch business solutions, though the infrastructure components frequently comprise major elements of same.

________________________________

Notes:

1. Assumptions I do not as a general rule hold, thogh I'm entertaining them here.

Or so little, with regard to apps.

Many consumers scoff at spending $5, or even $1, on an app, but they'll willingly blow it on a drink that will be consumed in 10 minutes.

Consuming a drink is a social act (i.e. you do it among other people, who notice that fact). We're social creatures who are wired to value such acts highly.

Replace social drinking with impulse buys in shops.

That doesn't account for buying it alone. The truth is you are buying a drug or chemical reaction to make you feel different.

Yes, and it really hurt the app industry. I wish Apple had made the minimum $5 or something. It ended up being a race to the bottom. You get what you pay for.

Interestingly, I find (at least in my case) this doesn't apply to games. I've often impulse bought games in the Steam sales for instance.

Something that is typically 30 or 60 dollars reduced substantially in price as one finds in a Steam sale is a very different value proposition from app store pricing, where the prices are always a few dollars generally speaking.

Steam sales feels like you are getting a "deal" so probably helps trigger impulse buys. Mobile platform App store pricing doesn't have this effect really, even if prices are always much lower. App stores have also never had something comparable to the precedent set by retail store game pricing, which the game industry has enjoyed and maintained (arguably there should be more outrage at this, given the large reduction in manufacturing and distribution costs Steam should theoretically be able to provide). App stores also grew in an age where consumers were much more used to "free" ad-supported services on the web.

$60 games or only their $8.99 games?

The more expensive games (or bundles) often. I think this breaks the theory that people give more consideration to software purchases, as I would probably give more consideration to spending $60 on drinks.

I think it's more about expectations. Many people expect phone apps to be free (no matter how good they are), the fact that paid apps are the exception calls for exceptional consideration when deciding to buy. On the other hand, people don't expect computer games to be free.

Although I think games may also be an exceptional case, in that they're thought of more like artistic works (like films) than as software.

That's a strange psychology that I can observe in myself.

The 5$ is what you pay for being at that place and with people. When people drink at home, they rarely pay that much.

With a drink, I'm getting a physical, tangible thing that I know individually took some amount of effort to make. I can watch it being mixed and poured and my reptile brain acknowledges that I'm paying for the raw materials and this bartender's effort (that's on display in front of me). Once I buy the drink it's mine. I can drink it, pour it on my head, give it to someone else, flush it down the toilet, whatever I want. I can take it home and put it under a microscope, maybe verify what it's made of. I can move it into a different container if I want and drink it from there. Due to laws, I have some level of confidence that it will not kill me or make me sick, and that it is fit for the purpose under which it was marketed.

With most software, I'm getting a copy that was made at $0 cost. Someone wrote a script long ago that could make an infinite number of copies of it, and the bits I have on my hard drive are from those copies. Sure, the very first copy took huge cost and effort, but my brain always needs to convince myself of this. Also, I'm not buying the software--really I have nothing after I hand my money over. Just some temporary license to use the software only in ways the manufacturer allows. I can lose this right at any time based on terms in some 100 page legal document that I didn't read but I know is written in favor of the manufacturer. I cannot do anything I want with the software. I can't learn how it works or modify it. I (often) am not allowed to move the software into a different computer and use it there. I have no guarantees that the software will work at all. The software is usually not warranted in any way, nor claimed to be fit for any purpose, and the manufacturer disclaims every possible liability for anything that happens as a result of using the software, again, in its 100 pages of legal documents.

I'm surprised it's normal to pay anything at all for software.

That's just a lot of hand waving.

People throw $5 in the busker's hat as they are casually walking by without a second thought.

Yet they have a hard time paying for a $2 android game that keeps them entertained for many hours.

Of course it's their money and they have the right to decide how to use it but it's also just plain odd and unfair to suddenly treat $2 as if it was $200 when it comes to software.

And most people hear or see the busker's act and don't pay.

For those who do, the value is immediately cognisible, and, given a cash drop, the transaction is entirely transactional and self-contained.

For software, value isn't apparent, the transaction (especially with any ongoing service or billing) is not contained, there is a vastly larger scope of unforseen or unanticipated consequences (both positive and negative, though the latter tend to dominate psychology and behaviour), and more.

Though topically similar, the experiences really are not comparable.

> For software, value _isn't_ apparrent, the transaction (especially with any ongoing service or billing) is not contained, there is a vastly larger scope of unforseen or unanticipated consequences (both positive and negative, though the latter tend to dominate psychology and behaviour), and more.

Again, we're talking about $1 and $2 here not $200 or $2000.

My point is the hypocrisy (if we can call it that) of spending that amount or more so easily and mindlessly with so little expectations and suddenly treating it as if it was $200 when it comes to software such that anything less than perfect means the software creator is not worthy of your precious $2 that you often throw at a beggar without a second thought.

There are tons of really shitty $1 & $2 apps.

My assessment costs for those are literally orders of magnitude higher than the price, and yet the price / access barrier, continuity and support risks, etc., literally get in the way of assessing true value.

Non-software, or at least bundled-software, example: the local organisation is exploring phone and comms options. Extant system is obsolete and failing. The market price of any given alternative is small, but the cost of sussing out features, suitability, comms link provider integration, and how that might change (POTS, VOIP, wireless, non-dedicated hardware, integrated messaging solutions via smartphone/tablet) opens a whole can of worms.

Meanwhile, staff in another department who aren't communicating or comprehending effectively have just purchased a system that manifestly doesn't consider any of this or even meet present needs.

Trying to impart on them even the basics of "POTS possibly won't exist in a year or few" has proved a challenge, let alone other organisational comms needs or wants.

Acquisition price of the system itself is actually, within a factor of 2-10, a minor concern, quite probably captured in ongoing service fees (or their elimination) in a matter of months.

Features offerred by various vendors, their communication of same, guessing future developments, configuration & integration, etc., are vastly more significant.

Price alone communicates effectively none of that. Markets are useful for commodity goods, far less so for complex and highly differentiated ones.

1.) The majority of those games won't keep them entertained for hours. Most of them you look at and never open again.

2.) Most of those games simply make waiting for bus or someone bit more fun, but most people are fine actually staring at wall during that time.

3.) This is mostly equivalent of not impulsovely buying anything else - candy every time you are an hour to diner and bit hungry, shiny enclave when you see it etc.

The developer needs to make enough profit/free cash flow/etc to build the next version. So it's essentially a bet on the expected lifespan of the current version. Which is why if someone is offering something for free, forever, or very cheap, forever, you can bet that they are fairly certain they will never be displaced.

Not always there is a period where new services are offered for free only to trap you into paying more later.

There's a literature about switching costs. One result suggests that the present value (technical term) of the customer to the company is equal to the cost of switching away, under certain assumptions. Obviously this formula is unhelpful/infinite in the case of total lock-in. I guess a good capitalist charges what the customer can bear, in that case.

Agreed. To this I'd add that the switching cost is highly dependent on perception. Costs are almost always as much about perception as they are about reality.

Hmmm, I guess if one has any alternatives at all, your evaluation of their desirability is going to be subjective. Including in the case of "offers one can't refuse". But, when are our evaluations not subjective? That is the human condition.

A cursory examination of the article seems to indicate that it is the DoD who finds software to cost so much. Maybe it is not that software costs so much, but that your procurement process is broken when it comes to software?

The page won't load, but in my experience the initial development costs aren't usually that bad. The cost to maintain, make bug fixes, and improvements is where everything starts to explode in cost.

One reason is developer time. A common contributor to that is poor documentation. Sparse, vague, and unclear documentation. Verbose documentation that explains too much theory without actually telling you what you want to know. Copious amounts of simple examples, but no intermediate or advanced examples.

Undervalued refactoring/modernization of legacy code, not necessarily implementing a whole new stack. Just paying technical debt.

But some think that if you stop providing features or fixes at the fastest pace possible or if you stop to refactor you're just wasting time or slacking off.

Why do movies cost so much to make?

Thanks to digital, it's substantially cheaper to make a movie today than it was just 10 or 15 years ago. From the film and cameras to the editing process to the effects to the distribution. Distribution is now actually risk-free -- just upload to YouTube if that's what it takes. Probably the only thing that hasn't changed is how much lenses and actors cost.

And we got the added bonus of removing nasty chemical film process from our lives!

So why are movie budgets higher? It's gotten so easy to make high quality content and distribute it that you have to out-spend/out-spectacle the others. Netflix, HBO, 400 hours of Youtube uploaded per minute is what you're competing against!

Software has a lot of similarities. There are a million apps in the app store. Cow Clicker won't get it done anymore. So while it's gotten cheaper and easier, there is more competition. This requires people to attempt larger/more difficult/less-well-defined things than before. This is true in consumer and business software.

Link to the MOOC: http://oli.cmu.edu/courses/all-oli-courses/causal-statistica...

Because it requires developers who inherently need to have fairly decent academic qualifications (though not always) but often food maths, which costs in terms of bigger salaries

Good maths

Depends on what you mean by software. Think spreadsheets, Google forms, Trello, Slacketc. What remains is special requirements.

tl;dr: Because hardware in all industries is made by poor third world slaves earning pennies per hour, and software is made by rich white Americans, mostly, and other western nations (you matter too!).
:)

Reply on news.ycombinator.com