Hacker News Reader

Comment by BowBun | Hacker News Reader

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates.

According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.

If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.

View on HN

I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.

Well, the metric is #papers and #citations to get funding. So yes that's true, ideally, but...

Relevant blog post - It's not the incentives it's you.

https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...

Could not disagree more. The premise of this piece is that dieting is just willpower, beating addiction is the same, an entirely individual problem.

The reality is that the incentives are what society is asking for, with ethics etc acting as constraints only, rather than actively rewarded. In that formulation it is inevitable (and optimal) that some people will skate to the edge, and if the edge is poorly enforced they will increasingly go over.

Policies must address the overall actual effects, not just a chimerical ideal.

(It is even more ridiculous coming from someone in psychology.)

While I tend to agree, the research we environment demands this nonsense. Also, they have to start somewhere to set a record to get future funding. Should probably do more survey studies of what data exists and where the state of things are then suggest new studies and seek funding for it.

Short of large and completely government funded development projects, I think it would be a struggle to get data. Few businesses would be willing to offer up development processes and surrounding data due to potential IP lost. Any organization that has good processes have it in their interest to striffle others from discovering how to improve theirs or learn from the success of others.

Part of the issue is that academia largely just doesn't pay for software development which they can leverage as an accessible cheap data source, it's done as a completely privatized exercise. Any research that requires highly protected commercial processes is pretty difficult to get any traction on unless you're inadvertently rediscovering the same processes (from my experience).

With that said, I feel like the amount of empirical data you'd need is going to be incredibly high, much if it not even currently being collected.

Research at business schools manage to find all types of data that companies believe is way more integral to their success than information about their software estimation process.

These researchers are choosing to waste their own time and governments money, because it's easier to play with new toy machine learning models than go to networking event and befriend software VPs in order to convince them to give you their data.

Nah. You just got a bone to pick. Christ.

Rereading my comment I think you're right. Been frustrated with bureaucracy related to covid (specifically the vaccine) and the lives it's costing. And that frustration was redirected to other people following silly rules with far less at stake.

I still think a lot of research in software engineering is useless because researchers spend too much time focusing on different methods and not enough going out into the world to collect better data. And researchers should be mildly ashamed for doing crappy research.

But also my tone should have been more constructive.

> How large are these datasets that have attracted so many research papers?

> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).

> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?

> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.

Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?

I think the term 'cargo cult' carries some religious history and other baggage that I wouldn't want conflated here, but your feeling that anything we've heard about software estimation is questionable is probably warranted.

IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.

Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.

I imagine that if Atlassian could get permission from its many JIRA customers, then doing a text-based ml-categorization of issue descriptions alongside completion times would be extremely interesting.

I'd wager that data is just as poor overall, but the sheer volume of data available might be able to help get some sort of consistent conclusions from them.

I could imagine getting data from tools like Jira, but there is so little consistency on how data is entered and updated, I would have a hard time swallowing any conclusions from that data.

A firm like Pivotal Labs (which, I just discovered, is now VMware Tanzu Labs, which is frankly the saddest thing I have read all day) would have a dataset and a good enough understanding of the assumptions in their Pivotal Tracker software to do good research here. I'm almost certain that they have internal papers that are far beyond the state of the art, so to speak, in the broader academic literature.

What data would be useful here? There are so many confounding factors.

Well, isn’t that the kind of thing researchers ought to be researching?

Ought to? Not my call, as I don't employ any.

I'd say Agile methodologies have done well in exposing the key factors for successful delivery, and they seem not very amenable to generalisation: - creative endeavours such as software creation are fraught with estimation difficulties (contrast, say, with migrating VMs from on-prem to cloud) - whoever prioritises / deprioritises features will have a large say in the accuracy - whatever technologies imposed/chosen will have a reasonable say in the accuracy - whatever architecture decided on will have a reasonable say in the accuracy - unforeseen requirements that may necessitate architecture / tech / people changes have a large say in the accuracy - individuals' skills in a team will have a very large say in the accuracy - how long a team has worked together / in an architecture / with a technology will have a very large say in the accuracy

We know all this. What can we learn? Genuine question.

If researchers aren’t doing anything to quantify and assess these ideas around estimation, I don’t know what the point of that entire sub-field of research is, beyond cranking out papers.

The issue here is that if you examine most projects today, it requires effort to collect data about what happened. So much that happens is untracked. That untracked stuff is a source of error.

Dont be so hard on HN, I would say given how niche the actual article topic is, the more generic discussion about software estimation is both relevant and relatable.

I'll be hard on HN here. The more generic discussion is barely relevant to the article. It's a prime example of people seeing an article title as a chance to say something they've already been thinking rather than a chance to read something new.

Reply on news.ycombinator.com