What happens when scientists admit error

I don't think you need to worry about more than a few principles -

1. Reproducibility

This is very important for scientists so I expect them to know about package management and containers. You don't need to know everything about it.

Virtual environments, docker and using a good package manager like poetry (which is just an abstraction over virtualenv) shouldn't take more than a few weeks. Once you are done with the basics, you can learn more as you build.

2. Git and documentation

Add another few weeks.

They don't need to write good code. They just need to document it which they should already be good at. Optimization, scalability, distribution and other things can be handled by engineers but without documentation, it makes the work a lot harder.

Physicist here. The lack of any formal training in most programs is a problem. In 2020 it's essentially impossible to do research without being able to write and maintain code, yet there are no real standards of knowledge for the dicipline as a whole. I dont know of any undergrad programs requiring physics majors to take even intro to cs. Most incoming grad students have never used version control, and probably about half of professors have never used it, let alone want to start using it. Larger collaborations tend to be more rigorous with coding practices because they have dedicated software engineers who enforce order and show people the way. Small groups with a few students and a professor are basically the wild west as far as writing software, and vary wildly.

In practice most grad students get coding experience by working on projects, which is a great way to learn, but its rare for anyone to look at the code itself afterward if the results are sensible. I dont know of any small group with a formal code review process. I think we could make relatively inexpensive improvements in science by bringing in more dedicated software people who can educate others and help with the software engineering aspects. Its not unusual for a lab with half a dozen groups and 50+ scientists to have no software specialists, and 1 or 2 IT people whose responsibilities are purely infrastructure oriented.

This is the core question. Can you teach good software engineering, or does it come from years of working with more experienced people/making mistakes and learning?

I have met a few "born SREs" who had little or no training but knew exactly what to do in production outage situations. But I haven't met anybody who was a "Great Software Engineer" out of the box.

And scientists aren't going to take a lot of time out of their education to get that day-to-day programming experience. I was lucky- I worked in a theory lab, next to a team of good software developers, and worked/learned from them over a period of years.

I think what we need are more research engineers, who are people who do software engineering for scientists, so the scientists don't have to become software engineering experts. The scientists also need to learn a bit better to ask for the right tools.

Except that it wouldn't speed things up at all. Academia writes run-once code, which changes spec fourteen times in one week. Their use case is orthogonal to industry.

Have you considered that maybe the academics actually know what they are doing?

> if you look at chemistry or medicine, there researches have good practices for managing the lab and respect them.

Their studies / experiments last years.

In CS/ML/Applied Math you sometimes have to write an experiment with a deadline next week. Excuse me if when I'm trying to scramble for a deadline at 3am I don't have my mind toward TDD or I'm not neatly packaging everything in a docker.

> in academia there is this idea that learning good practices is like a 'dirty' thing that is not required

I think you got me wrong. Shipping quality software is not 'dirty' but requires a specialised focus. One can not do everything by yourself - science and engineering are complementary skills. In your example of chemistry, the chemist who designs a molecule does not spend time to ship the molecule to the world.

I don't fully understand the relation between the difficulty of the work, the luck involved, and upside vs downside. I don't even know what you mean by the "downside", is that the downside of making the mistake, or the downside of not making the mistake?

I'm saying the downside of making the mistake (for her) might be larger than the upside of making the mistake. I don't think that's obviously wrong.

But my real main point is that speculation of this kind (in either direction) isn't very productive.

I'm not sure, I didn't understand fully the bug involved. Possibly such a control group would show the dependent variable as a constant plus some random noise. It is totally expected behaviour for variable.

>"This begs the question" is same as "This raises the question"

No it isn't, that is a common error in modern English. Raising the question means bringing a question into focus. Begging the question means to assume the conclusion is correct in the premise.

https://en.wikipedia.org/wiki/Begging_the_question

Your rebuttal stands but not your point, for the comment you were replying to didn't properly contextualize their argument in a scientific experiment. It's more like serve participants canned vegetable soup, but in the non-control group warm it up before serving. Here you'd reasonably expect the non-control to be universally more favorable.

They did have a null hypothesis and a control group. The problem was that the non-control was different from the control in a way they were careless with, that the scientific methodology does not catch (that peer review and replication ought to have caught). e.g. if the favorable results were better explained by subjects seeing soup taken out of the pot vs. from the can rather than the temperature itself.

You're nitpicking the example, rather than responding to the underlying point.

Unless she is independently wealthy there would be a strong incentive for her to not admit the error if there was an expectation to hand the money back. Where is it supposed to come from?

Some days at work, if I wrote buggy code, I go to my boss and return my paycheck. No! Wait, that never happens.

Sometimes there are going to be mistakes.

This is just an optimization problem with where you place the incentives. The follow-up study funding is not exclusively for false null hypothesis rejection due to chance. It's also to allow for some amount of error in experimental design or practice. No one wants 0% experimental design / practice error because that will harm total experiments done per unit time which is also a thing we want to go up. No one wants only certain results because we want novel results / unit time to go up too. It's just multivariate non-linear optimization.

Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate.

You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.