Comment by klysm | Hacker News Reader

Comment by klysm | Hacker News Reader

Parent Full thread

The no iteration thing is very real and I don’t think it’s even for particularly bad reasons. We iterate on models to make them better, by some definition of better. It’s no secret that scientific work is subject to rather perverse incentives around thresholds of significance and positive results. Publish or perish. Perverse incentives lead to perverse statistics.

The iteration itself is sometimes viewed directly as a problem. The “garden of forking paths”, where the analysis depends on the data, is viewed as a direct cause for some of the statistical and epistemological crises in science today.

Iteration itself isn’t inherently bad. It’s just that the objective function usually isn’t what we want from a scientific perspective.

To those actually doing scientific work, I suspect iterating on their models feels like they’re doing something unfaithful.

Furthermore, I believe a lot of these issues are strongly related to the flawed epistemological framework which many scientific fields seem to have converged: p<0.05 means it’s true, otherwise it’s false.

edit:

Perhaps another way to characterize this discomfort is by the number of degrees of freedom that the analyst controls. In a Bayesian context where we are picking priors either by belief or previous data, the analyst has a _lot_ of control over how the results come out the other end.

I think this is why fields have trended towards a set of ‘standard’ tests instead of building good statistical models. These take most of the knobs out of the hands of the analyst, and generally are more conservative.

  > Iteration itself isn’t inherently bad. It’s just that the objective
  > function usually isn’t what we want from a scientific perspective.

I think this is exactly right and touches on a key difference between science and engineering.

Science: Is treatment A better than treatment B?

Engineering: I would like to make a better treatment B.

Iteration is harmful for the first goal yet essential for the second. I work in an applied science/engineering field where both perspectives exist. (and are necessary!) Which specific path is taken for any given experiment or analysis will depends on which goal one is trying to achieve. Conflict will sometimes arise when it's not clear which of these two objectives is the important one.

There is no difference between comparing A versus B or B1 versus B2. The data collection process and and the mathematical methods are (typically) identical or subject to the same issues.

E.g.: profiling an existing application and tuning its performance is comparing two products, it just so happens that they’re different versions of the same series. If you compared it to a competing vendor’s product you should use the same mathematical analysis process.

I was kind of scratching my head at what GP was getting at as well; I suspect that "better" has a different metric in the second case: i.e., the scientist is asking which chemical A or B has the stronger desired medical effect; the engineer is assuming we're going with chemical B, and trying to drive down cost of producing the chemical or improve lifespan of the pills or decrease discomfort administering or increase absorption speed or tweak the absorption curve or something like that. Those metrics are often much easier to measure than the effectiveness of the chemical itself, and much less scientifically interesting.

This is how I perceived the difference: >SCIENCE< [a] create a hypothesis [b] collect all the data [c] check the hypothesis and publish; >ENGINEERING< [a] create a hypothesis [b] collect some data [c] refine the hypothesis [d] iterate over [b] and [c] until [e] PROFIT! (and maybe publish someday); the engineering approach is often better funded, allowing more data collection and better validation. If your engineering model is sufficiently deficient your product will be rejected in the market if it can even get to market. If your scientific model is sufficiently deficient, a researcher depending on that model will someday publish a refinement.

In particle physics, it was quite fashionable (and may still be) to iterate on blinded data (data deliberated altered by a secret, random number, and/or relying entirely on Monte Carlo simulation).

Interesting I wasn’t aware of that. Another thing I’ve only briefly read about is registering studies in advance, and quite literally preventing iteration.

Given the set of scientific publication assumptions (predominantly p<=0.05) this can easily allow one to find whatever proof you were looking for, which is problematic.

That being said, it's completely fair to use cross-validation and then run models on train, iterate with test and then finally calculate p-values with validation.

The problem with that approach is that you need to collect much, much more data than people generally would. Given that most statistical tests were developed for a small data world, this can often work but in some cases (medicine, particularly) it's almost impossible and you need to rely on the much less useful bootstrapping or LOO-CV approaches.

I guess the core problem is that the methods of statistical testing assume no iteration, but actually understanding data requires iteration, so there's a conflict here.

If the scientific industry was OK with EDAs being published to try to tease out work for future experimental studies then we'd see more of this, but it's hard to get an EDA published so everyone does the EDA, and then rewrites the paper as though they'd expected whatever they found from the start, which is the worst of both worlds.

Yeah it's essentially a way to reflect parsimonious assumptions so that your output distribution can be characterized as a law.

Iteration is necessary for any analysis. To safeguard yourself from overfitting, be sure to have a hold out dataset that hasn’t been touched until the end.

What about automated predictive modeling pipelines? In other words, I want the best possible point estimates only on future data. I’d think, regardless of the model selection process, I want to reestimate the parameters on the entire dataset before I deploy it, so as not to “waste” data? I.e. I want to use the hold out test data in the final model. Is this valid?

> What about automated predictive modeling pipelines? In other words, I want the best possible point estimates only on future data. I’d think, regardless of the model selection process, I want to reestimate the parameters on the entire dataset before I deploy it, so as not to “waste” data? I.e. I want to use the hold out test data in the final model. Is this valid?

Personally, I think that as long as you're generating data constantly (through some kind of software/hardware process), then you'd be well served to keep your sets pure and build the model finally only on data not used in the original process. This is often wildly impractical (and is probably controversial even within the field), but it's safer.

(If you train on the entire internet, this may not be possible also).

Reply on news.ycombinator.com