Hacker News Reader

I am often told that I should keep my nose out of other science domain's business because they know more than I do.

I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD.

As an actual scientist (rather than a political activist with a PhD), I strongly encourage you to stick your nose into any or all of my fields (quantum mechanics, PDEs, medical imaging, complex analysis, prediction markets). If you come up with dumb ideas, I'll even explain why they are dumb, rather than just demanding that you leave things to the experts.

Such statements also come out of biologists discussing evolution. This is not, however, evidence that they aren't really doing science. Instead it is evidence that they've been burned out explaining basics over and over again to Creationists and want to get on with their lives.

However some do take the energy out for those explanations. One of the results of their energy is http://www.talkorigins.org/.

Hopefully some day someone will take the energy to do the same with climate science. Because as much as there is a lot of politics there, there is some real science there as well.

> Such statements also come out of biologists discussing evolution.

A lot of my friends are scientists, including in a couple biochemists and some other people that do more or less serious research into topics like that. In my experience, you're off on how they deal with stupid people and stupid arguments - instead of "get out of biology" and moving on with their lives, they tend to address and correct errors, debate if necessary, or at least point the person towards a relevant piece that explains things and take new criticisms and arguments and address them if necessary.

The scientists I know always been open to me saying stupid things (and occasionally not stupid things) and correcting me if I'm wrong, or exploring together if I might not be wrong. Good biologists don't say, "Get out of biology".

> Hopefully some day someone will take the energy to do the same with climate science.

Oh, hah, I honestly responded quickly and missed the climate science analogy originally. If you wanted to make an apologia for climate science, then I do understand why you'd want to draw a biology:evolution:creationism to climate science:global warming:deniers parallel.

I was responding to the "want to get on with their lives" part as being wrong based on my experience, as scientists are usually rather encouraging and tolerant of dissent. Climate science is not so much interested in people and data which disagree with them, which is a pretty big problem.

I'm married to a biology PhD, and at one point spent a couple of years watching people, including biologists, deal with a constant stream of Creationists in places like talk.origins.

My experience is that if you're a personal friend, you get more serious conversation. If you're someone they know but not so closely, they'll have the argument if pushed but don't feel the need to actively educate. And if you're a random person spouting on the Internet, it isn't worth their time to get involved. If pushed to be involved, they don't feel the need to be pleasant about it.

After spending time myself explaining the same thing over and over again, I've come to feel the same way. I've also come to realize that there are plenty of smart people who do not wish to be educated. Including in my direct personal experience, at least one PhD in mathematics and another in molecular biology.

Based on this experience I have some sympathy for the position of climate scientists. You spend your life climbing around glaciers in Greenland, and you don't really feel like spending the rest of it convincing people who don't want to bother learning the basics about climate.

I've actually never heard that statement coming from biologists. Usually they just say "learn more about biology, look up $SOME_SPECIES on wikipedia."

> Such statements also come out of biologists discussing evolution

This is only the case when the opposing person is pretending to be a scientist but is really a creationist. Scientists tend to make arguments based upon facts and experiments. They start to get really mad when people don't follow the rules and start bringing non-scientific "evidence" into the argument.

There is plenty of things up for debate about how evolution works, but these are the details of the theory. Due to overwhelming evidence, the debate on if is over, however the debate on how is very much alive.

Education is a lot more complex than most people understand it to be. Even within the field few people actually have any understanding of how the brain and learning interact. Everything from diet, to time of instruction, to setting, to materials, to methods, to testing is all important. A classic example of which is how do you build a useful test? Well if you want to know how much learning took place you need to test them both before and after instruction. But you also need to test people twice without instruction to see if the tests give information away. You need a mix of problem difficulty to find out if someone understands the basics and at the same time, to see if they understood the minor details. etc. You then need to look at the test not just in terms of overall score but also how well they did on each type of problem.

PS: Education might seem like a flaky field and much of the discussions are devoid of good science. At the same time there is a lot of great research that has been which has real and important implications.

On the contrary, I think math ed types want to keep the actual mathematicians out precisely because the mathematicians recognize how complex the field is. Every time I've been told "stay out, you don't know our field", it was because I pointed out complexity or plausible alternative explanations.

For example, a puzzle: why do SAT scores underpredict female performance in Calc 1, and overpredict grades in higher level courses? I suggested girls are more conscientious than boys, and that a study being proposed did not control for such factors. I even suggested a test of this hypothesis: compare grades in classes which are conscientiousness-weighted (25% homework, 10% attendance, tests questions are practice problems with numbers changed) to ability-weighted coursework (50% midterm, 50% final, test questions all new).

I'm not sure my idea was a good one. But rather than explain why either a) this had been tried, and didn't work, or b) why my test wouldn't work, I was simply told to leave the field to experts.

I've come up with plenty of dumb ideas in other areas, e.g., global existence of wave equations. The closest I've ever gotten to "stay out, you aren't an expert" was "Terry tried that and failed, ask him why before spending lots of time on it."

[edit: really confused why you are being downmodded.]

Honestly, SAT scores vs. Performance is the type of (mostly bad) research I am talking about. The SAT is not designed to test math ability. It's focus is on how likely a student is to finish their freshman year of college and it does that fairly well. You can do a lot of useless research in this area and it tells you next to nothing. If you want to predict a highschool students ability in advanced collage math classes you could design a test that did that. But, people have easy access to SAT data so that's what they look at.

If you want to see real and useful research look into how long the optimal study period. There are significant and useful study’s which suggest 2 hours of nonstop instruction is less useful than two one hour periods with a moderate break in-between. Yet how many collage lectures follow this approach.

"There are significant and useful study’s which suggest 2 hours of nonstop instruction is less useful than two one hour periods with a moderate break in-between."

The professor of the last class I took was very conscientious about taking a break at the 1 hour mark of an 80 minute lecture.

What makes SAT vs performance bad research? The SAT may be worse than some specialized test, but so what? Breast self exams suck in comparison to mammograms, but that doesn't make studies into breast self exams bad research.

The methodology is what makes research good or bad.

The fewer unknowns the better the data and the more accurate the experiment.

In terms of predicting how likely someone is to finish their freshman year of collage having a test where you can increase your score significantly with moderate levels of preparation is not a bad thing. However, the fact you can easily game the test means it is a less accurate indicator of a student’s innate capability. The fact you can retake the test creates yet another sort of bias. etc.

More generally, it's easy to focus on defects in the test, which are irrelevant in a larger contest and subject to change.

PS: Education might seem like a flaky field and much of the discussions are devoid of good science. At the same time there is a lot of great research that has been which has real and important implications.

My wife and I were discussing this recently. She is in an unusual program where she should graduate with both a masters in history and an education certificate with hopes to teach in history at the High School level after that.

This is both second hand an anecodatal, but from what she tells me the education courses she takes are less rigorous and generally easier than the history courses. But the reason for this is that the education course she is taking are vocational. They are trying to teach her how to get teaching done in a modern classroom, not the science behind it.

It is akin to learning to the difference between training to be a mechanic and a mechanical engineer. Both are very hard and important jobs worthy of respect, but they have very different focuses. The education courses are focused on learning the technique of teaching effective, it looks like most of the research on the theory of effective education is done in departments like psychology, sociology, etc.

As another scientist I have to strongly second this. Some things might take real work on your part to understand though. Even within different fields of mathematics there are difficulties in understanding (complaint about this: http://www.math.rutgers.edu/~zeilberg/Opinion104.html).

http://abstrusegoose.com/211

I agree with the sentiment you express and thank you for pointing it out.

I do think that thinking outsiders should stay out is uncommon in the field of math though. I am currently a math grad student, and while I am only a grad student what I have seen of the mathematical community makes it seem like it is very open to cross fertilization from other sciences and from amateurs.

Admittedly he is historical, but Fermat (professional judge when he did most of his mathematical work) is revered and there are some more recent amateurs that have made genuine contributions. Much of mathematics is inspired by other fields such as physics.

I admit I know some professional mathematicians were tired of finding flaws in amateurs proofs of Fermat's Last Theorem and rebuting supposed refutations of Godel's diagonal proof that the reals are uncountable, but it is very different to be tired of dealing with a constant stream like that than to be hostile towards outsiders in general.

From my limited perspective, I think math is one of the fields most open to outsiders and amateurs coming in and making real contributions, both culturally and in the fact that we rarely need expensive experimental setups so the barrier of entry (in terms of money and equipment) is low.

Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the vast majority), simulations are not chaotic. They are based on stochastic models that have converging behavior. So, in general, it really is fair for most people to rely on the stability of the simulations.

As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.

Let me give an example from my field (high energy physics). The pre-eminent event generator of choice is called PYTHIA. It incorporates as much known physics as possible, and is continually updated. There are groups that regularly convene to compare the distributions it produces to those seen in real experiments and adjust the parameters to improve the results. The code is open source, so you can readily make improvements and submit them to the maintainers for inclusion in official releases.

Now the primary uses of these simulations are to either tune your analysis to separate signal from background or calculate corrections for different kinds of detector acceptance issues. No reviewer would ever accept a publication that used simulation if the paper did not include clear evidence that the simulation was valid, usually in the form of a data/simulation comparison.

Regarding reproducabilility, I think you're a little off the mark. The idea is not that you should simply repeat my analysis to get the same answer (this important verification step should be, and usually is, present in all scientific groups). Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer.

Edit: grammar

"As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.... my field (high energy physics)"

Bad example, inasmuch as it is too good. Particle physics has petabytes of data (exabytes yet?) to test against and is very connected to the real world. With that check you can't stray very far.

"Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer."

I would say a precondition of that step is first that I can reproduce your work. If I can't even reproduce your computation it's not even worth trying to reproduce the eventual results, as your eventual results are too questionable to begin with, as far as I am concerned. If I can't reproduce the computation you might as well have just pulled them from your bum.

Weather simulations have even more data; they are less accurate for other reasons.

Also, running someone else’s exact simulation again is useless. You need to start from scratch (or some vary well accepted baseline libraries) for it to be useful.

"Weather simulations have even more data; they are less accurate for other reasons."

I doubt anyone can beat particle physics for sheer information quantity. I seriously doubt that we have petabytes of real weather information to feed our simulations. I can find some references online to petabyte stores for weather simulation results online, but that's ultimately just cache, not information (in the information theoretic sense).

This is part of what I mean when I talk about the information theory, and how you can't get more information in the information-theoretic sense than the sum total of the simulation and the original data. Weather simulations may chew through terabytes or petabytes of RAM in the simulation phase, but they are not fed that much data. If the people involved mistake it for real information, then this is also part of what I mean when I say that once you get into using computers in a big way my training does indeed start giving me standing to complain again by even the most rigid "stay out of my science" standards.

Secondly, your assumption that running the simulation again is useless in a world where you can casually assume that you have all the data and the simulation and can have the contempt bred of familiarity for the whole process. In the real world, if you can't replicate the results, how can you criticize the model? I think there's still some "the computer said it, it must be right" underlying your answer; you can't assume the computer model is worth anything, it must be proved and debated and peer reviewed, which is not possible if you can't even get it to run and get the same results. The model is still subject to scientific inquiry, it can't be given a free pass.

Independent replication of results is also desirable, but you need both. In a world where nobody can replicate the results and it's hard to verify the simulation against the real world, it's too damned easy to end up with the Feynman electron mass situation where the selection effects from the researchers dominate the putative results of the simulations. The researcher summaries of the results of some runs of some models you can't see or execute and some data you can't get at happen to align... what does it mean? Frankly, who knows?

Several satellites take high resolution real time pictures of global weather they run 24 x 7 for years and that's RAW data. There are several of these plus radar stations etc. NOAA uses a subset of that information to make weather forecast (they normally toss out old data and data from the other side of the planet because it's not useful even if it might make a slightly better forecast). We can make highly accurate block by block forecast an hour ahead over major cities and just about any point in the US. However, there is little point in reading that level of detail from a forecast for such a short period of time. Sometimes when the forecast is 50% chance of rain over the next 8 - 12 hours they know where and when it's going to rain they just don't know where you are.

edit: One of the world's largest scientific data systems, NASA's Earth observing system data and information system (EOSDIS) has stored over three petabytes of earth science data in a geographically distributed mass storage system. that's just Nasa and a lot of their data does not make it into EOSDIS.

The goal of science is to understand the world. Running the same simulation on the same data and getting the same result only tells you that the machine running the simulation is not broken. What you want is to run a different program with different assumptions on different data and come to the same conclusion. This is actually used to make 7 day forecasts. They run a few different models with different assumptions and pick the average result. Over time each model is updated independently to maintain its independence.

PS: You don’t validate E=MC^2 by doing the exact same experiment 10,000 times. You do every type of experiment that you can think of which relates to E=MC^2 looking for anything which does not work out the way you think it should.

I think this ongoing discussion is fascinating, and I don't disagree with you.

But (and you knew that was coming), re-running someone else's code, or reproducing someone else's experiment is verification. It doesn't mean that what they did is valid, but it verifies that they did do what they said they did.

I agree with many of your sentiments. Here is a view from "inside" (I do PDE solvers from a mathematics and computational physics perspective, have written open-source models, and contribute to a popular open-source solver library).

1. Very little of numerical analysis is about propagation of rounding errors. I think you underestimate the reliability of floating point numbers in this context. The challenge of numerical analysis is not in representing the continuous one-dimensional object (real numbers), but in representing an infinite dimensional space in finite dimensions. You might appreciate Trefethen's "The definition of numerical analysis" (http://www.comlab.ox.ac.uk/nick.trefethen/publication/PDF/19...).

2. We can prove convergence of iterative methods for some problems of practical interest. More frequently, we can prove that if the iterative method converges, then it converges to a meaningful solution. Such results are unavailable for some systems, and this is where we enter the domain of Verification (the code converges to exact continuum solutions at the appropriate rate) and Validation (the continuum equations approximate reality). V&V is a rapidly growing field which offers a number of mature tools, but is still under-appreciated in many disciplines. The fact that V&V is rarely taught at university, even to applied mathematics graduate students, let alone physicists, doesn't help.

3. It takes effort to turn a one-off code that only runs in your special environment into portable, distributable software. Funding agencies and institutions place little emphasis on this process, so it is regularly neglected. It is exacerbated by the fact that few physicists have any background or interest in software engineering, thus making it more challenging to release non-broken software.

4. It is often not worth the effort to turn a one-off code into distributable software. There is a good chance that only a small number of people care about the PDE you are trying to solve.

I contemplated making a distributable solver for the Schrodinger equation, but there was little interest.

A quote I've heard attributed to Cleve Moler (chief scientist at Matlab): "PDEs are a niche market."

I think that quote holds less and less true over the years. It's not so much that there are huge numbers of people who need their PDEs solved, rather that those who do (aerospace, engines, circuits, oil, climate, defense, energy) really need them solved and are willing to pay for it.

One thing I enjoy about PDE solvers is that algorithmic improvements always pay off because every user is running the largest simulation they can on the most expensive hardware they can afford.

In information theory terms, a simulation can not contain more information that the sum total of the input data and the content of the simulation algorithm.

This is false outside of the most trivial definition, when you simulate evolutionary systems you can create systems far more complex than the algorithm used to generate them, the problem domain, or any other factor prior to running the simulation. That information is crated from your random input stream which does not contain information in the classical definition.

PS: As to the larger context of the discussion. From a simulation standpoint sending other scientist the code and having them run it is a bad idea. Reproducibility does not involve the lab that made the discovery sending the devices used to make the discovery to another lab. The goal is for someone to be able to see the same effect with the most independent setup possible. If scientist A says "building this a model with these assumptions > this result." Then another lab creates their own software which runs and finds either the same or a different result. Or, they can look at the assumptions or inputs and say this is bad data try again. Or, they can look at the code and say, here is a bug fix it and try again. But, those are three independent steps best carried out by diffrent groups.

"when you simulate evolutionary systems you can create systems far more complex than the algorithm used to generate them, the problem domain, or any other factor prior to running the simulation."

Possible. I have to admit I'd have to think more carefully about this, but I'm not sure it's a knockdown win either. In the real world, the extra information comes from the environment in its capacity as a selection mechanism. It is a large source of such information. Simulations tend to have a radically simplified selection mechanism, what with not containing reality and all. Just because something looks complicated and can't be effectively gzip'ed doesn't mean its actually high in information, it's tricky stuff. I'd be much more inclined to buy this, if evolutionary programming was much more impressive than it actually is and actually routinely pulled off programs that were clearly high in information value.

(I've actually spent a bit of time with evolutionary computation. It has one of the largest hype/reality ratios in computer science.)

"Reproducibility does not involve the lab that made the discovery sending the devices used to make the discovery to another lab."

This is an artifact of the fact that is impossible to do this with physical objects, so we've built the mechanics of science around not being able to do that. Don't mistake historical accidents with scientific imperative. While it is still absolutely desirable that simulations as independently-constructed as possible and give as similar a result as possible, if there was an equivalent in the physical world of being able to rigorously examine someone else's experiment we would be doing that. How long would Cold Fusion have lasted if that was available?

Science is inevitably a product of its environment. We don't demand particle-physics precision from the sociologists not because it is undesirable, but because it is impossible. We don't pass around experimental apparatus to each other for examination not because it is undesirable, but because it is impossible. It isn't impossible for simulations. We're allowed to tweak our procedure in response. Science is not a religion.

In general, science hasn't been as reproducible as one would like it to be, there's so much tacit knowledge that's necessary for experiments, many unknown variables, so many demonstrations are so complicated that hardly anyone even goes through the motions. In general there are many more people writing in detail than there are people read and understand, many more people focusing on some narrow branch of their discipline than there are people with genuinely strong interest and understanding of the fundamentals.

My point, primarily, is that the danger of people using tools that they don't understand is an old one; possibly worsened by the narrow focus of modern science, and how easy computing tools have made simulations. But it's been here all along.

As far as sticking your nose into sciences, your nose is invited into any domains of knowledge of mine, but do keep your voice down: there are too many people making too many bold pronouncements without real knowledge; persistent, diligent, and quiet questioning seem to be much more effective at getting the heart of the matter.

I certainly agree with you that there is a lot of publication of results generated with computers that is non-reproducible. However, I'm not sure publishing the complete codes that were used is necessary, as long as there is a detailed description of the algorithm used. In essence, I think it's essential that multiple people write their own codes and reproduce the result. The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.

The PYTHIA example someone posted is a good one: It's my understanding that basically everyone in high-energy physics uses it, and while that means it gets a lot of testing I'm not sure that many actually bother to look through the code in detail, because "it's what everybody uses, so if it had a bug it would have been found already".

The situation is similar, but not quite as bad, in astrophysical hydrodynamics simulations. Basically the entire field uses 6 different codes, but they do work on different principles, and the teams do conduct controlled tests of the agreement between them on specific problems.

About the chaos problem, I suspect that the problem is worse. My impression is that often there is not even a basic test that the results are numerically converged, regardless of esoteric strange attractors.

I'm curious, can you mention any specific cases that you are worried about wrt the chaotic behavior.

The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.

Having a small number of codes seems like an excellent situation. What you should have is large of tests and much test data to test the small number of codes.

Testing code in thorough manner is an important addition to looking at it.

“It is vitally necessary to be able to check the simulation against real data.”

Actually, that’s not enough either. In spam training, you have to have a completely set of data that you use for the final test but never, ever for training. Why? Because otherwise your spam filter will learn how to properly classify every message you train it with and nothing else.

For simulated science, this means it’s not safe to say, “Well, we trained the simulation until it could reproduce 2000-2010 when we gave it 1990-2000 as input data.” If you do that, your predictions for 2010+ are probably going to be worthless, since your simulation is like a cheating student who only knows the right answers on old tests he stole, not how to pass any test in general.

Even worse, having reviewed the code that leaked during Climategate, it's pretty clear that the researchers are not competent programmers. The revelation that was most off putting was the "commented out" code that injected arbitrary values that was "just used for testing". Testing is great, but it has no place in the production code. If the researchers method of testing is to write test code in the production code, run it once to make sure things "work", then comment it out, we certainly shouldn't be making any decisions based on predictions made by that code. Their testing framework is apparently comments in the code.

For a project I'm currently working on, I have a 5:1 test to code ratio, largely because I'll be processing credit cards. While my project will probably only ever deal with a few hundred thousand dollars in transactions, I need to be sure that those transactions will be properly handled. In comparison, these researchers are encouraging a trillion dollar piece of legislation, and their testing methodology is intuition and spaghetti code.

Further troubling is the assertion that this is science. As far as I can tell it isn't testable or repeatable. As I recall, that's a pretty big part of what science is.

To echo the meme around here, there appears to be an opportunity here for a startup :)

If you can get buy-in from the research community, even in one niche field, I could see great utility in a central escrow house of sorts, where people can post research software, datasets, etc. and others can download the stuff and validate the results only after agreeing in some reasonably binding way not to use that information to scoop or copy the original researcher.

Buy-in in this case means that everybody should have to do it to publish in a certain venue. You might even do something fancy like providing people access to a VPS which will run the server without allowing validators to scp everything back to their own machines. Of course this is impossible to do perfectly; once you allow people to view the data in a text editor or hex dump, it is all over in a strict sense (scripts + screen-scraping), but it is still a step up from just handing over the files, and keeps honest people honest.

You might also provide some anonymization/randomization service for datasets (for instance, by applying an unknown linear transform to absolute numbers in the case that the results don't change, similar to what was done here: http://googleresearch.blogspot.com/2010/01/google-cluster-da... .

I think there are 2 reasons why researchers can be reluctant to share:

1) They are afraid of getting scooped by somebody else leveraging all the effort they put into developing the tools and curating the data and leapfrogging them.

2) Their results are suspect.

Such an escrow house would ameliorate the concern behind 1) and help suss out 2) :)

Anyone else have thoughts on this?

In biomedicine, the National Library of Medicine has been taking steps in this direction. There are repositories for sequence, protein structure, and microarray data, for example, and many journals require that relevant data be deposited in these repositories as a condition of publication.

Efforts such as the Science Commons are also thinking about/working towards these sorts of solutions.

Actually, the NIH has already done this with many recent grants. Everyone funded through certain grants has to make their data available in dbGAP (genotype and phenotype), sometimes even pre-publication (which is raising some feathers).

Point 1 will always be true when people share methods and/or data pre-publication. If point 2 cannot be discerned with currently available methods (and I would argue that it can), then we are in a world of trouble.

jeff:

i'd like strongly to encourage you to offer your opinions on things on which you are nonexpert. having only experts allowed to opine, with experts also relaining monopoly on experts, leads to echo chambers.

however, you seem to be reacting to a different article entirely. nowhere did the article mention simulations. if you look at Stodden's CV (the woman whose talk motivated the article) she's a statistician. The need for code-sharing is in fields like climate and bioinformatics where people's analyses lead to changes in politics, policy, pharma/health, etc; not for people deciding whether lorenz goes chaotic at precisely the right bifurcation parameter value.

more and more and more science is becoming statistical and data-driven. chaotic dynamics, though very important, is not the issue here. it's analysis of complex data coming from real-world systems.

there are many interesting and wise comments in this thread but they are almost all about an article on simulations -- ie. not the article the OP points to.

Can you provide references that would allow a researcher to educate himself? I'm genuinely interested, and there seem to be lots of fields that use software in experimentation and simulation but where there isn't much emphasis or education on doing it properly.