anon_barcode

True. Nobody can check if your data + your methodology = the results you claim.

But it is quite common for someone to take the same problem, collect their own data and run their own analysis (both steps can be completely different from the original statement-- collecting data of a different type, from a different instrument, etc. Running an analysis of a different paradigm, running the same analysis to a different precision, running the same analysis via a different algorithm, etc.).

If you read the arxiv on a daily basis you can see huge academic arguments unfolding over the course of months and years.

There seems to be this idea that the conversation goes: "Yo I found this hypervelocity star" "Dope, let's move on"

It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different"

Authors are called out and proven right or wrong on a daily basis, even if we can't watch them code over the shoulder. I actually think that's one of the beauties of it-- most of our methods are invented by trying to prove or disprove something in a new way.

The monster codes like GADGET (which ran the millenium sim everyone's seen) are usually made public after ~5 years of proprietariness.

In astronomy we are still fighting for public data. Verification of results comes from another researcher collecting his own data and running his own analysis.

Out of curiosity, do you work for industry or academia?

Valid.

But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer, if you disagree, you implement a different methodology, which necessitates new code anyways, and publish your contra-finding.

We usually converge on answers over years of varying different methods and attempts. No one in their right mind reads a paper and says "well, they found out the sun is actually in M31, I believe that now" They look at the culmination of the literature (fun fact, we still aren't positive exactly where the sun is, or how fast it's moving) and I suppose that is the black box.

What I was really referring to though was students who walk up to me and say they gaussian smoothed a sample, and have no idea what I mean when I ask if 3 sigma outliers were used in the fit or trashed. They just used some gaussSmooth algorithm and may or may not know what a gaussian even is.

"If you are publishing articles with no source code, how is that even science?" - We have a methods section in the paper, just like in the science fair.

"After all, if you publish details of how you worked stuff out, even without source code someone else might reproduce your work." - hopefully they do! and hopefully they do it in a different way, doing the same thing twice is pointless. This is why the Hubble Constant started off being 150, then was 50, and now hovers around 69. Because everyone did the problem in their own unique way and didn't borrow each other's code and procedures.

"there would still be the same number of jobs in astronomy" - yes, that's the problem, there aren't enough. So the job goes to the guy with the most citations and publications-- I want that guy to be me.

"And don't you think if you are using the software from another group you would cite the authors?" - that's not the whole point. Writing a simulation may take 3 years, then analysis can take another 3 years. If I wrote it, I want first dibs on analysing it, I don't want someone else taking the results I created and publishing them before me.

The system is cutthroat, sure, but it does have benefits if you know how to play the game. Politics is surprisingly omnipresent here.

Several reasons.

Firstly, it's a boys club up top-- a lot of PhD requirements are build around basically hazing rituals. It's designed to make you "prove" yourself-- instead of remembering the Frats founding members, we memorize moments of inertia and rotation matrices, but it's really the same thing, seeing as how I don't deal with those items on a daily basis at all, and a quick duckduckgo will give you more information than I ever could.

Secondly, astronomy and astrophysics is arguably the least essential of the sciences, so our funding shrinks accordingly when economies crash (what do we really make, coffee books?). We don't make quantum cryptography, and we can't levitate objects... although we have some remarkably military grade trajectory maps for all your satellites...

Thirdly I think it really is an attempt to make us work well past what any other industry would consider remotely acceptable. For a bit of insight into the situation, check this completely serious correspondence sent to an unnamed faculty which is now famous in the astrophysical community: http://jjcharfman.tumblr.com/post/33151387354/a-motivational...

I personally work with a 40 year old doctor, a leader in their field, who has not published a major paper in the last 3 years. They're being deported and are moving back in with their family if they can't get a job in the next month or so after having 2-to-3 3-month "pity" contracts. Of course this is just a hearsay example, not the norm, but it illustrates.

Another thing to remember is that we basically cannot have real families-- our jobs frequently require us to change states or countries at the terminus of our postdocs, and our salaries are on the order of entry position BSc coders 50-80k usually depending on skill. The hotshots in the field get 5-year contracts. At age 30 a 5-year contract is pretty much the best you can possibly do.

As an astronomer I hope this does not take off.

The threat to job security is not just "perceived". Astronomers are frequently kept in a state of constant job-induced anxiety by the prevalent practice of fighting for 6 month - 2 year "postdocs" for the first 15-20 years of their careers (seriously, go to an astronomical conference, you would not believe the amount of people greying under 35). To "keep" your job (in reality, get another postdoc) you must do two things: author papers and get papers cited.

This practice makes collaboration actually detrimental to a researcher's career unless one of two things happens: 1) they lead the collaboration and get their name as first author 2) the collaboration allows some kind of recompense for the time invested into the collaboration.

For this reason, many collaborations have a period of proprietarity, a time when the collaboration data is available exclusively to the people who have sunk their time into the collaboration. Imagine, if the people behind the Millenium or Aquarius simulations were forced to publish their code as soon as they had run the simulations (this is true for any theorist)-- now these people have spent the past years of their lives, tweaking and perfecting code _for which they get no publications or citations_ and before they can even begin to analyze the easiest results ("low hanging fruit", the simplistic papers that usually go to collaboration members), the simulations are being run and analyzed all over the world. They have gained nothing by their efforts in actually developing the code (our world does not work like industry, we don't get a pat on the back and a raise for being team players).

For theorists, often entire careers revolve around codes that have been developed over the researcher's whole career-- to be forced to hand it over to all the first year PhDs in the world is a tremendous slap in the face.

For me, as an "experimentalist" (read, data-analyst), I also maintain job attractiveness by having sets of code that no one else has. I'm experienced in a variety of pattern finding algorithms, and have even invented a few myself for very specific problems-- and within my little sphere, people know this about me, I'm the person people come to for certain things. I spent years of my life perfecting these, learning about algorithms, learning the intricacies and quirks of the various datasets we use-- if someone wants to take my place in this community as "that guy" then I expect them to devote as much time as I have to learning these techniques inside out and then to do something better than me. I have no desire to pass my code off to a masters student and let him naively plug some dataset into it-- firstly because no one should ever rely on a black box in this field, and secondly because these codes all need to be tweaked to account for the different instruments and data structures being used.

Perhaps if my contract didn't have a built in expiration I would be willing to care-bear-share my fractal search methods and spend my Thursday afternoons showing you how to Savitsky-Golay smoothing works, but as long as I'm out of a job next July, and you're the guy applying for my job, you can keep your filthy hands off my code.