Comment by spenczar5 | Hacker News Reader

Comment by spenczar5 | Hacker News Reader

For the HN crowd, it may be especially interesting to note that this was pretty much a software discovery.

The dataset here was DES, the Dark Energy Survey. It’s a big dataset that has been around for a few years, and probably thousands of astronomers have worked with it. But its just a huge amount of data, covering a lot of the sky.

Bernardinelli’s software discovered a blip in that big pile of data. He crunched through the dataset and got a weird answer for the size of the solar system (basically - its a bit more complicated). Almost everyone, when they get something like that, tweaks the parameters of their analysis code: “I must have the threshold for object detection set too low.” The real discovery comes from trusting the software and pursuing the question.

This is a lot like software developers and their persistence. When your production service has a mysterious error in the logs, do you shrug and say “well, probably some transient network bug, it won’t matter” or do you dig in to understand what’s going on? Both approaches are reasonable, and it takes a balance of pragmatism and curiosity to be effective.

But anyway - there is an increasing demand for applying rigorous software engineering techniques to astronomical datasets. A problem that we are facing in astronomy software today is that the datasets have gotten way too large for astronomers to just pull up images in DS9 and eyeball stuff to make sure it’s right, so they need software that is much more trustworthy and bug-free than before.

—-

Edit to add: DES data is completely public access: https://www.darkenergysurvey.org/the-des-project/data-access...

I love this point, and I have to go on a tangent on this:

> This is a lot like software developers and their persistence. When your production service has a mysterious error in the logs, do you shrug and say “well, probably some transient network bug, it won’t matter” or do you dig in to understand what’s going on? Both approaches are reasonable, and it takes a balance of pragmatism and curiosity to be effective.

I find that it really depends on how harried I am. Am I (or someone on my team) able to dedicate time to this issue, or do we have other higher priority concerns?

Every single time I can think of where we've had a "that's weird", it was a bug in the software. But it would take days to weeks for someone find the root cause (these were 1 in 100,000 race conditions), and we just can't afford to dedicate that amount of time to each.

So we have to pick our battles.

When the system is the Solar System, or the human nervous system, I say society has an ROI worth multiple PhDs to invest - but for an ad service that is likely to be completely replaced in a decade - yeah it's a lower ROI

But this raises the interesting (to me) question - almost every enterprise I know operates at the "harried" level - no one is so well funded they can saunter around

But science is basically saying "we trust you to find something worthwhile. Go saunter"

Maybe it's all about trust

There's plenty of sauntering in industry. Especially enterprise. "Enterprise" is notorious for not actually getting anything done, even though many of the individuals may feel stressed most of the time.

> "Enterprise" is notorious for not actually getting anything done, even though many of the individuals may feel stressed most of the time.

Enterprise is also way more driven by personal egos and certainties about being right. I doubt too many scientists would say "I'm going to reform the department, let's throw away the last 5 years of work and start again" – but decisions like that are commonplace in enterprise software.

> But science is basically saying "we trust you to find something worthwhile. Go saunter"

Not... quite.

Typically, scientist in academia are underfunded and overloaded with teaching and management duties. The way to get round to actual research is to hire phd students / postdocs. But there is no institutional funding, so you have to write a grant proposal. Since every academic is in the same boat, project calls are swamped with proposals and can only funds 10-20%. Since you want to get funding, you create a very clear project proposal with detailed timeline of expected publications. Because funders don't award mediocrity, your proposal is ambitious in this.

Now if you get the funding, you suddenly have a significant, detailed, ambitious research obligation to fulfil for the next #PhD years. You get to do research, but there's not much leeway.

No project funding? Here are stacks of committee work and teaching load. Oh, but we do expect you to keep publishing actively. Maybe make time on the evenings and weekends for research?

All in all, not really "go saunter".

> When your production service has a mysterious error in the logs

...such as a 75 cent accounting discrepancy? ;)

https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)

Please say more. How does a software engineer with a data engineering background and interest in astronomical phenomena learn about/start contributing to this area?

Most of such jobs pay peanuts (relatively speaking). Academia is something to consider as a hobby or after you retire.

There is huge need for these skills in astronomy, but it can be tricky to find the right way to plug in.

There is plenty of opportunity to contribute to open-source —-astropy is one of the most active projects, as a starting point.

Deeper (volunteer) engagement probably requires finding a willing scientist to plug you in to a project. That can be tricky get going and to make mutually beneficial.

There are full time research software engineer positions—these will pay less than industry, of course; for some folks the subject matter and environment make that trade worthwhile.

No option sans degree, I assume? Or at least reasonable compensation while pursuing one.

I've always wondered that and would probably do it for free, but my guess is that you run into the problem that all those gigs are reserved for grad students.

I do remember seeing a website that attempted to connect SW engineers willing to volunteer with scientists but I think that effort died out. Not a bad idea.

Reply on news.ycombinator.com