Hacker News Reader

Comment by ezequiel-garzon | Hacker News Reader

If I understand correctly, this data was recorded in 2011. It amazes me that it could take NASA this long to sift through its own (massive) data. As datasets are getting ever larger (outpacing gains in computing power, I feel) I wonder if some similar discoveries will take decades (centuries?) in the future.

Does anybody know if these teams make it a point to keep datasets "small enough" to be manageable in reasonable time frames?

View on HN

It should be noted that NASA makes essentially all of it's data public. In the astrophysics community, this means that researchers write scripts to run over all of the data for exploratory research and then do more complicated analysis later. It's also important to note that one of the best and most accurate space telescopes ever made (Kepler) was keeping the entire photometric community very busy.

Looks like this is the public repository of all lightcurves recorded by Kepler: https://archive.stsci.edu/kepler/publiclightcurves.html

That's the first Kepler mission (which lasted 4 years, after which two of the reaction wheels used for fine pointing failed). There's also the K2 mission (which is a resurrection after the reaction wheel failure), which has much more interesting stars from an Asteroseismic point of view: https://archive.stsci.edu/k2/download_options.html

You can also get the raw pixel data if you're doing some more esoteric data analysis.

Thanks for your comment! Do you feel that, in the medium term, as ever larger datasets are generated the overall computing power will be unable to catch up?

To be quite frank, the limiting factor is not computing power, it's code written by physicists. I'm a software developer by trade, and it's sometimes painful to see the type of code my supervisors write. I get the feeling that "our simulation takes half an hour" is acceptable and they won't try to optimise it unless they're near a deadline.

Obviously, that's not deriding them, they have much better things to do than to micro-optimise their code. But in general I prefer writing all of my analysis scripts from scratch after discussing what methods they used. :P

Data gets bigger because instruments have bigger/better sensors (the biggest of which are built like computers) and there is more storage and bandwidth. So it basically advances in lockstep with computing power.

There's a lot of hand-waving and details, but that's the overall answer.

I think this was a case of researchers going back and analyzing data to look for a particular event, as opposed to "they've been crunching through the stream for years and are only now up to 2011".

No, astronomy keeps insane amounts of datasets, because things turn up often enough that what was interpreted as a star fifty years ago is actually a planet. Going through, looking for precovery images means that the more data you have, the more you can sift through later with more knowledge. The old data may never be analyzed, but it still waits there until we discover more about the universe to find its significance.

Reply on news.ycombinator.com