Hacker News Reader

A few years back I parsed PDS files as part of my rendering-the-moon-with-added-vegetation project (<plug> http://cowlark.com/flooded-moon/ </plug>).

Wow. It's so... special.

That said, even though the format is clunky and horrible, it genuinely is self-documenting. I managed to parse it in a surprisingly small amount of code without needing to refer to the documentation at all (which is good because I couldn't find any). Some datasets needed special treatment because the value that came out of the formula specified in the PDS label needed to be offset to produce a physically valid value. But it all worked. They're even mmap friendly. So, ancient and loathesome yes, but also pragmatic and effective yes.

The C++ parsing code, if anyone cares, is at https://github.com/davidgiven/flooded-moon/tree/trunk/terrai... in the pds.h and pdslabel.h files.

Incidentally, if anyone knows of:

(a) a raytracer with volumetric rendering that has a programmatic API (right now I'm using a hacked version of Povray) (b) higher-resolution terrain data of the moon than the 6GB Lunar Reconnaissance Orbiter DEMs available here: http://wms.lroc.asu.edu/lroc/view_rdr/WAC_GLD100

...please let me know?

I did similar a few years back, but for Mars DEMs, also in PDS format. At first I despised it, but yes, there is a certain pragmatism to the format. And, yeah, the best way to figure out what was going on was to just look at the files themselves.

The thing that most annoyed me at the time was there wasn't a good open source JPEG2000 decoder I could use to slice up the imagery to go along with the DEMs. But that must have changed by now...

When i get home I'm going to interrogate your code to find the bug in mine... I hope. Although the data source I had was much bigger than 6GB :P

I don't know if many existing ray tracers, but if you want to build one give me a shout. Rendering spheres with height maps and volumetric covering is am evening or weekend project from scratch once you know what you doing :)

Actually, I did build a raytracer. In Ada, of course. https://github.com/davidgiven/flooded-moon/tree/t3-pureada/t...

Rendering the planets themselves was simple enough once I figured out how raymarching worked, but once I started adding volumetric atmospheres with scattering (needed to make clouds and their shadows work), it got complex. Coming up with the optimal arrangement of atmospheric samples and how many subrays to cast got very complex very quickly and while I could make it work, it was never very satisfactory and Povray, with its years of optimisation, was vastly faster.

...although I see now that Mitsuba has had a new release. Hmm.

> its a bit of a shame that the fortran friendly file format still seems to the the most popular way to store this kind of data... but its easy to parse

Haha, don't get me started on FITS files. It's essentially impossible to analyse big postage stamps because you need to load the entire FITS file (and array) into memory. It's annoying that this is the standard we use.

> http://vizier.u-strasbg.fr/viz-bin/VizieR - a huge collection of catalogues of stars, galaxies and other things that can be searched and browsed.

SIMBAD (also from that site) is incredible for cross-referencing and finding references for indexed bodies. I used it quite a lot while I was doing astro research.

I'm trying to remember the format of the LRO LOLA data... I never quite managed to put the tiles together right without seams everywhere.

Too much container complexity for what it is...

I used to work in IT at a place where FITS was the standard image format. Made me want to tear my hair out daily.

ImageMagick supports FITS, and will happily convert it to PNG or some other rational format. (It will make FITS files too, if you're feeling masochistic.)

Problem is that you need to have the actual photon counts. NASA (for Kepler at least) uses FITS files to store essentially a 2d array of photon counts. So converting to another format is not actually helpful.

But I agree that if someone is using it as an _image_ format they should reconsider their life decisions.

Sure, but for generating a quick thumbnail, it works just fine.

It looks like there's plenty of libraries out there that can deal with FITS files without bringing the whole thing into core.

Assuming you mean FITS when you refer to the weird file format, you may be interested in this (well-maintained, mature) Python wrapper library: http://docs.astropy.org/en/stable/io/fits/

FITS does a lot of stuff that PNG and friends can't, so it remains in use to this day.

If you're looking for astronomical data from Hubble (and some other space telescopes), you should check out https://mast.stsci.edu/ (or http://archive.stsci.edu )

For planetary data, the PDS Imaging Node has pages both at NASA and the USGS. Here is the USGS page pointing to much of it: http://astrogeology.usgs.gov/facilities/cartography-and-imag.... The PDS formats can be difficult but tools like GDAL now deal with it. Here is tool I wrote to search much of it: http://pilot.wr.usgs.gov.