Hacker News Reader

Comment by comstock | Hacker News Reader

I found this really interesting. I was curious about the Japanese realtime sea-floor sensors they mentioned so I went hunting.

The data appears to be available here:

http://www.jamstec.go.jp/scdc/top_e.html

After registration [1] you can download data in some funky binary format. Looks like there's a 10min delay on data. It looks like an interesting dataset, and I hope to play with it a bit.

[1] includes such joys as password being set to username, and the seeming inability to change your password citing "invalid chars: 1" error.

View on HN

I used to work at an exploration geophysics firm, these data formats less scary than you'd often encounter in that world. There are a number of interchange formats in geophysics, notably SEG-Y, which date back a long way – most vendors then have their custom formats that they ingest that interchange format into:

https://en.wikipedia.org/wiki/SEG_Y

Fundamentally you've usually got one to three dimensional data, metadata for the entire file, and then metadata for (potentially) each column along some axis of the file. It's interesting that CDF, etc have had zero traction: I guess it's just that each field has its little data format subculture and there's not much incentive for collaboration?

I currently work at an exploration geophysics firm and was just talking to my boss about this yesterday. His comment was that data acquisition had an arms race to the bottom in the late 80s and has become complete commodity object. This has results in data from the 80s through today being no different as they're still using the (mostly) same tools and techniques.

There is no incentive to better your seismic surveying since stuff from the 80s is "good enough" for all the processing shops.

Yep, that sounds about right, at least on the data formats. A survey now is gathering orders of magnitude more data than it would have in the 80s (especially 3d/4d surveys), but I guess the formats didn't really need to change.

I don't miss writing code to guess whether a file has traces encoded in IBM floating point or IEEE; or handling SEG-Y headers which ought to be EBCDIC but are sometimes ASCII :-)

Did you find any documentation about the data format?

Yes, it's documented. Doesn't look to bad. They appear to do some custom delta compression but nothing too bad.

Reply on news.ycombinator.com