Hacker News Reader

I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores.

I could be missing something here so anybody more knowledgeable should chime in.

View on HN

It might not be optimal; however I wonder if most of the primary sources this is intended to target (i.e. govt) prefers to use textual data?

>I wonder if most of the primary sources this is intended to target (i.e. govt) ...

The second sentence on their website appears to state otherwise: "As a team we have a bias towards supporting scientific + research data use cases."

Edit: Well, on further reading they do go on to say, "What we're building isn't quite ready for prime time yet, but if you want to play around...", so perhaps I should not jump the gun :)

Nope. Any really large government data is not text -- it's mostly in older open binary formats (e.g. weather data in GRIB from NOAA) or expensive proprietary binary formats (e.g. healthcare data in SAS from CMS).

Reply on news.ycombinator.com