For example, if you browse the UC Irvine ML datasets
https://archive.ics.uci.edu/ml/index.html
You'll find that many are in csv format.
If you do a search on data.gov
http://catalog.data.gov/dataset#sec-res_format
You'll see that it's about as popular as JSON.
Also, the World Health Organization
http://www.who.int/tb/country/data/download/en/
Also, many of the datasets at kaggle are in csv format.
https://www.kaggle.com/datasets
And this isn't that surprising, it's human readable, and gets the job done, and zipping will give decent compression.
I'm not sure who you think the target market for this would be, but I'm sure that if it's an efficient local format, you could probably get the ML crowd on board.
Also, we routinely talk to developers who complain about the difficulty of consuming data snapshots from partners, parsing it, trying to understand how it has changed since last time, etc.
With high value datasets, people frequently build an API to combat these problems. But it's hard to design a good API, and even if you succeed, it has to be secured, documented, scaled, and maintained indefinitely.
or would it be 'use this hostname to sync the nom to your own computer'
Or would it be a dsn sort of and you just instantiate a client and your on your way?
Or, some combo?
``` <a href="http://www.who.int/tb/country/data/download/en/::case-data/b... the Data</a> ```
The super REST harmonious way to do this would be to define a new media-type for Noms databases with a smallish document that links to the component parts. Like torrent files, but using URLs (maybe relative URLs) instead of SHA1 hashes for the components, maybe?
The way we have it now is nice in that any valid URL can be used to locate a database. I am loathe to restrict that.
Interesting point though - thank you!
For example, (1) our command line tools use URL-like paths which implies "use this hostname" (to copy-paste into terminal), (2) we have some in-browser visualisations like http://splore.noms.io/?db=http://demo.noms.io/cli-tour which implies more of a "click here" type UI.
I don't need to make sure I've got postgres 9.5 setup with a particular user account & set configs for the password, start ES (but not version Y because of a feature change) on port Z, etc. I don't need to manage making sure the two branches I'm looking at don't overlap or try to write to the same database. Keeping multiple results and comparing their output can be easily done as they're just files to be moved. Small tasks that read a file and spit out another can be checkpointed just by making them look to see if the file they expect to create already exists.
I'm hugely in favour of CSV for external data too. Sure, provide other options as well, but I love that the "get all the data" command can be as simple as a curl command. I don't want to read your API docs and build something custom that tries to grab everything, I don't want to iterate over 2M pages, I don't want to deal with timeouts, rate limits, etc. Just give me a URL with a compressed CSV file.
All the problems that come along with it, for me, are related to poor data management which I doubt a format change would fix.
Maybe CSV isn't the best internally, but for a vast amount of cases it's nearly the best and gives you a lot of flexibility. My general advice would be to start with CSV unless you've got a good reason not to, and then try and move to a different line based thing (jsonl, messagepack?). It is highly unlikely to be the biggest problem you have with your data, and the time spent putting it into a more "sane" format is often (in my experience) better spent on QA and analysis of the data itself.
I'd say the current problem is that lots of data is available only either in excel files, pdfs, and APIs pointing to a possibly constantly changing data store.
At first glance, it strikes me as a solution for people storing something like scientific data sets rather than application data. In which case, posting CSV files on a website is pretty much best-case scenario.
EDIT: Although, in the "scientific data set" scenario, I'm not sure how much value there would be in storing version history.
For large scale datasets, I'd be looking at GIS data maybe.
If this is supposed to replace CSV cool, but there is a lot of ways to cross that bridge.
Curious however, what the target use case is. Is it a format, or a database , or both?
And cool if so, but the CSV analogy is really confusing.
When I think of CSV I think of a ghetto data exchange format that I can send or receive to a less technical person. As I understand nom, it does not sound like its for non technical people.