Comment by rspeer | Hacker News Reader

Comment by rspeer | Hacker News Reader

Parent Full thread

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it.

I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been getting the data -- building it requires about 100 GB of raw data or 15 GB of computed data that can be imported into PostgreSQL.

I once tried git-annex. It turned out not to be a good choice -- its tools were flaky, its usage patterns confusing, it leaves a permanent record of your mistakes in configuring data sources, and it was very hard to convince to use ordinary HTTP downloads instead of trying to get read-write access to S3 (which wouldn't work for anyone but me). Now I have weird branches and remotes in my repositories, and weird data in my S3 buckets, that I can't get rid of in case someone tries to use git-annex in a way I told them would work.

After that I just went with distributing the data with plain HTTP downloads from S3. I wish I could do better than this. The only semblance of versioning is putting the date in the URL, and also people in Asia tell me that the build fails because their downloads from us-east-1 get interrupted. Oh, and if I ever stop paying for S3, everything will break.

If I tried making data reproducible with Dat, would it be safe to promise people that they could use Dat to get the data? Even if in the future I don't like Dat anymore?

For instance, do I have to commit to hosting the data somewhere? If not, who does? Does it disappear when people lose interest, like BitTorrent?

[1] http://conceptnet.io

I see you’re using JSON-LD in conceptnet. If you start passing that data around using distributed systems, you'll inevitably want to start incorporating content-addressed links into the data. I recommend looking into IPLD as a data model for handling that. https://ipld.io The spec is still open -- this would be a good time to give feedback and/or spell out your use cases in this space.

Thankyou for your work on ConceptNet. It's the best public knowledge graph in existence.

Just today I was using the multilingual Conceptnet-numberbatch word vectors[1], which would not be possible without your work.

To your point though - you can use Amazon S3 as a seed for Bitorrent downloads, which might help some and reduce what you pay. See [2]

[1] https://github.com/commonsense/conceptnet-numberbatch

[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...

Yep, ConceptNet Numberbatch is my work too, and it's been the most effective way to show that knowledge graphs matter -- that there is more to know about word relationships than you can get from distributional semantics ("word2vec") alone.

Oh really? Very nice... although I'm only using the aligned distributional semantic nature of them.

I have some background in question answering over knowledge graphs, though, so I'm familiar with their strengths.

I'd be interested to hear about what you're doing with it.

In my company Luminoso's work, it's important in building domain-specific models that can be used for topic detection, search, and classification. Beyond that, I use it for mostly the basic demos -- word similarity, text similarity, analogies, et cetera.

I believe based on its performance there that it should be a pure upgrade to the kind of applications that use word2vec, but I'd like to know what particular applications it's being used in besides my own.

Conceptnet looks really cool, I'm going to dig into that later tonight.

Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon.

Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and I can help you get started.

Should I just try using the tools at https://datproject.org/ ?

It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.)

And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.

We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm).

Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls.

> Maybe Dat plus Zenodo could do the trick eventually.

Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site.

We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`.

We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.

> We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper.

Awesome, congratulations! :):)

For distributed/p2p software these are just amazing times to be alive :)

Hmmm, is the data set something which would fairly naturally fit in a series of SQLite databases?

100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch).

However, if we gain traction and become cash positive, data sets this size would be good to cater to. :)

I used to keep it in SQLite (much easier to distribute than PostgreSQL). It worked a lot better than many other options I tried. However, rebuilding the database from updated data would take more than a day, and some queries were too slow.

Switching to PostgreSQL sped things up, at the cost of requiring a separate database process, dealing with psql's weird access control, and adding an inconvenient step of loading the data using COPY commands.

Hmmm, sounds like the data itself would be feasible then. SQLite could be considered just a data transport container for this purpose. :)

What underlying ontology does it use?

Its own, I guess? ConceptNet is really not so much about having an upper ontology, it's about relations between natural language words and phrases. Its set of relations is effectively a superset of WordNet's.

Hmm but for example. If I search "harry potter", one of the things I get is "harry potter is defined as... " "boy who lives under the stair". What can I do with this? "boy", "lives", "under", and "stair", while present in conceptnet, have no logical relation to anything else...

Reply on news.ycombinator.com