Hacker News Reader

Comment by chaxor | Hacker News Reader

IPFS or torrent are the best options for distributing data

That is incorrect for scientific data. The limitations are:

a) Massive data volumes (~100 Gb - 1 Pb/project)

  ai) This means that data is typically stored on limited access machines like HPC clusters

  bi) This also means that shipping this data around is financially expensive, and cannot be supported purely by small client machines

b) A low number of seeders; scientific data is not exactly popular, and there may be network restrictions on uploads through the typically used networks;

c) The requirement for a data legacy; torrents are fantastic for ephemeral data (e.g. operating system builds), but are terrible for data that must be archived and kept for potentially decades to centuries.

Most scientific datasets are not that large. For every CERN type study there are 1000x biology papers with n=3 where the collected results sit in a single tab of an Excel document.

Very true, but if you're planning on expanding the work of a small, pilot study like that and you don't have the people who were involved in the original you probably need to recreate the study (to shake out the kinks in the protocol, confirm results for yourself, etc).

It would be challenging to find a solution robust enough for CERN type data but also simple enough for an n=3 undergraduate research project (that may have yielded some interesting results).

I don't know what the solution is there. My intuition is that university libraries could be involved, and that a data librarian could help you get your small study into shape or be embedded at a percentage effort on a large study.

Those do not belong in IPFS. They won’t be replicated and may die.

Depends on what you mean by "biology". Maybe some traditional naturalist-style data about where butterfly species were spotted or something like that could just be a spreadsheet, but in more modern biology studies things like RNA-Seq are used, which generate gigabytes or even terabytes of data per paper.

Those biologists might have huge image archives, even if they use microscopes.

In biology we now routinely produce datasets in the multiple terrabytes range. It can easily be n = 3 x 10 TB such as for example imaging 3 fly embryos by light sheet microscopy.

Scientific datasets like that can be very easily hosted at one of the repositories such as zotero. The only reasons people don't do that is a vague sense of insecurity about having someone declare their analysis botched, vague legal worries, vague unwillingness to do the very small amount of work required to publish data, or the hope to milk a dataset for more papers before anyone else gets a chance.

I guess you meant zenodo, not zotero.

Yes, oups... I always mix them up

I'm not sure I agree with this view. If the NIH or DoD, etc serves data over FTP, they can certainly also serve it via IPFS. It would be much better for them to do so. Then, if anyone pulls that data (even if it's only a small dataset, say 2 TB, to their house or lab) it is now more available (since now NIH and other labs are sharing it). I would imagine that the data centers serving these files would be happy to reduce the bandwidth they have to serve by allow other labs to help out as well.

It only adds. I don't understand how it subtracts.

Not in out 5 year experience trying to use with GeneNetwork.org to share large and small datasets. IPSF is marketed as simple but is complex—or even over-engineered from some perspectives. Hate to say it, but Dropbox is much easier and stable.

Hoping IPFS makes it someday because the idea is great.

It seems you have confused 'distributed' with... Something else. Regardless of how easy it is for you, or how complex you found it, the data is distributed via ipfs.

What data?

Any data shared in the network.

I don't understand what you're responding to. GP said they tried using IPFS with their project, but it ended up being too complicated and they opted for Dropbox instead.

I don't understand what you think GP was responding to. Dropbox isn't distributed. The problem they articulated isn't relevant to the problem solved by ipfs.

OP was responding to this:

> IPFS or torrent are the best options for distributing data

And suggesting that IPFS is not a good option for distributing scientific data due to its complexity.

I think you are thinking "distributed" in the "distributed computing/decentralized/p2p" sense.

In this discussion it means "distributed" in the "made available" sense.

The parent did not use IPFS because it didn't work for them. So no, the data was not distributed via IPFS.

Every time I remember that torrents exist, that blows my mind.

Reply on news.ycombinator.com