Hacker News Reader

That's a different category of big data. I worked for a big pharma and they were building their big data department with Spark and friends. I was quite surprised that their biggest dataset had something like 200 GB.

At the same time, though, there was a lot of DNA sequencing data, we were designing CRISPR probes etc. But Spark and Hadoop aren't really that helpful in this area, so the Big Data team wasn't involved in those.

View on HN

I think it does depend on the problem. Genetic stuff always seemed not easily parallelizable like my field (physics simulation) is. That said, the culture here is that MPI works and thus cray still builds computers that work better with it, so we use MPI so it works...etc etc.

> Genetic stuff always seemed not easily parallelizable

Not really. Processing one sample may take a few hours but if you have hundreds or more samples, it's an obvious axis for independent parallelization.

The cool kids use Nextflow or CWL these days. It's something like `make` - it remembers what you've already computed and what the dependencies are - but it uses a batch engine like SGE/Condor/AWS Batch to actually execute the jobs.

Reply on news.ycombinator.com