Hacker News Reader

Comment by astazangasta | Hacker News Reader

At first glance I'm not super into this... this seems like a bunch of opaque, hard-to-use utilities wrapped around already opaque, hard-to-use utilities (like GATK). What am I getting for my effort? I'd rather just use the low-level tool directly.

This seems more a way to pipe people into using Google Cloud Services for storing data than it seems a useful way to do bioinformatics. As with most of these cloud-genomics things, it's not useful to have canned analyses that I can't extend (for example, I am so far from giving a shit about transition/transversion ratios and would never think to store that information in my variant table) What's the big plus?

View on HN

We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc.

Disclaimer: I work on Google Genomics.

I'm really excited about moving to R/Python computing for my bioinformatics work, it's a much better workflow than the typical method of emailing matlab files to an in-house Xeon workstation, and it'll be easier to virtualize. This is something I've seen in dozens of labs and there's a huge amount of room to streamline workflows.

Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.

The problem with spot market and preemptible VMs is that much of the existing bioinformatics code often has run times of days to weeks with current sequencing depths. What's really needed is a way to have a VM suspend instead of shutdown when costs get high so it can resume once prices go down.

It depends. If you picture an entire pipeline what often happens is that you end up with both long and short running tasks. Tuning your pipeline to an environment like this involves figuring out what can go where. As an example we turn around a whole genome in just over a day but that's comprised of hundreds of individual units of work

Assuming you're running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs.

Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume).

Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you.

(note: I work on GCE)

I have to say that PDs attached read-only to multiple VMs is a very important part of my pipelines. And also a distinguishing feature of GCE.

Clound Dataflow on the other hand is difficult to use in practice - one of the main reasons is the large footprint of intermediate files (sometimes 10x the size of the input)

Currently we are running monolithic pipelines on the cloud where one instance does the job from start to finish (if it get's preempted - I restart). I plan the next version to be in some way "split-apart" but have not found a good strategy/tooling - any advice?

Hi! I'm an engineer on the Dataflow team. I'd love to understand the issue you mentioned about size of intermediate files, and help make Dataflow work well for you. Could you send a note to dataflow-feedback@google.com and elaborate?

@jsolson agreed on your points although I will say from experience that cloud dataflow doesn't work well with most bioinformatic workflows though unless one wanted to rewrite the tools from scratch. You end up fighting the system too much to make it work. Perhaps as the space evolves beyond the software wild west it is now this will change, I hope so.

Agreed. I'm more interested in the cheap semi-quick data storage ($0.01 / GB / month) than any of the workflows they offer. I'm also not interested in learning a new syntax to replace what I can do with UNIX / python / R (ggplot2) already.

If storage is cheap enough, and abstract, we can expect there to be a thriving ecosystem of tools to apply to it. They don't even need to be embedded in a single platform, but could be distributed, streaming, and on-demand.

Here are two examples from the iobio project.

A heads-up display describing a BAM (alignment) file streaming between two remote servers:

http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA128...

Streams another BAM from the 1000 Genomes project through two variant callers, with dynamically-updated statistics about the results. You can modify the variant quality threshold for each caller and the results are recomputed in real time:

http://iobio.io/demo/variantcomparer/

Edit:

Very nice visualization of interesting statistics from a VCF file, provided by URL or uploaded:

http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/A...

I agree the API is at this point is rather limited and it is difficult to use it in a way that would conveniently or efficiently replace any part of a genomic pipeline.

But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.

I strongly agree with the idea of deconstructing legacy files (BAM, VCF, etc.) and moving to an API-centric (vs. file-format-centric) world. The Global Alliance for Genomics and Health (GA4GH.org) is working on that, with good early success -- for example, IGV (genome browser) can now work with either files or standard APIs, as can Picard (one of the popular suites of genome analysis tools). See https://www.youtube.com/watch?v=d8EvXtz2uiA for details on the Picard support.

Disclaimer: I work on Google Genomics, and on the GA4GH APIs.

Reply on news.ycombinator.com