Hacker News Reader

Comment by i000 | Hacker News Reader

I work at an academic institution that generates and analyzes large amounts of genomic data (clinical sequencing of cancer patients). We were really impressed with the Google cloud and executed several large proof-of-concept projects on it. Honestly, being able to spin up 1000 machines to process 10s of thousands of samples within hours (using cheap preemptible machines) is a huge step forward.

The biggest challenge we face is data protection. Sequencing data is not considered protected health information so theoretically HIPAA regulations do not apply, but this may change in future so nobody is willing to take responsibility. If sequencing data is considered PHI, HIPAA requires a BAA between the institution and Google. The process of getting these in place was so convoluted and difficult that it was just easier to use another provider more closely associated with our institution.

View on HN

Glad to hear this worked out for you (particularly preemptible VMs which I worked on). Sorry about the HIPAA/ BAA issues, which institution are you with so I can have someone do the work so you can have a better experience next time.

Disclaimer: I work on Compute Engine.

Yes please -- if you're able to share which institution you're with, I'd like to follow up on the BAA hassles. (We do regularly sign BAAs with institutions, but I know that, as with many things legal, simple things don't always stay simple.)

Disclaimer: I work on Google Genomics.

I'd love to do more with Google at my university but we couldn't get Google to even start talking about signing our BAA or other agreements. It was a non-starter for our legal department.

We ended up moving towards AWS because they were willing to sign. It sucked from my perspective because we already had such a tie in with Google with our email/ collab platform.

In the couple years since then our agreements have only gotten stronger.

The best way for Google to address this is to sign something with Internet2 and then the institutions can use that contract instead of negotiating separately.

In our case Google was very proactive and willing to sign a BAA. But the terms of those agreements have to satisfy both partners which for some reason ultimately did not happen.

For the layman, is a BAA this? https://en.wikipedia.org/wiki/Broad_Agency_Announcement

(The other terms were a bit easier to google)

Here are the other terms:

HIPAA: the acronym for the Health Insurance Portability and Accountability Act that was passed by Congress in 1996

PHI: Protected health information

BAA = business associate agreement. Contract between an organization that handles personal health information (e.g. a research lab in a university ) and a business that does work with that data (e.g. Google Genomics).

Edit: link to relevant definition http://searchhealthit.techtarget.com/definition/HIPAA-busine...

No, it's a Business Associate Agreement, part of HIPAA. (See a sample here: http://www.hhs.gov/hipaa/for-professionals/covered-entities/...)

Basically an agreement between organizations that there is an understanding that PHI (protected health information) is at stake and that both parties are aware of their obligations to protect it.

Just curious, was this primarily hotspot or other smaller panels? We looked at this internally as well but the available machine types have very limited ram for things like the exome panels we're starting to work with.

Please take another look; we introduced custom machine types back in November:

  http://googlecloudplatform.blogspot.com/2015/11/introducing-Custom-Machine-Types-the-freedom-to-configure-the-best-VM-shape-for-your-workload.html

Which would let you adjust your RAM to vCPU ratio (as low as ~1 GiB per vCPU and up to 6.5).

Disclaimer: I work on Compute Engine.

I am wondering if you did a cost analysis. Last time I checked a dedicated on-premise HPC was still cheaper than using the cloud. Actually the CPU hours are cheaper in the cloud but data-transfer (bandwith) and storage make it more expensive (might not apply to storage anymore but AFAIK data transfer costs are still an issue). What is your experience ?

Of course we did this, but cost is not the only consideration.

Frankly, CPU-hours and TB-years are still within 5-10% of the total cost providing sequencing results to the patients. What is really important is that the cloud allows us to determine exactly our cost-per-analysis without paying for excess capacity or maintenance.

What matters also is reliability, redundancy, and peak capacity, where the cloud (both Google and the other provider) win hands down.

Just a note, you don't pay for sending data to any provider (ingress) only for pulling data out / serving it (egress). You can put your GCS buckets behind Cloudflare to take advantage of "interconnect" pricing of $.04/GB.

Disclaimer: I work on Compute Engine.

good to know. Thanks for the info !

You can use https://aws.amazon.com/directconnect/ to reduce egress costs from AWS down to $0.02/GB.

As boulos points out ingress is free -- the other question is how much you actually need to egress. Transfer within a GCP region is generally free as well, so unless you really need your giant result set in its raw form, you can potentially just leave it in GCP and do any analysis/summarizing work you might otherwise have done externally within GCP.

Taking this to the extreme, you might even fire up a Windows VM so you could run some analysis or reporting tool there (say, Tableau), within GCP and then only need to egress the Dashboard or Report that it generated. You'd still have the raw data, but you would only be paying to store it. Especially if you're only keeping it around for archival purposes you could put it in a Nearline bucket and really minimize your costs.

(note: I work on GCE)

You could do the same before on AWS. And AWS spot instances are way way cheaper than Google's offerings.

I bet your university has hundreds of computers idling at night - why not use them?

Sounds like exactly what NPS (netapp private storage) was designed for.

I'm confident a Google version could be spun up if there were the appropriate demand.

Reply on news.ycombinator.com