New businesses eye the opportunities in managing genome data

- FASTA: a sequence or set of sequences representing an entire genome. Usually used (in human genomics) to represent the "reference genome", but very rarely individual genomes.

- FASTQ: a set of sequences with associated probabilities at each position (representing the probability that the base at that position is correct). Used to represent the output from a sequencing experiment, where you may get 100s of millions to billions of short reads (order of 100 to 10000 bases in length) from a biological sample.

- BAM: "binary alignment map". Stores the data from the FASTQ (sequence and quality) in a way that "aligns" it to a reference genome -- identifying where the read "came from" (more formally, mapping each base in a read to the most likely corresponding base in the reference genome)

- VCF/gVCF: "variant call format". You can think of this as a diff between the individual sequenced and the reference genome.

In most cases in human genomics you wouldn't construct a full FASTA from the individual (something you would do through a process called assembly). Instead, you would sequence the sample (producing FASTQ), align it to a FASTA reference genome (producing BAM), and call variants (producing VCF). The VCF is much smaller than the other formats, and that size differential is most of where GDC would get its performance: all in all, we're usually not that different from the reference.

The big storage problem usually ends up being storing and manipulating FASTQ and BAM files, because these are the (almost complete) original data from the sequencing run, and occasionally there's a need to keep them around:

- you may want to run a new analysis that wasn't done and so wasn't encompassed in the original VCF - you may want to know the underlying quality of the data that created a variant call. Sequencing is a stochastic process, subject to a variety of types of error. Even though VCF calls typically have an estimate of quality, in many cases there's no substitute for looking at the original underlying data. - you may have a legal or contractual obligation to maintain this data. (e.g., under CLIA regulations laboratories offering clinical sequencing may be required to store the raw data underlying a clinical result for a number of years).

So, how do you store FASTQ or BAM more compactly? BAM is already compressed -- block gzipped, to be specific -- but it still stores all the information explicitly. The obvious first step is reference-based compression (e.g., CRAM: http://www.ebi.ac.uk/ena/software/cram-toolkit), which elides sequence data that is identical to the reference genome, and actually gets rid of almost all of the storage needed to store the sequences.

Other than some easily compressible stuff, like read names, the main thing left over is the quality scores -- the sequence of numbers telling you how "high-quality" each base sequenced was. In raw form, the qscores take an equal number of bytes as the sequence reads, but once we've used reference -based compression to elide the bases, they're by far the dominant component. Unfortunately, compressing quality scores well is a difficult problem. There's been a lot of work done in this area (e.g., http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3832420/), but it's by no means a solved problem.

So if you're into data compression, coming up with a better way to compress FASTQ/BAM files is still a valuable line of work.