Hacker News Reader

Comment by 10g1k | Hacker News Reader

For those who don't recall: Back in the Dark Ages, there was a race to decode the human genome. The leading competitiors (wealthiest) were Celera Genomics and the Human Genome Project. After some time, Celera (headed by Craig Venter), announced they had done the deed. However, what Celera had actually done was used what they called a "shotgun method", which meant they took small samples here and there, then built a model of the genome with various statistical shenanigans. About five years later, after all the hype, they admitted they had not sequenced the entire genome.

Ref: https://www.technologyreview.com/2007/09/04/223919/craig-ven...

View on HN

As noted, the shotgun sequencing wasn't any worse than the traditional chromosome walking method used by the other human genome project, and neither created a truly complete sequence, which is only possible today with improvements in technology.

And it is worth noting that despite all the critiques of shotgun sequencing at the time as "cheating", chromosome walking sequencing is dead, and shotgun is what we use today. If anything replaces that it will be long read technology like that done by Oxford Nanopore sequencing, although we aren't really there yet (you still need to assemble that data and don't really get end-to-end sequences yet).

I feel a joke coming on.

Mother: "Amy, dear, you hardly touched any of your vegetables. Most of them are still on your plate."

Amy: "That's not true, Mom; I completely sequenced them into my tummy using the shotgun method."

Someone submit this to smbc comics !

yes! or XKCD

Not true now with PacBio Revio and new Oxford Nanopore chemistry and flow cells. We still use a bit of older Illumina data for polishing, but long read methods now as accurate. HiC and Bionano more important as supplements than 150 nucleotide pair-end reads.

And the cost of a T2T using a combination of these technologies is already well under $10,000 in reagent costs. The assembly is still a complex art especially for a messy chromosome like Y.

TIL that. I was 100% sure the DNA was completely sequenced. Even have a friend with a sequencing company. It is incredible how ignorant one can be. Thank you 10g1k and HN.

Thought so too, turns out we're right now

I'm not really sure what it even means to be 100% sequenced.

Does that mean that every possible polymorphism is accounted for?

Does that mean that 100% of possible polymorphic sites are known?

What about weird edge cases. You have a tandem repeat that is known, but in some small population of humans, there's two tandem repeats with an island of something else in there.

Found out you were wrong on the day you became right!

FWIW, the DNA was 100% sequenced. However, a proportion of that data that came back from the sequencing machine was too similar to each other, so it was impossible to assemble it into a coherent full set of contiguous chromosomes.

Imagine it like putting together a jigsaw puzzle, except that the picture on the front of the puzzle has lots of repeating motifs, so you end up with multiple pieces that look identical. You won't be able to tell how the whole thing fits together, but you can assemble the bits that are well-behaved and unique.

Modern technology gives us larger jigsaw pieces, which allows us to distinguish between almost-identical parts of the puzzle better. But I would note that the project linked did a huge amount of sequencing using very expensive methods to be able to resolve the whole thing.

For anyone interested, the problem mostly boils down to the sequence lenght we can read.

Modern sequencing techniques allow us o read 200 to 500 bases at a time. So after hat we need do find a way o arrange these short sequences into a single sequence. And this can be prety hard, especially when you are doing 'de novo' assembly[0].

Besides that, there is the fact that some regions of DNA are repeated[1].

[0] - https://en.wikipedia.org/wiki/Third-generation_sequencing#De...

[1] - https://en.wikipedia.org/wiki/Repeated_sequence_(DNA)

You are a bit behind on the available read lengths. The Oxford nanopore can produce reads as long as 300kb (k as in kilo) and the PacBio hifi can product 15 to 20kb reads. Finally a now defunct technology by 10x genomics called linked read sequencing also permitted longer regions to be sequences using short reads.

Thanks for pointing that out.

My knowledge cut-off period in this domain is around 2018. So it's not suprising that things moved on.

So if one orders WGS at 30x or 100x from someplace like Nebula, what do you actually get?

Unless they're explicitly saying that you'll be getting long read sequencing, what you'll be getting is a paired-end short read sequencing, likely Illumina (although similar output can be achieved using BGI and Element machines as well). You'll be getting fragments of DNA which have been size-selected to around 400-500bp in length and then sequenced 150bp from both ends, with an unsequenced gap of unknown length in the middle (and possibly an overlap if the fragment is smaller than 300bp), sufficient to cover the mappable parts of the reference genome with around 30 or 100 sequence reads on average. That data can be supplied in a couple of FASTQ files (usually gzipped). They may offer to align that against a reference genome for you, where you'll get a BAM file - make sure you have the same reference genome handy to compare against. They may also offer to detect variants, that is places in the sequenced DNA that are different to the reference, in which case you'll get a VCF file.

Because the sequencing uses short reads, it will not be able to resolve parts of the genome that are repetitive with a repeat unit longer than ~150bp. You'll have all the jigsaw pieces from the puzzle, but you won't be able to reconstruct some parts of the picture. Long read sequencing can help with those, but that's more expensive.

Yes, but the HGP didn't make a full sequence either, using their scaffold-based contig assembly. Both groups declared a truce and announced they were "finished with the first draft"

35 years ago: https://www.nytimes.com/1987/12/13/magazine/the-genome-proje...

33 years ago: https://www.nytimes.com/1990/06/05/science/great-15-year-pro...

29 years ago: the competition gets fierce https://www.nytimes.com/1994/02/22/science/scientist-at-work...

24 years ago: the race is alive! https://www.nytimes.com/1999/03/23/science/who-ll-sequence-h...

23 years ago: draft is complete https://archive.nytimes.com/www.nytimes.com/library/national...

20 years ago: https://www.nytimes.com/2003/04/15/science/once-again-scient...

2 years ago: https://archive.nytimes.com/www.nytimes.com/library/national...

The celera approach using shotgun was tricky because assembling the bits required a great deal of computational finesse and horsepower; "The final assembly computations were run on Compaq’s new AlphaServer GS160 because the algorithms and data required 64 gigabytes of shared memory to run successfully."

At the time the GS160 was the monster machine, a classic big iron UNIX, which could be tightly clustered- unified IO/filesystem between all the machines. The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST, and invented the Suffix Array with Udi Manber.

The public project had its own issues, as many of the teams assigned to work on it still had the "cottage industry/artisinal/academic" approach, then Eric Lander came along and turned it into an industrial process, and parlayed that into running the Broad Institute, a privately funded MIT/Harvard research institute in Boston.

These days petabytes of sequence data are generated every day and stored in clouds. The genome has been a fundamental tool for shaping our studies of humans, although its true potential for understanding complex phenotypes remains elusive.

> The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST...

So... um... BLAST processing?

I think blast is for mapping sequences on existing genomes, which uses similar algorithms but applied for a different purpose.

Lets not forget how Perl Saved The Human Genome Project [1], which I remember seeing in a print magazine at the time.

As someone who is in the genomics world now as a software person, I find it amusing that I've gotten more into perl over the last year or so. It has its place, in the way that grep/sed/awk/etc does.

[1] https://news.ycombinator.com/item?id=30327812

Lincoln Stein is great, and perl was certainly critical to many processes, but it was and remains fairly niche in genomics, which used much more C++, Java, and later Python.

IMHO the person who "saved the genome project" was WJ Kent, who developed the assembler, BLAT, that the public project needed. I strive to point out that he wasn't a sole hero, nor was Lincoln. What I really like about BLAT is that while Celera was using Big Iron UNIX (massive 64-bit 64GB machines with 10s of terabytes of central high performance storage), BLAT ran on a cluster of linux machines, right around the time that people were waking up to the fact that linux was becoming a useful tool for scientific data processing. BLAT's design allowed it to work on a cluster of cheaper/smaller machines, while celera's algorithm really needed a massive shared memory single machine. it was sort of a microcosm of the larger battle being fought between Big iron UNIX and little intel linux at the time.

I know hn isn't the place for memes, but, I'm obligated. Imagine what they could do with a beowulf cluster of pentium pros?

I feel like this is the one domain where the beowfulf cluster meme is still funny

There was a NYTimes profile at the time of Kent's efforts, and they IIRC literally got the UC Santa Cruz administration to hand over desktop computers destined to go to various people for a few months to assemble into a cluster. So it was likely a bunch of pentium pros or similar. Not a Beowulf cluster, of course, but whatever clustering software Kent had hand rolled out of C for the purpose.

They weren't using pentium pros, they were using Alpha's because the memory interconnect between a beowulf cluster of pentium pros would be orders of magnitude slower. So likely 'the same, but much slower'.

oh, ha. yeah, that was sort of the root of the joke. companies like Celera, or whoever had the money would write a (large) check for big iron.

smaller organizations were realizing they could get slower computers power pretty cheap, and they could get a whole lot of them. There was a little flurry of activity with flocking algorithms, objective-c had a little renaissance due to swarm computing. probably the most famous and lasting was map-reduce, from alphabet, but they were called google back then.

there were a bunch of clever little tricks, like channel bonded network cards to make multiple cards look like one fast card, so you could double or triple bandwidth.

The beowulf cluster joke was kind of the spirit of the scrappy, make something cool out of junk approach while calling companies like sun, hp, compaq dinosaurs. and it continued for a while with people building stuff like this - https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-p... I think there was some weirdness with export controls of ps2's for this kind of stuff.

I don't remember what pc chip was the new hotness back then, it was a good 20 years ago and my memory is dim.

but that, I think, captures the gist of the meme.

www.clustercompute.com

I agree from the human genome sequencing perspective and the superlatives of saving the HGP. But at the time I was a standard software dev who thought the topic was cool, as was this new fan dangled language named perl.

However, I got into the genomics world in the early aughts, and the perl hung on and on and on. I remember starting a new job in the mid-teens, and one of the first tasks I had was to port over a legacy perl script. Such is the world of scientific software. 99% of the people I know who have touched perl since ~2003ish are in the bioinformatics space.

Small correction: BLAT is a local alignment tool Jim Kent also wrote. I think his assembler you're referring to is GigAssembler (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/).

I also love that all of their genomic tools are released as a virtualbox virtual machine and their entire site can basically be run locally with personal genomic data. I think it’s called genome in a box.

And if I remember correctly BLAT was so useful because it could be run on machines with less cpu power by loading more data into memory…or was it the other way around?

I had the privilege of working a co-op term at Stein's lab at the OICR. I encountered quite a bit of Perl during my short time there, and have yet to see it elsewhere in the modern enterprise tech world. BioPerl in particular stands out as a fairly substantial project in the bioinformatics space.

Amazon still has some Perl kicking around, mainly on the main website. I don't think there's any new development though, just maintenance and gradual replacement.

And Jim Kent!

Hi, I'm in software and looking to get into genomics/biotech. I'd like to learn more about how you made the move. Is there a way I could dm you? I saw no email in your profile.

Re: Perl, I very much agree. I've mostly switched to Ruby and `grep -P` but Perl really does have a place. `perl -pie` is still the most expressive/powerful option in some cases. Even if not directly used much anymore, I think most people vastly underappreciate how much Perl affected it's successors. Even as historical significance it's worthy of admiration.

  > The primary author of the shotgun assembly was Gene Myers

No conflict of interest there!

His wife, Chromosome, was an uncredited contributor.

My recollection of the contentious issues at the time are that a) the Celera project needed the information being made public by the NIH/international Human Genome Project in order to assemble sequences, but was keeping secret its own results, and b) it was intent on acquiring IP rights before the information entered the public domain.

Yes this is correct. Shotgun sequencing needs to be aligned to a reference which in this case was publicly free to access.

This comment is needlessly negative. What they did had value and they did not hide what they did either, at least that wasn't my impression. The statistical "shenanigans" is better termed as an innovation.

Pretty sure that the announcement was that the sequence was "essentially complete" (or something like that). Here's a press report from the time that provided the caveat:

http://news.bbc.co.uk/2/hi/science/nature/2940601.stm

> The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with.

I don't think this was a secret. I remember reading an article (IIRC in Readers Digest) nearly ~20 years ago which specifically referred to it as a "shotgun method". I wasn't (and still am not) enough of an expert to have known that was a less authentic/accurate method, but at least at the time the tone was not that they were doing anything illicit.

if you go over the maths on the shotgun method, not taking into account systematic difficulty in sequencing (like tandem repeats), the likelihood of incomplete coverage vanishes much more rapidly than you'd expect, and after about 5-10x coverage volume, coverage is basically linear in the total amount of sequencing.

Shotgun sequencing could be better explained than "statistical shenanigans": imagine blasting a huge collection of identical long strings with a shotgun, each breaking into different fragments, and realizing overlaps in the fragments, collectively, can be enough to uniquely restore one copy of the original string.

Anybody familiar with news sources which questioned Celera accomplishment before they admitted how they did it?

Parent is weird in claiming celera was hiding things. The second half of the 90s was a very vocal spat between Celera and the HGP about sequencing methods and Celera saying shotgun sequencing was superior. And the HGP was publishing updates to their genome every 24 hours while Celera published their sequence on completion of the initial draft, which showed their coverage. The fact that there wasn’t 100% coverage wasn’t hidden to anyone who read the published papers. (The abstract for the Celera paper even states “reduced the size and number of gaps”, which plainly indicates there were still gaps…)

Huh, narration matters. Thanks.

http://news.bbc.co.uk/2/hi/science/nature/2940601.stm

> The decoding is now close to 100% complete. The remaining tiny gaps are considered too costly to fill and those in charge of turning genomic data into medical and scientific progress have plenty to be getting on with.

OP is being silly. Nobody was fooling anyone.

Reply on news.ycombinator.com