Comment by dekhn | Hacker News Reader

Comment by dekhn | Hacker News Reader

Parent Full thread

Yes, but the HGP didn't make a full sequence either, using their scaffold-based contig assembly. Both groups declared a truce and announced they were "finished with the first draft"

35 years ago: https://www.nytimes.com/1987/12/13/magazine/the-genome-proje...

33 years ago: https://www.nytimes.com/1990/06/05/science/great-15-year-pro...

29 years ago: the competition gets fierce https://www.nytimes.com/1994/02/22/science/scientist-at-work...

24 years ago: the race is alive! https://www.nytimes.com/1999/03/23/science/who-ll-sequence-h...

23 years ago: draft is complete https://archive.nytimes.com/www.nytimes.com/library/national...

20 years ago: https://www.nytimes.com/2003/04/15/science/once-again-scient...

2 years ago: https://archive.nytimes.com/www.nytimes.com/library/national...

The celera approach using shotgun was tricky because assembling the bits required a great deal of computational finesse and horsepower; "The final assembly computations were run on Compaq’s new AlphaServer GS160 because the algorithms and data required 64 gigabytes of shared memory to run successfully."

At the time the GS160 was the monster machine, a classic big iron UNIX, which could be tightly clustered- unified IO/filesystem between all the machines. The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST, and invented the Suffix Array with Udi Manber.

The public project had its own issues, as many of the teams assigned to work on it still had the "cottage industry/artisinal/academic" approach, then Eric Lander came along and turned it into an industrial process, and parlayed that into running the Broad Institute, a privately funded MIT/Harvard research institute in Boston.

These days petabytes of sequence data are generated every day and stored in clouds. The genome has been a fundamental tool for shaping our studies of humans, although its true potential for understanding complex phenotypes remains elusive.

Lets not forget how Perl Saved The Human Genome Project [1], which I remember seeing in a print magazine at the time.

As someone who is in the genomics world now as a software person, I find it amusing that I've gotten more into perl over the last year or so. It has its place, in the way that grep/sed/awk/etc does.

[1] https://news.ycombinator.com/item?id=30327812

Lincoln Stein is great, and perl was certainly critical to many processes, but it was and remains fairly niche in genomics, which used much more C++, Java, and later Python.

IMHO the person who "saved the genome project" was WJ Kent, who developed the assembler, BLAT, that the public project needed. I strive to point out that he wasn't a sole hero, nor was Lincoln. What I really like about BLAT is that while Celera was using Big Iron UNIX (massive 64-bit 64GB machines with 10s of terabytes of central high performance storage), BLAT ran on a cluster of linux machines, right around the time that people were waking up to the fact that linux was becoming a useful tool for scientific data processing. BLAT's design allowed it to work on a cluster of cheaper/smaller machines, while celera's algorithm really needed a massive shared memory single machine. it was sort of a microcosm of the larger battle being fought between Big iron UNIX and little intel linux at the time.

I know hn isn't the place for memes, but, I'm obligated. Imagine what they could do with a beowulf cluster of pentium pros?

I feel like this is the one domain where the beowfulf cluster meme is still funny

There was a NYTimes profile at the time of Kent's efforts, and they IIRC literally got the UC Santa Cruz administration to hand over desktop computers destined to go to various people for a few months to assemble into a cluster. So it was likely a bunch of pentium pros or similar. Not a Beowulf cluster, of course, but whatever clustering software Kent had hand rolled out of C for the purpose.

They weren't using pentium pros, they were using Alpha's because the memory interconnect between a beowulf cluster of pentium pros would be orders of magnitude slower. So likely 'the same, but much slower'.

oh, ha. yeah, that was sort of the root of the joke. companies like Celera, or whoever had the money would write a (large) check for big iron.

smaller organizations were realizing they could get slower computers power pretty cheap, and they could get a whole lot of them. There was a little flurry of activity with flocking algorithms, objective-c had a little renaissance due to swarm computing. probably the most famous and lasting was map-reduce, from alphabet, but they were called google back then.

there were a bunch of clever little tricks, like channel bonded network cards to make multiple cards look like one fast card, so you could double or triple bandwidth.

The beowulf cluster joke was kind of the spirit of the scrappy, make something cool out of junk approach while calling companies like sun, hp, compaq dinosaurs. and it continued for a while with people building stuff like this - https://ncsa30.ncsa.illinois.edu/2003/05/ncsa-creates-sony-p... I think there was some weirdness with export controls of ps2's for this kind of stuff.

I don't remember what pc chip was the new hotness back then, it was a good 20 years ago and my memory is dim.

but that, I think, captures the gist of the meme.

www.clustercompute.com

I agree from the human genome sequencing perspective and the superlatives of saving the HGP. But at the time I was a standard software dev who thought the topic was cool, as was this new fan dangled language named perl.

However, I got into the genomics world in the early aughts, and the perl hung on and on and on. I remember starting a new job in the mid-teens, and one of the first tasks I had was to port over a legacy perl script. Such is the world of scientific software. 99% of the people I know who have touched perl since ~2003ish are in the bioinformatics space.

Small correction: BLAT is a local alignment tool Jim Kent also wrote. I think his assembler you're referring to is GigAssembler (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311095/).

I also love that all of their genomic tools are released as a virtualbox virtual machine and their entire site can basically be run locally with personal genomic data. I think it’s called genome in a box.

And if I remember correctly BLAT was so useful because it could be run on machines with less cpu power by loading more data into memory…or was it the other way around?

I had the privilege of working a co-op term at Stein's lab at the OICR. I encountered quite a bit of Perl during my short time there, and have yet to see it elsewhere in the modern enterprise tech world. BioPerl in particular stands out as a fairly substantial project in the bioinformatics space.

Amazon still has some Perl kicking around, mainly on the main website. I don't think there's any new development though, just maintenance and gradual replacement.

And Jim Kent!

Hi, I'm in software and looking to get into genomics/biotech. I'd like to learn more about how you made the move. Is there a way I could dm you? I saw no email in your profile.

Re: Perl, I very much agree. I've mostly switched to Ruby and `grep -P` but Perl really does have a place. `perl -pie` is still the most expressive/powerful option in some cases. Even if not directly used much anymore, I think most people vastly underappreciate how much Perl affected it's successors. Even as historical significance it's worthy of admiration.

  > The primary author of the shotgun assembly was Gene Myers

No conflict of interest there!

His wife, Chromosome, was an uncredited contributor.

> The primary author of the shotgun assembly was Gene Myers, who previously had written BLAST...

So... um... BLAST processing?

I think blast is for mapping sequences on existing genomes, which uses similar algorithms but applied for a different purpose.

Reply on news.ycombinator.com