siganakis

What we have is 32 bit integers (or anything really) that we store using bit vectors (which are implemented as 64 bit integers).

These structures are read only. I believe that there is some work on making updatable Wavelet trees but I think that this would be very expensive.

Reads are primarily Seeks (maybe 70%) I think, but Access is also important - so the order must also be encoded.

Storing keys and values is expensive in terms of memory (storing everything twice) so what we are trying to do is to manipulate the data such that we can do these operations quickly while using the minimum number of bits.

Edit: Fixed iPhone related spelling / grammar.

Just to pick you up on one thing, we have sequenced thousands upon thousands of genomes since the human genome project finished.

For example there is the 1000 genomes project [1], and a project I am working on is sequencing about 100 ovarian cancer tumor / normal pairs. Most of this sequencing is complete, its the bioinformatics that takes the time.

GWAS studies (studies of correlation between disease and common SNP's) have not told us much that is actionable, but have provided us with "low hanging fruit" for further study, which is valuable.

Its very early ways, but genomics will completely change the diagnosis and treatment of cancer over the next 10 years.

[1] http://www.1000genomes.org/

Does that mean that if I have 2 workers that I can be certain that they are on different logical machines in the event of a machine failure? Or may my 2 workers actually be on the same machine?

I have been using AppHarbor for the past few months and have been very impressed.

The new pricing seems reasonable, but I would like to know exactly what a "worker" is (e.g. is it a worker process, a thread, or a server?).

All in all, its possible to get up and running with a < 10GB shared SQL Server instance running a site with your own domain for $20/month. Seems like a pretty good deal to me.

I would be very interested in hearing more about that project, as it sounds very similar to what I am working on for my cancer research.

The two problems I keep running into are that run length encoding means you have to read the entirety of the bitmap to get to the value you want, and looking up a particular value for a particular row (where you have to scan all the bitmaps for that column) - which as you state makes high cardinality columns slow. I would be very interested in learning about ways to overcome these issues.

Hit me up at terence [at] siganakis [dot] com if you would like to talk more, or on twitter at @siganakis (since I can't find any contact information in your profile).

Cheers

Yes but each additional index adds to the storage requirement of the data. With column oriented designs each column is often indexed with bitmap indexes which are very small (often the index + raw data is smaller than the raw data stored in an array thanks to compression). Index intersection is also much faster with bitmap indexes rather then b-trees.

It is also worth noting that some of the technology underpinning FastBit is actually patented (the Word Aligned Hybrid method of run length encoding).

http://www.freepatentsonline.com/6831575.html

Basically the patent covers the method of run length encoding. Each bitmap is stored as an array of words (32/64 bit ints), with each word being either a "fill word" telling it that the next x words are all 0/1 or a "literal word" containing an actual sequence of 0's and 1's.

Its a pretty sorry state of affairs when core, relatively simple data structures are patentable. Imagine if someone had patented the linked-list or binary search tree!

Interestingly, FastBit (which uses this IP) is released under the LGPL

[Edit: improved clarity]

Thanks for that, I'll check them out.

From my work there are 2 key advantages that using a column oriented approach bring:

1. There is far less storage required to represent data in a column oriented database due to the ability to compress like data. This storage benefit is really important when you get to tables, as it means you can hold more of your data set in memory at any one time.

2. For tables that have a lot of columns, but queries that only touch a few at a time, it means less work for the query to do, since there is no jumping from row pointed to column pointer.

Cheers

My understanding is that its not the gene that is patented, but the processes surrounding the isolation of it, or the actual isolated product.

There would be no patent infringement if you sequence an entire genome (including BRCA1 and BRCA2), as you are not dealing with the actual chemical processes or products covered by the patent.

They have a monopoly on their specific test, as it uses the chemical products covered by their patents (although the test aspect of the patent appears to have been thrown out).

Creating a different test based around, say Illumina or SOLiD sequencing is not going to infringe on this patent.

I'm clearly in the minority, but I'm not sure I would really be that attracted discounted sky-diving. If I ever try it, I want it to be with some delta force or SAS guy who is VERY well paid.

The scans that that companies like 23andme complete are for common SNPs (single nucleotide polymorphisms), not full DNA sequencing, which is much cheaper but provides only a sample of your genotype.

Research scientists use the same technology to conduct GWA studies (Genome Wide Association), which seek to determine if these SNP's are associated with a disease phenotype across a population. These studies typically find one or two SNP's that are associated with a disease (e.g. Breast Cancer, Diabetes, Macular degeneration, etc), but with extremely small odds ratios. These studies help narrow down possible genetic areas of interest for other researchers.

As they are population based studies, they are of no use clinically at all. As one of the first people to document the GWA study methodology put it:

"Patients inquiring about genomewide association testing should be advised that at present the results of such testing have no value in predicting risk and are not clinically directive." ~ Teri A. Manolio. Genomewide association studies and assessment of the risk of disease. The New England Journal of Medicine, 363(2):166–176, July 2010

The trouble is that companies like 23andme tell their clients that because they have a particular SNP, they are likely to get a disease which is extremely disingenuous. They do this by reporting the odds ratio found by a GWA study as their personal odds ratio of getting a disease. The research they use to make their findings is not applicable to single cases yet they claim it is. This leads to stress in clients and then ties up medical resources better spent chasing actual disease, rather than phantoms.

I'm all for open access to genetic data, but the current crop of consumer genetic companies are pedalling fear rather than open information, and probably deserve to be regulated.

One of the things that cancer is good at is switching "on" telomerase (enabling the cancer to spread), so naturally working out how to switch "off" telomerase in cancer tumors has actually been one of the key vectors in cancer research.

Learning about the four elements is useful if one approaches it from the perspective of learning about how such models are built.

Its not always about the outcome, but rather the process of reasoning.

Its also an excellent way to think about your app.

What are the core things your app should be doing from a data perspective? Expose those things in an API.

Making the API available over REST or as a Web service is extremely simple, and its amazing how useful it becomes over time in your own development.

It also makes you think twice about changing how these core methods change, which is often a good thing (especially when you are working in a small team and its tempting to make "little" changes to solve your current problem without thinking about its implications).

Cheers

With regard to #2, if your friends aren't on Facebook or Twitter, they are probably even more unlikely be on this new service.

I think that this problem is pretty much solved by mobile phones (sms, call).

In terms of discovery of casual events, there might be something there though.