Practical advice for analysis of large, complex data sets

We couldn't justify Hadoop until we were +6TB... that was some years ago, pre ssd and pre big disks - our old rdbs server had got horribly complicated because it hit that scale - when it got to 10TB its raid controller corrupted the control files for the db. We had support and the consultants who came did a bunch of mind blowing work to restore it.

But by then we had everything on Hadoop and we were only maintaining one legacy app on the old box - which we had told the users was dead and buried in any case (but kept to maintain friendships and so on).

Back to my main point - some people think that analytics starts with a question, I don't, I think it starts with the data and instruments to inspect and understand the data. I think that you should get the data on some sensible box, put a good database on there that your guys can use (as they are SQL folks get an SQL one) and put R, R-studio and R-shiny there and start inspecting and understanding it.

If no questions or insight arises then little is lost and it can go on a cheap fileserver until someone needs or understands it. If nuggets of gold start appearing then you can invest further in both skills and kit. I would recommend Hadoop either if other datasets of the 10gb+ scale start appearing or if this one gets >4TB. The other database recommendation would be because of mega joins - which Hadoop does well, and the general need to build a cheap EDW - if you have lots of cash you can build an expensive EDW instead! 4TB because currently disks are ~6TB and there are nasty scaling bumps in not Hadoop world that start kicking in as per my experience.

IT are right to charge once this is established to be valuable; if it's work £250k a year to your business then it makes sense to be spending £50k a year to put SLA's and resiliance on it. If it's worth £2k... well...

As for the original question - I think most people realize that the big in big data rarely refers to the actual size (volume) of data but rather that we still do not have very efficient techniques for doing certain kinds of data processing when the underlying data simply won't fit into the relational model. A good example is text - there is a reason why Google uses some kind of inverted index and not a relational database in which it stuffs all the web page text.

I don't really know if the medical field has any use for text mining, graph processing and the like. While recommending Hadoop just because the size is in excess of 1 TB looks knee jerk, someone being down-voted for a fairly non-opinionated comment where they suggest exploring use cases before deciding is even more knee jerk.

I keep hearing about Hadoop and Spark for graphs and models, but in actual practice (WHICH DEPENDS ON THE SIZE AND SHAPE OF THE DATA), an awful lot of data-feeding problems are more easily solved by columnar data stores, or graph data stores. For text mining and NLP, you are almost certainly better off with redundant unstructured storage due to the fundamentally unstructured nature of text and communications. For massively parallel model exploration sometimes it's better just to spin up a bunch of huge EC2 instances. Hadoop and/or Spark aren't necessarily critical aspects of solving the problem. Once you get a redundant, distributed infrastructure to run things on, you may not need to jerk around with name nodes and boring HA grunt work. Sometimes an existing purpose-built implementation (often on top of well-oiled infrastructures) does the trick. Sometimes not.

One interesting project I've kept an eye on (OK I lied, I'm a contributor) is storage of genomic data against a distributed, graph-structured reference. It's really fucking hard. Even with Google, UCSC, the Broad, and the usual heavyweights involved, the milestones have occurred on a timescale of years, and the eventual adoption is projected on a scale of decades. This is with some of the best in the world working on it, in every time zone. Again: DECADES.

So... I'm not against distributed computation. But you had better damned well know what you're getting into. I worked at Google. I work on GA4GH. If you can afford to take the long view, maybe your problems are complicated enough to invest in that sort of infrastructure.

But maybe they aren't, and Amazon or Google or Microsoft has already built what you need because they needed it too, and they hired a hundred graduates from the best engineering schools in the world to support their implementation. Is it worth reinventing the wheel when there are people racing at Formula 100 level out there? Only you can answer that.

There's a reason people don't swat flies with Buicks. Sometimes all you really need is a flyswatter.