Hacker News Reader

Comment by leoc | Hacker News Reader

> “Big Data” data tends to be cold data, that is, data that you aren’t actively accessing and, apart from analyzing it, probably never will. In fact, apart from analysis, it could be regarded as frozen. It may be fed with fresh rapidly cooling records and the cooling records analyzed for up-to-date analysis, but the “Big Data” pool should be at least conceptually separated from the live data; mingling the two’s requirements can easily end up in an unsatisfactory lowest-common-capability situation where neither is optimal.

So in other words, “Big Data” is what used to be known, somewhat less sexily, as “data warehousing”?

View on HN

I think of data warehousing as being concerned with what the article might call "formerly hot" data. That is to say that it deals with data that was used in critical business functions at previous points in time, e.g. a grocery store's orders for potato chips in the third week of November, 2003. This is in contrast to big data such as the number of times the store's automatic door opener operated during the same period.

Big data is about finding the most obscure statistical correlations between phenomena with obvious relevance and seeming irrelevance. It's the mythical relationship between Wal-Mart, strawberry poptarts, and hurricanes

http://www.hurricaneville.com/pop_tarts.html

So the author is suggesting.

I think that the two are different, though. The difference: scale. Data warehousing solutions simply can't scale to "big data". Many companies have a "data warehouse" that is usually a single (albeit very beefy) server running SQL Server. But "big data" is the kind of data that can't be supported by a single server or by a traditional database. It's the difference between warehousing activity in a network of a 10k employees vs. warehousing activity on a website with a million daily uniques.

That's my imrpession, anwyay -- "data warehousing" is dominated by the RDBMS, but "big data" is dominated by highly scalable, distributed databases.

Most people who use the term "big data" are wannabe big data people, but will never have data too large to fit/process on a single server. In this use, it is really just OLTP vs OLAP all over again, which leads us naturally to ETL our data into a DW.

I'm going to somewhat disagree. Traditional data warehouses (teradata for example) can handle well into the petabytes. I think Ebay's main TD cluster was around 15 or 20 PB last I knew.

I think you're more along the proper direction in the single server vs N server. Every data warehouse I've worked on in the past decade or so has been multi server.

Well, I think Teradata probably falls under the big data umbrella, given that its basically a distributed database, right? I was thinking more about just plain Oracle or SQL Server or whathaveyou, which can scale pretty big, big enough for 99% of companies, but not "big", and are notoriously difficult to cluster/distribute.

Fair enough. I think the reason people struggle with classifying this stuff is because there are a lot of gray areas and it's difficult to verbalize all the details meaningfully. One could say "Oh...RDBMSes!"...but teradata is still relational...it's just distributed. "Oh! OLAP vs OLTP!"...but it's not really that either. One _could_ use teradata for a couple gig data mart just fine (and argue that it's not big data). That's why i tend to fall back to the 3 / 4 Vs that have tended to typify the 4 aspects of "big data" (volume, velocity, variety, veracity).

There is a qualitative difference in the way people are analyzing large amounts of data. Data Warehousing typically means fairly traditional typical SQL queries on vast amounts of data; a window function might be considered sophisticated.

Now, technologies that can do much more are becoming more accessible. Graph, time series, path analysis, fraud detection, customer churn analysis, statistical methods, etc.

Disclaimer: I work for a "big data" company that is also a data warehousing company.

As a data warehousing/BI specialist I'd like to share my interperetation.

'Data' data (as in your company's data warehouse) is primarily internal data. By this I mean: data generated primarily by your source systems or associated business processes.

'Big' data is external data. It is tweets that express opinion about your brand. Weather patterns that might influence supply logistics. Demographics of markets you wish to penetrate.

I think there are a great number of slightly different interpretations out there but pragmatically speaking, what I have suggested here makes sense.

I think that _can_ be a starting point; however, there are definitely cases where it's not the case. Take ebay or facebook for example. I'm pretty sure their "data" data is "big" data...would you concur? I think the external data increases the variety (naturally) as well as has some effect (depending on the sources) on the volume and velocity of said data....thusly pushing it over the edge from "data" data to "big" data potentially.

>would you concur?

I see what you're driving at and I don't disagree but I bet both Ebay and Facebook have a bit of data in a ruddy old fact/dimension warehouse and some neat dashboards and reports over the top of it.

In which case I'd stick with my definition and simply state that in these examples, they have both types of data.

I just realised I know zero about the infrastructure of either of those businesses so I have some reading to do... oh, and everything I said about them might be crap.

Precisely.

Reply on news.ycombinator.com