You don't have big data
blog.mongohq.com
blog.mongohq.com
So in other words, “Big Data” is what used to be known, somewhat less sexily, as “data warehousing”?
I think that the two are different, though. The difference: scale. Data warehousing solutions simply can't scale to "big data". Many companies have a "data warehouse" that is usually a single (albeit very beefy) server running SQL Server. But "big data" is the kind of data that can't be supported by a single server or by a traditional database. It's the difference between warehousing activity in a network of a 10k employees vs. warehousing activity on a website with a million daily uniques.
That's my imrpession, anwyay -- "data warehousing" is dominated by the RDBMS, but "big data" is dominated by highly scalable, distributed databases.
I think you're more along the proper direction in the single server vs N server. Every data warehouse I've worked on in the past decade or so has been multi server.
Big data is about finding the most obscure statistical correlations between phenomena with obvious relevance and seeming irrelevance. It's the mythical relationship between Wal-Mart, strawberry poptarts, and hurricanes
Now, technologies that can do much more are becoming more accessible. Graph, time series, path analysis, fraud detection, customer churn analysis, statistical methods, etc.
Disclaimer: I work for a "big data" company that is also a data warehousing company.
'Data' data (as in your company's data warehouse) is primarily internal data. By this I mean: data generated primarily by your source systems or associated business processes.
'Big' data is external data. It is tweets that express opinion about your brand. Weather patterns that might influence supply logistics. Demographics of markets you wish to penetrate.
I think there are a great number of slightly different interpretations out there but pragmatically speaking, what I have suggested here makes sense.
I see what you're driving at and I don't disagree but I bet both Ebay and Facebook have a bit of data in a ruddy old fact/dimension warehouse and some neat dashboards and reports over the top of it.
In which case I'd stick with my definition and simply state that in these examples, they have both types of data.
I just realised I know zero about the infrastructure of either of those businesses so I have some reading to do... oh, and everything I said about them might be crap.
My experience is the opposite: big data tends to be a lot of rapid streaming data, near-real-time, with inputs from many sources and sensors. This overwhelms traditional databases.
For us, big data means we want stream filtering, heuristic sampling, map reduce, and the like.
If you need to store the data (Twitter-like data) then storage becomes your primary concern. Analysis can be done later and in less-than-real time. But if your data are coming in so fast that keeping up is the problem, then storage isn't even a consideration (you generally just don't) and analysis becomes the challenging part, you need to aggregate do be able to update your metrics on the fly.
So they really aren't even overlapping problems outside of the fact that they both deal with lots of data.
I guess my response would be... who cares?
While the techniques and infrastructure for high volume can be different from those for high velocity (near-time / real-time analytics is different than the more batch-based whole hog analytics associated with high volumes), they are often related. Similarly, there is often commonalities in dealing with the various Vs which may just be of different scale.
Would they argue that one shouldn't be looking at using chef/ansible/salt, etc. as long as you're dealing with a handful of machines? If so, I would think most would argue that laying a good baseline so that when growth occurs is a good thing to do. From a "big data" perspective would they then argue that one doesn't need an ETL/ELT pipeline process? That one shouldn't think about how to deal with the high volume? That would seem to ... less than optimal.
"MongoDB can't handle Big Data so we are going to redefine Big Data for our own convenience as cheerleaders of MongoDB and then assert that Big Data, per our new definition, is really not that important."
The blog post makes quite a few dubious assertions, apparently for the sole purpose of justifying MongoDB's inadequacies as a large-scale data platform.
So I don't think your translation is correct. I actually think it's pretty far off. We're probably some of the most cynical MongoDB users you'll meet.
(disclosure: I'm one of the founders of MongoHQ)
I get the argument that customers should not over-engineer their database systems but in other databases a lot of that "over-engineering" comes almost for free in terms of user effort.
Also, for many types of analytics, there really isn't a concept of "cold" data. A single query should be able to access data inserted milliseconds ago and data inserted a month ago as though it were in the same table. A lot of "real-time" analytics work this way. This does not need to be done purely in-memory if the storage engine is designed well. The old OLTP/OLAP dichotomy of the 1990s has been slowly fading for a long time.
Some DBs handle this by requiring compromise at the very beginning, which makes sense when you're going to have a huge amount of data from the get go.
There's no DB on the planet that lets you go up into the TB range without having to make some compromises (either up front or down the road).
For things like joins, query selectivity on multiple columns, etc not so much. You don't even need secondary indexing or denormalization to do things like graph analysis or polygon searches on a table (in the same query even) at scale. All of the access method related operations can scale very efficiently to massively parallel systems if you use the appropriate data structures and algorithms.
MongoDB uses an approximation of the correct algorithms for gigabyte scale systems. Those algorithms are just wildly inappropriate when you start talking about terabyte scale systems. I have no investment in MongoDB negative or positive, but like all databases it is going to be lousy outside of the implicit scope supported by the design and architecture. In the specific case of MongoDB, and as someone that has designed their share of database engines, the internals are not designed to support non-small databases to any significant extent.
And honestly, a terabyte is a pretty trivial database these days. That is the kind of thing you run on a single server with ease. Smoothly scaling that to dozens of nodes as though it was a single system is something you can buy. I really don't understand the assertion that scaling to 10TB is difficult or requires anything different than scaling to 10GB. That is demonstrably untrue.
Personally, I think there are some holes in the story they are telling in general, but this is not one of them. The question I have about their product is: why not just store the JSON in a SQL database, many of which are adding JSON as a native type? Did they really need to reinvent an entire database system? Why not just take postgres and build on that like everyone else does?
In other words: the less costly a useful task becomes, the more we tend to do it "just because".
Seems like the author is trying to say: storing a lot of data vertically "just because" does not big data make. But the author does not fully explain why this is significant.
It is significant, though, because Jevons Paradox would predict that eventually companies will want to access so-called "frozen" data more frequently and will want to make that frequent access as cost-effective (and payload-efficient) as possible, which distributed NoSQL-type DBs do very well.
Granted, this was the 1980s when mainframe / centralized computing was big (client-server was still a few years off), and the desktop revolution was just getting underway.
Also his usage of childish memes makes me unable to take it seriously.
The solutions are obvious - store it compressed https://jira.mongodb.org/browse/SERVER-164 - and tokenize repeated values like field names https://jira.mongodb.org/browse/SERVER-863 - note how many years those have been open without progress!
Not that it really matters - mongodb's single per database lock means you'll have trouble inserting, updating or deleting data quickly enough for web scale!
I think it is useful to have some rules of thumb as to when you need to apply more exotic techniques and tools vs. something where simple stuff works. So in that sense asking do you have "big data" or not can be useful...
Around which dataset sizes are which methods appropriate?
Really, why don't we put down the pitchforks and call "big data" what it really is: data mining. You can perform data mining on small sets and large sets. Nearly infinitely large sets, given the tools to manage that data. "Big data" is just a buzzword that doesn't mean anything more than data mining (which includes machine learning, AI methods, etc) on a really large data set.
No, you probably do not necessarily need a Hadoop cluster to work with a 120GB file. With Python and Pandas you could probably run through it on a budget laptop. But data will always be relative to your companies size and current know-how. In the most banal way: Data can be big data, because some manager can't open it in Excel.
Just because you have a lot of data and you don't know how to put it together doesn't mean you have Big Data. More often than not (in my experience), it means you have useless data. Take that money and hire someone who actually understands the right metrics to capture.
"Building a Black Swan: Disrupting NetApp, EMC, Amazon’s S3 … maybe all BigData": https://medium.com/money-banking/b8427c23bf0f
https://news.ycombinator.com/item?id=6028334
Also worth skimming is the blog post introducing Manta:
"Hello, Manta: Bringing Unix to Big Data": http://www.joyent.com/blog/hello-manta-bringing-unix-to-big-...
And, (more closely) related to the "you don't have big data", I found this little post about Manta also interesting: http://building.wanelo.com/post/54110156963/a-cost-effective...
(as an example of how keeping things simple (here: simple log files) can combine to provide the scalability needed to collect "BigData" -- and then the Manta architecture can help with turning that data into information)
(No affiliation with Joyent/Manta -- but it does seem like they have a pretty good product concept/idea -- and even if not using Manta or Joyent -- building on their ideas and architecture -- seems like something that might be useful also for smaller installations).
One could even go further back to the 16th century (and earlier if you're really adventurous) where the ideas of life statistics applying to groups (but not individuals) was first explained.
In short, 1990 is sort of an arbitrary date and does not accurately reflect the origin of Big Data. We have wanted to record the sum total of humanity for as long as we have had the ability to record things; it's simply becoming more reasonable to attempt today.
[0]http://www.officemuseum.com/1907_Actuarial_Division_Metropol...
We can look at size and we can look at rate (read or write).
One size classification is: data we can store in memory vs. on drive. I wouldn't refer to anything <64GB as "big" data. If you can store it on one machine it probably shouldn't be considered big from a size perspective (so let's say 4TB as some sort of threshold).
More generally, if you need more than a single server to store or do real-time processing of your data I'd say it qualifies as big. Otherwise probably not.
Another factor a lot of people often don't look at is efficiency. Yes, if everything is a JSON string with a lot of static annotations we can make a small amount of data look very big. You need to look at a more information theoretic "size". This often applies to processing... Yes, you can spend many CPU cycles parsing incoming HTTP requests but that doesn't mean the data is inherently big.
In my view there has always been "big" data. As others have pointed out, none of this is really any different from the data warehousing days. How much data you can store is a function of how much money you are willing to spend, and all that has changed is the amount of disk storage you get per dollar.
I would propose a definition as follows:
S_t > N * H_t
Where S_t is "big" data when it is greater than some constant # of hard drives N and H is the size of the average consumer HDD at time t.
So if we assume that today the average HDD is 2TB and we define C as say 200, big data is 400TB. 10 years ago "big data" would have been say 20TB. Simple.
Maybe some other letters would make the mnemonic more interesting :)
Til you max out a box with those specs, you don't need big data.
We're handling a very modest small data set (1,000 messages/sec) with just a few people. That's thanks to the awesome tools coming out of this fruitful (and overhyped) industry.
If you want to refer to real big data, perhaps just switch to the term "50PB of data" or whatever order of magnitude it is.