Hacker News Reader

Comment by alexholehouse | Hacker News Reader

Slightly off topic, but at what point does "big data" just become data?

According to wikipedia (which is in line with how I think about big data) "Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. "

However, as "big data" becomes more mainstream and more tools/services exist to accommodate data at the peta/exobyte scale does that whole definition stop being relevant for kinds of data we're talking about?

View on HN

Working in the risk analytics space, "Big Data" seems to be marketing speak for "you can dump all sorts of loosely structured data into this big bucket and our tools will help you find meaningful trends in it." I've yet to see an installation approaching anything near 300GB, so I think of big data as the new sexier label to put on ad hoc data mining applications.

But if it is for running ad hoc data mining on relatively small amounts of data, wouldn't a traditional SQL database do the job just as well? Especially with with features like the JSON support in PostgreSQL.

Yes, you are perfectly correct. But a lot of people are paid a lot of money for reinventing the wheel every few years, so expect your words to fall on deaf ears.

I think the question is how much effort will it take to get all kinds of data from all kinds of sources into a nice, coherent schema.

The "schema-less" aspects of NoSQL data stores is a big part of the marketing appeal, I think.

(Maybe the JSON support in PostgreSQL makes this moot, but that's the thinking.)

You always have a schema. The only question is if you know what it is, or not.

In the next few years there will be a lot of money to be made getting data out of these stores and back into relational databases.

PostgreSQL has terrible sharding and clustering support.

So no I would not be using it for any big data projects.

I was going by mcphilip's description which was <300GB. For running data mining on so small datasets you do not need any sharding.

Such general statements can only be inaccurate.

We're having about 3TB of PostgreSQL data on 4 servers. It works very well for us.

> "Big Data" seems to be marketing speak for "you can dump all sorts of loosely structured data into this big bucket and our tools will help you find meaningful trends in it."

That's the description of a data warehouse.

It's just a buzzword now. "Let's use cloud to do big data analysis to proactively synergise our core competencies".

you forgot "disrupt"

One rule of thumb I've had for people is, "if you can fit it in one computer, it is definitely not big data".

My company actually bought some "big data"-machinery plus consulting and they actually delivered their hardware in a Very Large case. Truly one of those '90s dual big towers. I thought that was pretty funny, considering they put less data in it than fits on a $200 hard drive.

I actually first heard the term used in what has turned out be to be a very unusual sense. "Big Data" is like "Big Oil" - it's a term for industries that get their economic clout from the value of their data. Google, for instance, is "Big Data" because it's able to offer the advertising platforms and spam filtering services that it does because of the data it's extracting value from.

CmonDev put it nicely. Buzzword. Same when you hear, most of the time, how someone is leveraging Hadoop for whatever.

There are various definitions of what is big data. I like to think of it as whenever the only way to deal with your data is by doing full table scans, then we're in same domain of thinking of big data.

I guess so and 16 MB was certainely considered too large and complex to process it with traditionel data processing applications a few decades ago.

Reply on news.ycombinator.com