Hacker News Reader

Comment by gsich | Hacker News Reader

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year.

The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. And yet so much energy goes into just getting the wrong tools to do more...

All of the source text for all of Google+ Communities posts is a few hundred GB. This for 8.1 million communities and ~10 million active users.

Add in images and the rest of the Web payload (800 KiB per page), and that swells to Petabyte range. But the actual scale of the user-entered text is stunningly small.

> The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools.

Rings way too true for me atm.

My current workplace is currently struggling, because one of our application stores something like a combined 300G of analytics data in the database with the application data. Modifying the table causes hours of downtime because everyone claims that backwards compatible db changes are too hard. And everyone is scared because with more users there's "so much more analytics data" incoming. Yes, with 300Gb across 3-4 years.

And I'm just wondering why it's not an option to just move all of that into one decently sized mysql/postgres instance. Give it SSDs, 30 - 60Gb of ram for the hot dataset (1-2 month) and it'll just solve our problems. But apparently, "that's too hard to do and takes too much time" without further reason.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.

If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.

True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well.

That’s not really ‘big data’ in my opinion.

That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.

I think the limit is more in the double digits TB range right now.

See? As I said this not easily definable. A single HDD is though. What you could do is a single RAM module.

Idk, it's fuzzy, obviously, and a moving goalpost, but in the end it comes down to this: do I buy/rent a bigger server? Or do I put more engineers to the task of replacing the naive algorithm. The latter is Big Data and usually the case when you run out of RAM, not out of disk. And the first one is an underrated option.

"Big Data is any thing which is crash Excel."

https://twitter.com/devops_borat/status/288698056470315008?l...

"EC2 High Memory instances offer 6 TB, 9 TB, and 12 TB of memory in an instance. "

https://aws.amazon.com/ec2/instance-types/high-memory/

You can still connect a fairly big NAS to a beefy server and do the processing, unless thoughput rate becomes an issue. Saturating a 10Gbit link means you can probably process up to 100TB a day.

Medium data. Substantial data. Just-enough-data.

My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.

This is pretty much where we are at, especially with a lot of transforms and processing on high-speed video captures.

The one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sleeve.

A while back on RDS we did our migrations by standing up a super fat instance, fail over, migrate, fail back and shutdown the fat instance. This was a great asset, and AWS made this easy.

Reply on news.ycombinator.com