So my definition of big data was data so big it cannot be processed on a single machine in a reasonable amount of time.
I guess they’re using a different definition?
I guess they’re using a different definition?
very much so…
You have phones that are faster than cloud VMs of the past. You can use bare metal servers with up to 344 cores and 16TB of ram.
I used to share your definition too, but I now say that if it doesn’t open in Microsoft Excel, it’s big data.
As you say, single machines can scale up incredibly far. That just means 16 TB datasets no longer demand big data solutions.
Many people like to think they have big data, and you kinda have to agree with them if you want their money. At least in consulting.
Also you could go well beyond a 16TB dataset on a single machine. You assume that the whole uncompressed dataset has to fit in memory, but many workloads don’t need that.
How many people in the world have such big datasets to analyse within reasonable time?
Some people say extreme data.
Google has big data. You are not google.
Or one could define it as too big to fit on a single SSD/HDD, maybe >30TB. Still within the reach of a hobbyist, but too large to process in memory and needs special tools to work with. It doesn't have to be petabyte scale to need 'big data' tooling.
8TB is a couple hundred hours of 4k RAW video assets.