10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.
It makes me want to cry, knowing we handled that with a single server and a relational database 10 years ago.
Lets also not forget that everyone today forgets that the majority of data actually has some sort of structure. There is no point in pretending that every piece of data is a BLOB or a JSON document.
I have given up on our industry ever becoming sane, I now fully expect each hype cycle to be pushed to the absolute maximum. Only to be replaced by the next buzzword cycle when the current starts failing to deliver on promises.
I sometimes wish I was unscrupulous enough to cash in on these trends, I’d be a millionaire now. Instead I’m just a sucker who tries to make solid engineering decisions on behalf of my employers and clients. It’s depressing to think what professionalism has cost me in cash terms. But you’ve got to be able to look at yourself in the mirror.
"Resume driven development" is a great term.
BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.
What about too large for Excel, is that “big data” too?
(Neither are)
This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.