Hacker News Reader

A company I worked for defined big data as anything not analyzable with a relational database - a PDF or video was big data. Companies say they want the latest buzzword even though it’s against their own interest and other companies say “we can give that to you” and instead deliver something that is of actual value so the license gets renewed. Same thing happened with SOA, REST, microservices, blockchain. IBM developed a blockchain product just to get the sales conversations going and then sold something else. Imagine being on that product’s team.

I would go with "big data is when you have to spend more effort on the mechanism of performing the computation than on the details of the analysis itself."

I think it's kind of pointless to fight for an exact meaning for these buzzwords. "Big Data", "Webscale", "Web 2.0", "NoSQL" and whatever other trendy hashtag du jour have lost most of their meaning because they hardly had any to begin with. They're marking drivel, not technical terms. And since everybody wants to look cool we all find ways to say that we're doing it, one way or the other.

Other than “webscale” I think the other terms still mean something. They don’t need to have a precise, agreed upon meaning to be useful words.

“Webscale” is just a buzzword, though.

Sorry I can't resist: https://www.youtube.com/watch?v=b2F-DItXtZs

This video is one of the best critiques of modern software development, and I suppose IT in general, that I've ever seen. Even if I've seen it a hundred times, I always appreciate seeing it again. Thank you!

I have suspicion that a lot of people who jumped to NoSQL just didn’t like the aesthetics of SQL syntax, in the same sense as people would not be ecstatic about Visual Basic.

NoSQL was jargon speak for document store. It's really schema on read vs the traditional database where schema is defined on (or before) write. And there are plenty of ways to use sql on various data formats including json, presto being one.

Around the time of peak NoSQL buzz, I went to Oracle Open world in the early-mid 2010's and there was an army of MongoDB folks outside the convention center grounds holding signs and handing out swag. NoSQL was definitely a good marketing play to try and take customers away from Oracle.

Mongo can do some amazing things. Things that it absolutely should not be.

Company; Document store? Let’s store billions of PDFs in it!

My soul dies each day I deal with that system.

Friend in one of the big data vendors: "we mostly work with medium data"

The way I always think of it: Medium data fits on disk but not in memory. Big data doesn't fit in either.

I like this definition too; the advantage is it scales with advances in technology. What has medium a decade ago, might be considered small (ish) now.

But people underestimate how much memory you can put in a server, and for how long it stays cost effective to do so.

Excel and business and computer science gets weird.

I’ve seen companies spending half a million to build a report that excel and a pivot table could do in an a day.

Few weeks ago company was looking at scaling up to max possible EC2 nodes in order to get something to work. I spent a couple hours tweaking the algorithm and now it’s on a micro node running 20x faster.

Some data problems are truly hard and interesting and world changing.

Excel covers almost everything else.

Kudos for that clarification. Years ago we did a bit of consultancy at a fashion company who drove their biannual fashion releases via data held in Excel, and their pain point was that even scrolling through the rows was agonisingly slow. Clearly all those people needed was either better desktop machines and/or an RDBMS, not an entirely new technology !

Many Years I had to drive an hour to help someone that had a corrupt CSV file. Column needed to made a little wider so it show the photo number properly. The person had a masters in CS.

I work for an average sized Danish city. Even when we combined our datasets with 10 other average sized cities and accumulate the biggest national data set and let the big data wizards have a go at it, it wasn’t really large enough to amount to anything useful.

Don’t get me wrong, Watson came up with a lot of BI suggestions that were useful and some of the scientists came up with semi-interesting prediction models. The thing is though, our analytics team has much better BI models and the our finance department does the prediction much more efficient and, well, legal. The prediction could become a useful aid, if it ever became legal, but not really at the license fees we were looking at.

Not sure if automated BI is ever going to become good enough. It’s impressive that Watson can come up with stuff an university post-bachelor intern can, but it’s license is more than our entire analytics team, so yeah...

I work in water management in the Netherlands and it feels the same.

A human domain expert knows what kind of thing to look for. Custom tools can help to get an overview (eg we visualize water velocities of a whole regional system on a map), but generic machine learning can't add much even though we record basically everything everywhere at five minute intervals.

The value of ML is perhaps in things that give little value in the individual case but can be used a huge number of times, like image classification.

Human directed machine learning then? Presumably a domain expert who can also expertly leverage ML/AI could make some useful tools?

There are no such people though.

I can give one example where the ML/AI toolkit is useful, even if only the largest providers in the world may need to work with real Big Data[tm] volumes. Fraud detection. And you don't need to be a particularly big operator to get sufficiently large & varied data sets.

In a way I think it helps that fraud is an ever-evolving, fiercely adversarial domain. New avenues are explored all the time. Occasionally old tricks are revived for a while, because they may work in the margins but become distinguishing features as soon as they see more use.

And even if your ML is based on nothing more than a random forest, you can still get surprisingly useful feature combinations out of it. (Or as our data scientists said: individually meaningless features may become a valuable signal when enough of them occur at the same time.)

Since the new breed of "data scientists" using R and Python dataframes replaced Excel "data analysts" they realized the "big" of big data hasn't disappeared and started hiring data engineers for developing and explaining the new big data platforms for stuff which didn't fit into their pandas dataframes.

Still, some call it medium data since this spans the range of higher gigabytes to lower terabytes. There are still couple of business problems in the petabyte scale, but they are probably so niche that they deserve a custom solution anyway.

Luckily that blog post is out of date and there are a lot of options now to handle datasets larger than 5TB without needing Hadoop.

Awk, for example.

Thanks for that link. Interesting that it was written a year-or-so after the phrase became popular.