Hacker News Reader

Comment by mrweasel | Hacker News Reader

Most companies don't need a lot of things.

I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine.

Once you hit the scale where you have "enough" data I'd agree with many of the other comments: Managing open source or home grown solutions quickly become more expensive than just paying for a service. Not quite the same, but we considered deploying a few solutions on OpenStack, but once you paid for training and staff it turned out to be cheaper to just give VMWare more money.

View on HN

> We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine.

I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how they had big plans to use “all this data” to get an edge up on their competitors. They talked about the big data platforms and technologies they were evaluating and how they’d probably have to hire more people to handle it all.

When I pressed them for details on it, they eventually explained the data they had on each customer and the approximate number of customers. I did some quick math and calculated the total amount of data on all of their customers was under 100MB.

They had managed to blow it up to occupy a couple orders of magnitude more space in one situation (think storing time series data as JSON where every point had duplicated tags attached due to ineffective structure) but even that was small enough to fit entirely in RAM of a decent sized server.

Remember some industries are perversely incentivized to drive up costs because profits are limited as a percentage of total cost.

Could you please provide examples of that kind of industries?

Health insurance is the example that comes to mind.

You can buy a single server with TiBs of ram. That’s why I think motherduck/duckdb really shines. It’s the best solution for 99% of users whose data can be processed relatively quickly on a single machine.

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions.

I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard' solutions are for something at much bigger scale. Start with blah, spin that up on a k8s cluster, run this in docker, blah blah. FFS, I don't need that, I just want some nice graphs.

So I just shove it all into a postgres database (since we are already familiar with it) and visualize it with Grafana. I'm sure people would complain that it's not the proper way, but I prefer spending my time solving my users problems, not self-inflicted ones.

I feel this in my bones. I really find it hard to figure out the right solutions for scaling things that will be "bigger than small" but also "smaller than big" for the foreseeable future. That is, when the "starting point" conventional wisdom ("just use postgres / sqlite for everything; anything else is yagni") isn't quite up to snuff, but the "scale up" solutions are designed to handle far bigger use cases and bring some baggage along with that.

I have honestly found Snowflake pretty useful for this, because it seems pretty cheap to me for a small amount of usage, but is also elastic up to this middle-ground of usage. But I suspect that at "truly big" I would / will be interested in different solutions.

For the graphs I don't know, but I was happy to learn that Journald can actually ship logs to other Journald installations. Haven't used it yet, but I'm still looking of a place where it would make sense.

Fairly confident anyone outside of sales is going to tell you the solution that works is the best solution.

A Terabyte of data can be still vast, depending on how different and unstructured they are. If you have thousands of different logfiles-formats, datadump-formats and various complex fileformats, then you need a very different solution than you would need to handle a terabyte of uniform data in a csv-file or similar.

Spark wont inherently help you solve that problem though, and neither will most of the big data solutions.

Reply on news.ycombinator.com