Hacker News Reader

Comment by jimmyl02 | Hacker News Reader

pretty new to the large scale data processing space so not sure if this is a known question but isn't the purpose of spark that it can be distributed across many workers and parallelized?

I guess the scale of data here ~100GB is manageable with something like DuckDB but once data gets past a certain scale, wouldn't single machine performance have no way of matching a distributed spark cluster?

View on HN

Yes, but such situations are incredibly rare IMO. I was just recently in a job where a self described expert was obsessed with writing all data pipelines (in this context, 1-2GB of data) in spark, because that was the “proper” way to do things since we had Databricks. It was beyond a waste of time.

Yup, everyone thought they had “big data” for a moment in the industry, and it turns out, they all just read the same trendy blog posts. I made the mistake of “clustering” at my previous company, and it introduced so much unnecessary complexity. If you are really worried about future proofing, just design a schema where the main entities are denormalized and use UUID for your primary keys. If you need to shard later, it will be much easier. Although, again, it’s likely that you will never need to.

> Yes, but such situations are incredibly rare IMO.

They happen all the time if you work for banks, large finance companies or government.

It’s not just the 2TB databases - it’s the 100 analysts all doing it their own thing with that data at the same time.

Maybe "rare" was the wrong choice of word. Maybe I should have said such situations are "highly contextual". It's one of those things where if you have to ask the question "Do I need this?" you almost certainly don't. And even in situations where you actually do need it, it's strongly preferred to "find a way not to need it" for day to day things (aka downsample, or be judicious about what you load into the analytics environment). I can only speak to my experience in FAANG, but even though we had the infra and budget to run huge distributed queries, it was highly discouraged unless absolutely needed!

This is my problem with databricks. It seems like in the course of selling their product they have taken the received wisdom of "do not run expensive and complex compute clusters unless absolutely necessary" and turned it into "it's fun and easy to run distributed compute clusters - everyone's doing it and you should too" regardless of how contextually appropriate it is.

My impression is that companies like databricks because of the framework it offers for organizing data workflows. Spark is then used since it is how databricks does its processing.

Microsoft’s synapse is their «ripoff product» which tries to compete directly by offering the same thing (only worse) with MS branding and better azure integration.

I’ve yet to see Spark being used outside of these products but would be happy to hear of such use.

So many think they (their company) have “Big Data” but don’t.

My employer's product database is about 2TB currently and is still perfectly manageable with a single machine, and to be honest, it's not even all that optimized.

People make the mistake of thinking it's a single 100GB dataset.

When it's more common to be manipulating lots of smaller datasets together in a way where you need to have more than 100GB of disk space. And in this situation you really need Spark.

Can you elaborate? I had no problem with DuckDB and a few TB of data. But it depends on what exactly you do with it, of course.

few hundred TBs here - no issues :)

I think it’s quite rare that a dataset is too large to be processed on a single volume. EBS volumes max out at 64Tb

If you're using EBS for shuffle/source data you've already behind.

If you can afford it you should be using the host NVME drives or FSX and relying on Spark to handle outages. The difference in performance will be orders of magnitude different.

And in this case you won't have the ability to store 64TB. The average max is 2TB.

It's really the only reason to use Spark in the first place, because you're doing non-local level processing - when I was throwing 100TB at the problem it made more sense than most of the data science tasks I see it used for.

This magnitude of data fascinates me. Can you elaborate on how that much data came to be and what kind of processing was needed for all that data? And if not, maybe point to some podcast, blog that goes into the nitty gritty of those types of real big data challenges.

Click/traffic data for a top 100 website. We weren't doing a ton, but basic recommendation processing, search improvement, pattern matching in user behavior, etc

We normally still would only need to process say, the last 10 days of user data to get decent recommendations, but occasionally it would make sense for processes running over the entire dataset.

Also this isn't that large when you consider binary artifacts (say, healthcare imaging) being stored in a database, which pretty sure that's what a lot of electronic healthcare record systems do.

A random company I bumped into has a 40TB OLTP database to this effect.

Financial market data (especially FX data) can have thousands of ticks per second. It's not unheard of for data engineers in that space to sometimes handle 1TB per day.

It's not just disk capacity though. Disk bandwidth, CPU and memory capacity/bandwidth etc can also become bottlenecks on a single machine.

Reply on news.ycombinator.com