Hacker News Reader

That's absolutely true, but I think the author's argument is likely to be that the popularity of Spark and Hadoop in the first place is misguided, and in a world where people were paying attention to performance instead of novelty, we'd have something like Spark and something like Hadoop that was optimized for a single computer, not for scaling for the fun of scaling.

   scaling for the fun of scaling.

I think your arguments are misguided, for every compute bound task where Hadoop/Spark undeperform, 1000 other ETL type tasks where hadoop is indispensable. As a result any organization running a large hadoop cluster will already have underused compute capacity for free and a maintainance staff which already taking care of the cluster. Thus from an organizations perspective the time difference is not material especially for batch jobs, this is the reason why Presto and Spark have been so successful. They enabled underutilized hadoop clusters to be used for ML and data science while delivering reasonable performance for Zero cost.

> ”... Staff already taking care of the cluster.”

“The” cluster, singular ...

This is the catch. If you need security and compliance, you can’t today derive the benefits of re-use by other teams.

Given today’s distros, and assuming your threat model needs to account for insider threat, you need a different cluster for each data ownership grouping and data sensitivity level. For four teams with three levels of data, you’d need twelve completely independent clusters.

Unless, as noted above, you’ve done a ton of in-house multi-tenancy work to provide full stack security and compliance assurances and audits.

However you may split clusters by role/access. You will have I/O bound ETL/Interactive tasks as primary use case. Thus the investment in security will be constant whether you reuse the same infrastructure for compute/memory bound ML tasks or not.

I agree, but at what cost? 10% performance hit, sure, no problem, VM's take about that much from running on real hardware. but a 100% performance hit or worse... you must be crazy.

Also, downloading 1GB is no big deal, downloading 1TB is a big deal, so I think your example has more merit in the > 1TB range. i.e. things that don't fit on a laptop easily.

1Mb vs 1Gb vs 1Tb distinction is a function of information contained in data. If its PII even 1Mb is an orginizational headache from security/risk-management perspective.

Also most commodity compute nodes have allocation around 16 Gb. The big difference comes in 16~256 Gb range where having a single powerful node can make a huge difference.

Sure, but 'laptop' can be changed very easily into any container or VM box somewhere that is tied to security/etc but still on a single box. Again sizing here is sort of beside the point, the point is, if its 500% worse to use than just doing it on a single instance/compute node, you better be offering A LOT of magic awesomeness that makes the performance drain acceptable.

10% cost for the benefits that this stuff can give you, that is pretty easy to swallow. 100%-500% perf. cost is just a very, very, very hard to swallow pill.

I will note that his local computation could just as easily have been done on a single, fully managed/secured "cloud" node with much the same performance. I think it misses the point to say that not downloading the data is where the cost lies.

I agree but sadly we don't have any plug-in solutions that operate in similar manner. How do you allocate capacity? What if your node has hard upper limit? How do you ensure security e.g. secomp, jails etc. The moment data crosses the DB boundry you need significant investment in ensuring compliance.

The reason Spark ecosystem has been so popular is because it enables these types of computations without breaking the model.

This line of discussion worries me, in case others think your security assertions are true and could be lead astray.

Today’s OSS big data query execution environments are BnL neither secure nor compliant.

This is not a knock. They were simply not designed for those constraints. They were built for academic or single tenant use cases without separation of duties / control.

This also applies to commercial products such as Splunk.

There is tremendous investment going on the last couple years in raising the security and compliance bar. We’ve worked on this with the usual suspects.

But barring a handful of proprietary stacks (the big three CSPs, and a couple enterprise on prem bare metal distros) getting close, we are not there yet.

Trying to land with your security or risk teams or regulators that the CSP pulled it off will likely take you longer than provisioning a compliant laptop you’d keep locked up with two keys.

Unless you have several tens of man years invested in in-depth security wrapping these environments, or can choose a big three CSP w/o answering to anyone, today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations.

    today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations.

You are utterly wrong. Algorithms and computations (especially ML kind) are never sensitive, its the data which is always sensitive. And that ALREADY exists on the cluster.

If an Organization already has a Hadoop cluster containing data. You are suggesting that somehow having it downloaded to a secured laptop is better? Than say Spark running on top of the cluster? I think you are deeply mistaken. The cluster instances are already protected (if not you have a bigger problems). Also while an organization might not have Hadoop, they surely have an RDBMS, in which case the algorithms are even more useful.

The code can certainly be more sensitive than the data. Consider codified expertise applied to public data.

> However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc.

You are conflating a couple of things here.

Of course for real analyses, you want to run them on standardized environments with appropriate access controls.

But nothing says that "standardized environment" has to use a distributed database that, from all accounts, seems to harm more than help.

You could just have a RDBMS or even filesystem, which is appropriately backed up or replicated, on a single beefy VM or physical machine.

> The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.

Only if the tools used to provide that ability actually allow you to take advantage of it. If scaling up to 160 nodes doesn't allow you to be any faster than 1 single laptop, you probably could have spent a lot less money on just adding more RAM to a single beefy server than a 160 node cluster.

Auditing is a real argument; but is decently-written specialized code really that much less reproducible than Hadoop/SQL? (I'll accept "yes", but note that that also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing.)

    also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing.

This is essentially the reason why all organizations are adopting Spark, since it allows you to write imperative code on dataframes and build ML models.

No, "all organizations" are not adopting Spark. You do not need Spark to use dataframes. And distributed computing is terrible for machine learning.

Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way.

    No, "all organizations" are not adopting Spark.

All organizations which already have a Hadoop cluster.

    You do not need Spark to use dataframes.

Never claimed this. To clarify Spark allows you to directly port Pandas code while leveraging existing Hadoop cluster infrastructure.

    And distributed computing is terrible for machine learning.

Distributed computing (Both traditional hadoop/spark and latest TF/PyTorch with parameter server) are essential for scaling ML beyond a certain point.

    Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way.

If you have experience routinely training models on Terabytes of data intended for production deployment. I am happy to hear. There is a vast difference between training a model on your machine for research and building a reliable ML system that scales across large datasets and teams while taking infrastructure costs into account.

"All organizations which already have a Hadoop cluster are adopting Spark" is not a very interesting claim.

Here's how I would leverage Hadoop infrastructure to use Pandas: delete Hadoop so I've got more disk space to run Pandas.

I don't get to train ML on terabytes of data very often. I do NLP, so "terabytes" means training a background model on the entire Common Crawl. Usually I'm doing something more specific and interesting than learning about random web pages. But when I do deal with the Common Crawl, I deal with it on one computer. Terabytes are not scary.

How does distributed computing even help? ML models need memory locality, sometimes to the extreme of being localized within a GPU's memory. And the limiting factor is the ability to iterate over the data. Sending the data over a network during training would be the worst thing you can do there.

    delete Hadoop so I've got more disk space to run Pandas.

Except when you have ~2000 node cluster that runs 10,000 ETL tasks daily all of which are IO bound, And you cannot "just" uninstall. However during certain periods the same cluster has significant underutilization this opens up possibilities of doing lots of cool stuff for almost zero cost.

I can understand your confusion. Training embedding model on Common Crawl is a toy problem. I recommend you thinl from perspective of a Tech company ideally in a production ML setting to understand the cost tradeoffs that go into making these decisions. Regarding memory locality if the problem is small enough its possible to tune Spark to use fewer or even just one worker with enough amount of memory allocated.

    Sending the data over a network during training would be the worst thing you can do there.

When your data itself is in 100s of Terabytes and sharded across multiple racks, often a properly tuned spark pipeline is more reliable and performs well. Again there is a difference between using Common Crawl subset that you manually download filter train etc and in ensuring that models are updated/trained automatically daily across 100s of TB data.

Well, thanks for telling me what I do, that was very informative, except for the fact that it's all nonsense.

You are posturing.

"A factor of 10x isn't a horrible tax to pay for re-using re-usable infrastructure, like your database. A factor of 60x-130x is more debatable, but I think the CIDR paper is totally worth talking about. I'm a bit disappointed that there aren't reasonable baselines, but not as much as that the SIGMOD paper doesn't have any baselines at all."

His "smarter algorithms" aren't.