Apache Spark: A Unified Engine for Big Data Processing | Hacker News Reader

Apache Spark: A Unified Engine for Big Data Processing

cacm.acm.org

162 pointsakashtndn34 comments

The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model.

Much nicer IMHO

https://databricks.com/blog/2016/03/03/introducing-graphfram...

GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.

I'm curious just how beneficial a giant all inclusive framework is for this. My gut is it will be a huge boon for most of us, but you will still find very valid use cases for rolling your own.

That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.

I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs.

I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on.

I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface.

However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.

Do you have a write-up of your experience by any chance?

We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.

We don't do a lot of ML processing..so my experience is very limited.

Do you see yourself getting better performance with the Dataframes+ Pipelines infra...or is it simply a better mental model.

Reynold from Databricks and the Apache Spark project here.

Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?

Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.

Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able).

Can you talk about these limitations and your experience?

https://issues.apache.org/jira/browse/SPARK-139

https://issues.apache.org/jira/browse/SPARK-1476

https://issues.apache.org/jira/browse/SPARK-6235

You'll hit this bug when your model size is larger than 2GB.

BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that.

https://issues.apache.org/jira/browse/SPARK-732

https://issues.apache.org/jira/browse/SPARK-5490

If you follow the Spark community closely, you won't find any new information here. However, as a software consultant, I find this to be a useful nontechnical overview to send to partners and clients.

> The RDD programming model provides only distributed collections of object

It is ok for a non-technical high level overview but RDD should probably be referred to as a data model (how data is represented and managed) while map-reduce is a programming model (how data is being processed).

No, the rdd interface has methods on it that directly relate to how data is processed.

You are right, but what I meant is that delta between Spark and Hadoop is RDD (RDD - Resilient Distributed Dataset) which is data structure first of all. Thus Spark and Hadoop have the same data processing model implemented over different data representation models (and hence the performance gain).

> delta

why can't we use the word difference? delta, mathematically, suggest some space and measure.

Question for those of you who use Spark: Is it possible to use with Go, Rust, Nim or C++?

I'm not a fan of those other languages, and I'm also trying to reduce the amount of context switching these days.

It would be possible in principle (especially if you can write a Python module bridge that can wrap your C++ / Rust / etc. code), or something that you can load in Java. But, all of the RDD and SparkContext APIs are going to be available only in Java / Scala / Python without a significant amount of work. It's probably not worth it to use another language and wrap it in that way.

I would recommend using Scala (for the static typing) or Python (numpy and other libraries are very useful with Spark). They're not hard to pick up, especially when using pyspark + ipython for prototyping locally.

Funky: 10 to 20 local disks, for approximately 1GB/s to 2GB/s of disk bandwidth

The new MacBook Pros apparently have north of 2GB/s of disk bandwidth for their internal SSD.

https://9to5mac.com/2016/11/01/the-late-2016-entry-level-13-...

You can pick up a wide variety of PCIe SSDs with that kind of bandwidth as well.

It's worth keeping in mind, because it means that a whole lot of workloads that people think they need a cluster for end up being far faster and cheaper to put on a single server these days, given the cost and overheads of sufficiently high speed network interconnects to move the data around at a fast enough to compete.

Of course there are plenty of workloads where you still need a cluster, but people often don't realise just how much you can put into a single commodity server today at a reasonable cost, nor what kind of overheads going from a single server to a cluster tends to add.

If you could fit on one box, would you be better off using a sql database (assuming you knew you didn't need to scale beyond one box), or is a single node spark box faster?

Or does it depend on the scenario?

Spark and a SQL database have different use cases. Spark is a n analytics engine to do processing on data. SQL databases are for storage and retrieval of data(they just so happen to be able to do various analytical type stuff).

Depends on where your bottleneck is.

If its disk, maybe a one box approach is suitable.

If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help.

Additionally a cluster can manage a larger working data set in RAM than a single box can do.

> If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help.

If you're going to partition the data because of limitations like that, it may still be faster to run a "cluster" on a single server, depending on data volume.

> Additionally a cluster can manage a larger working data set in RAM than a single box can do.

That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm sure it happens, but it's not common, and it's fairly unusual for people to even approach the point where adding ore RAM starts getting more expensive than adding more boxes.

You can currently fit at least 8TB of RAM in the biggest x86 boxes; possibly more by now, and boxes that can fit 1TB-2TB are still relatively cheap.

Unless you plan to exceed at least 1TB, it's likely going to be cheaper to stuff more memory in a box than add enough extra boxes to compensate for the communications overhead.

Of course there are plenty of datasets where you would exceed 1TB for your dataset, but most people don't get anywhere close to that.

I would say that before a hardware cluster (as opposed to dirty "hacks" like sharding on a single box) makes sense if you are buying new hardware (the maths of course looks different if one option is to make do with existing servers), you will need to have requirements that exceeds one or more of:

- 48-64 x86 cores.

- 1TB-2TB of RAM.

- 2GB/sec aggregate disk bandwidth.

For all of these, it's worth noting that you can't compare like for like, as the moment you go to a cluster you have the according overheads in both CPU, RAM and disk IO of having to spread the work out, write data to disk more places, duplicate data in memory etc..

Note also that you can do better than that on plain x86 hardware, but the above is the point where the incremental cost of increases starts to really hurt and eat into any savings vs. a cluster.

> That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers.

I'm currently experimenting with a 4 node cluster running Apache Drill (on top of MapR's Hadoop distro), with 500GB RAM, 96 cores, and 28TB of SSD. For some analytic queries that span multiple date partitions, its actually easy to see 70% RAM capacity being utilized, and 80% of the CPU. The size of the data set is currently about 7TB, but that replicates for both data locality and redundancy.

We are a Postgres shop, and have tested with some of the options that run a "single node cluster" but one of the limiting factors is that RAM is divided between those DB processes, and performance is impacted. Plus we then need to implement some form of replication to obtain redundancy, fault tolerance, or HA.

Quite simply we are not using a cluster because our data won't fit, but because we can get better performance and reliability on 4 nodes. We likely won't have the need for 400.

There are cases where for non-interactive data processing we definitely don't need a cluster. At its simplest, multiple copies of Awk scripts can be run using GNU Parallel, Python scripts can be parallelized with the Multiprocessing module, and Go has been useful for concurrency (especially to us Python programmers).

Yeah but can you store same amount of data on one ssd as on 10 disks?

This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".

Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else

100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.

Depends on how you're setting it up.

  aws s3 cp --recursive bunch-o-data s3://some-bucket/
  spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster

Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.

As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1].

[0] http://spark.apache.org/releases/spark-release-2-0-0.html#re...

[1] https://github.com/amplab/spark-ec2

I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.

Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...

True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.

No, but you also won't be using all that data at the same time. On Hadoop (and Spark on HDFS) clusters you'll find that most disks are either not that big or are heavily under-used.

Our HBase cluster had 6 * 2Tb disks: about 8.5Tb of usable storage (the other 3.5Tb accounts for data replication/duplication) per host in the cluster. However, you need about 200 bytes in memory per kb on disk and should assign only 32Gb of heap to HBase. That's 2.5Tb wasted, per host. Couldn't just plug those disks out and use them somewhere else: you need all the disks in parallel to overcome the IO/bandwith bottleneck.

Reply on news.ycombinator.com