Hacker News Reader

Comment by supergirl | Hacker News Reader

Yeah but can you store same amount of data on one ssd as on 10 disks?

No, but you also won't be using all that data at the same time. On Hadoop (and Spark on HDFS) clusters you'll find that most disks are either not that big or are heavily under-used.

Our HBase cluster had 6 * 2Tb disks: about 8.5Tb of usable storage (the other 3.5Tb accounts for data replication/duplication) per host in the cluster. However, you need about 200 bytes in memory per kb on disk and should assign only 32Gb of heap to HBase. That's 2.5Tb wasted, per host. Couldn't just plug those disks out and use them somewhere else: you need all the disks in parallel to overcome the IO/bandwith bottleneck.

This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".

Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else

100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.

Depends on how you're setting it up.

  aws s3 cp --recursive bunch-o-data s3://some-bucket/
  spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster

Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.

As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1].

[0] http://spark.apache.org/releases/spark-release-2-0-0.html#re...

[1] https://github.com/amplab/spark-ec2

I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.

Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...

True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.

Reply on news.ycombinator.com