Hacker News Reader

Very nice. I prefer using the built in command line ec2 scripts packages with all Spark versions.

https://spark.apache.org/docs/latest/ec2-scripts.html

You can even specify the spot instance price, instance type, number of nodes, etc.

e.g.

./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --spot-price=0.2 --instance-type=m4.4xlarge launch my-spark-cluster

View on HN

Indeed, the script works quite well for Spark - but we also wanted to provide a guide for those who would like to further understand the config files, especially for those who want to 'tinker' with these down the road (e.g. adding nodes to the cluster). Also, we find it helpful for learning to set up Spark on other platforms, and setting up other systems that don't have ec2 scripts yet.

Reply on news.ycombinator.com