Hacker News Reader

Didn't see the video, but what platform did you spin up the cluster on?

On any public cloud I have a hard time seeing this make financial sense other than Google, which bills by the minute.

EMR = Aws

We're also looking at making this a bit easier with some API changes (additions, really) and work with other OSS projects, like Apache Airflow. Streaming is a very interesting use case; a focus for us as well. Keeping clusters running for months/years presents some interesting challenges.

Disclaimer - I'm the Cloud Dataproc PM. :)

Same here. Built a pooling algorithm so we can have a maximum number of pools for each job type and distribute more jobs onto them when the maximum is reached. Clusters shut down when idle. It's served us well and saved loads of time and money.

Cool! EMR takes minutes, though, to spin up a cluster, and the billing seems to be different and less convenient than GCP.