Hacker News Reader

Comment by Tsarbomb | Hacker News Reader

Can someone explain to me what the big draw was for Hortonworks or Cloudera?

Working as a lead in a small team that deals with a colossal amount of data (human genomics), it was always easier for us to hand roll deployments with terraform/ansible in either baremetal or OpenStack environments.

In the public clouds like AWS we are using the managed services like EMR.

The whole sales pitch I've always got from either hortonworks or Cloudera always seemed more aimed at nontechnical stakeholders than the technical ones. Am I wrong? Have I missed out on some cool stuff?

View on HN

You’re not wrong. It would be hard to sell CDH if you only engage with technical stakeholders. I can imagine the pitch now...

“So what you get is really old versions of everything, plus some shaded jars that will break if your classloader ever changes load order. As far as cloud goes, we’ll give you this half-baked automation (Director) and we’ll also constrain you from using any features of your cloud provider (availability zones, load balancers, Amazon Linux 2). Finally, we require you to use Oracle’s JDK but we won’t distribute it so that you’re indemnified.”

But you have heard of Director :)

I have used both Ansible and and Ambari (from Hortonworks) to deploy and maintain our big data stack. They each have pros and cons

Ansible

Pros:

- More flexible

- Easier to update separate components

- Bring-your-own monitoring & alerting

Cons

- Higher initial time to setup playbooks

- Rolling update configuration is a pain

- Upgrading components require more compatibility testing

- Lack built-in monitoring & alerting

Ambari

Pros:

- Easier to setup from scratch (step-by-step wizard for everything)

- Changing configuration is easy, you change one components, and other components are update accordingly. It also notify you which software in which nodes need to be restarted and allow you to do rolling restart seamlessly.

- Components integrate well with each others, so less time is needed for compatibility testing.

- Back-ported of important patches .

- Built-in monitoring and alerting

Cons:

- Older software version, albeit with back-ported patches

- Cannot upgrade separated components

- Less easy to integrate with your existing monitoring infrastructure.

I also find it easier to train new team members to use an existing Ambari installation than to maintain Ansible playbooks. We are now using Ambari to maintain the more stable parts of our big data stacks (HDFS, YARN, etc), and use Ansible for the part which are still improving rapidly (Kafka, Flink, Presto, etc)

terraform: 2014 ansible: 2012 openstack: 2010 cloudera: 2008 hortonworks: 2011 emr: 2011

(those were quickly googled, apologies if i got any dates wrong)

timing is important. cloudera had good, early timing and looked promising because of it. you are right that EMR definitely hurt all the other hadoop vendors, though i think people overestimate how comfortable big enterprises are with moving to public cloud. way more comfortable today; back then everything was a lot less certain. cloudera's name is unfortunate given they never got anything cloud-based successful, maybe they'd be in a better place now if they had.

but some of the key technologies you suggest as better options came 4-6 years later. that's 4-6 years of providing value, gaining traction, and building a committed customer base. 4-6 years is a long time, and even with how slow many enterprise projects run, more than enough to get entrenched, build tooling that makes a bunch of stuff easier, build mindshare, etc.

> Have I missed out on some cool stuff?

stuff that makes companies money doesn't always look cool.

Plain-Old-Bash-Scripts 2018 ~

Both companies make most of their money providing professional services. I was going to touch on EMR but I think the “small team” comment is a better differentiator. When you have a large cluster with a large volume of workloads distributed computing becomes much less forgiving. The algorithm that was so fast and so valueable when you first deployed suddenly becomes a bottle neck for the rest of your operations because not enough time was spent thinking about partition strategies... Horton and Cloudera have made their money giving big companies advice when they get in over their heads. It’s rarely in the beginning but usually several years after companies have made a hard commitment to Hadoop and distributed computing.

Reply on news.ycombinator.com