Hacker News Reader

Comment by barhun | Hacker News Reader

2k nodes, 100k containers https://blog.online.net/2016/07/29/docker-swarm-an-analysis-...

5k nodes, 1 million containers, 5 minutes https://www.hashicorp.com/c1m/

(Disclaimer: I'm on the Nomad team but wasn't at the time of the post)

I don't know much about Nomad and couldn't work out from the repo what the jobs were. If I guess correctly, it's an app using Redis. Is that correct?

Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

> it's an app using Redis. Is that correct?

Yup! Repo could definitely be clearer, but here's the code:

https://github.com/hashicorp/c1m/blob/master/schedbench/test...

Basically calls an increment in Redis and then blocks forever.

> Disclosure: by coincidence of market forces, we're mortal enemies. Let's send christmas cards!

Haha, hi mortal enemy! Christmas cards it is! If you're ever in Portland, OR I'll buy a beverage of your choice as well. :)

I extend the same offer in New York!

Since we're playing this game:

1.55k nodes, 250k containersed applications[0].

Mind you, it's hard to compare these as there's no real "cloud bench". For pure benchmark porn Nomad are the undisputed champs on their 1 million case.

The Cloud Foundry scaling test was intended to show a system with fully service-configured, fully-routed apps, with varying app characteristics (memory and RPS). To further stress the system, thousands of apps crashing and are relaunched on a continuous basis.

Cloud Foundry installations with >10k containers have been ordinary for a while now; the 250k thing was to ensure we had lots of headroom and shake out chokepoints in Diego.

[0] https://content.pivotal.io/blog/250k-containers-in-productio...

Disclosure: I work for Pivotal, the majority donor of engineering on Cloud Foundry.

Big respect for your achievements. I guess at some point it just becomes the question of "where do i get a 1000 nodes" vs. "how do I run a 1000 containers". Or, more the justification for that amount of hardware - I mean, the one dream job which I would probably want is getting paid to cut out all the hardware use while keeping reliability/availability/functionality. Like these guys who cut their AWS bill by $1mil/year in about 3 months - https://segment.com/blog/the-million-dollar-eng-problem/. The thing is that I'm not exactly sure where I'd fit in more - running this thing, or just fixing it for somebody else. I definitely know that I'm mostly dealing with pets and not cattle :)

Well, Cloud Foundry is deployed by BOSH. So you can, if you wish, use RackHD to deploy it to naked hardware (instead of OpenStack, GCP, AWS, Azure and I forget what else).

Your apps will still be containerised, distributed and wired up the same way.

There's always a point at which it makes engineering sense to flip the switch to doing it yourself. But that frontier is never static. We (plus our peers in the Cloud Foundry Foundation) and others in this space like Red Hat OpenShift are constantly pushing back the tipping point at which it makes economic sense to DIY.

We already have very large customers with very large engineering teams, who've built platforms before. And they are switching because that effort no longer makes business sense. It's an expense they don't need for a platform they're the only maintainers of.

One of our peers at IBM wrote about DIY[0]. We have our own much more markety-businessy whitepaper, with a very detailed case, on the same topic[1].

[0] https://hackernoon.com/stop-spending-engineering-effort-solv...

[1] https://content.pivotal.io/white-papers/the-upside-down-econ...

Disclosure: I work for Pivotal, etc.

Reply on news.ycombinator.com