Comment by chrissnell | Hacker News Reader

Comment by chrissnell | Hacker News Reader

I'm a huge believer in colocation/on-prem in the post-Kubernetes era. I manage technical operations at a SaaS company and we migrated out of public cloud and into our own private, dedicated gear almost two years ago. Kubernetes and--especially--CoreOS has been a game changer for us. Our Kube environment achieves a density that simply isn't possible in a public cloud environment with individual app server instances. We're running 150+ service containers on each 12-core, 512 GB RAM server. Our Kubernetes farm--six servers configured like this--is barely at 10% capacity and I suspect that we will continue to grow on this gear for quite some time.

CoreOS, however, is the real game-changer for us. The automatic updates and ease of management is what took us from a mess of 400+ ill-maintained OpenStack instances to a beautiful environment where servers automatically update themselves and everything "just works". We've built automation around our CoreOS baremetal deployment, our Docker container building (Jenkins + custom Groovy), our monitoring (Datadog-based), and soon, our F5-based hardware load balancing. I'm being completely serious when I say that this software has made it fun to be a sysadmin again. It's disposed of the rote, shitty aspects of running infrastructure and replaced it with exciting engineering projects with high ROI, huge efficiency improvements and more satisfying work for the ops engineering team.

There is a huge difference between managing your own cloud infrastructure and letting clients run your big SaaS app in their own data centre. I think "on-prem" in the context of SaaS usually refers to the latter.

The article talks about something else: vendors should be able to run their big SaaS apps on customers' private servers as easily as they're doing it for public SaaS customers.

It effectively gives you, the SaaS vendor, the best of both worlds:

  - Access to markets you can't serve otherwise.
  - Development agility with frequent deployments.
  - Scalable ops/support for cloud/private deployments.

We see Kubernetes as a key enabler of this model.

Software vendors have often done some sort of limited operational management support in well-defined parameters. In a way I can see that SaaS is no different. For example, I was involved with a software product that we provided as a running system to the customer and took responsibility for maintaining: this was when everything was about "virtualization". Technically, we tightly defined the whole stack from hardware upwards.

Outside the excitement of containers and composability I'm not sure that the problem has been fundamentally simplified. The challenge is that a fully "operational" system from a customer perspective has to fully integrate with the rest of the neworks services. That's primarily around all the other elements such as monitoring, intrusion detection, backups and secure remote access: granted the article starts to address some of these. As a SaaS vendor you need absolute repeatability and a constrained set of 'additional services' to integrate with - otherwise the support matrix is going to grow massively. From a customer perspective I can see the advantages in being able to flexibly move across the boundaries depending on the circumstance.

Honest question, if you're running at 10%, why have you gone with 12-core, 512 GB RAM servers? Couldn't you start with a more reasonable 4-core, 64 GB RAM until some threshold? What value do you get from such an early overallocation of resources?

because the marginal cost of hardware once the chassis is racked is trivial compared to the public cloud, which is insanely marked up for retail on exactly this concept.

I have heard from a staffer on the GCE team that they deploy three CPU cores for every user facing core. That might have something to do with the cost.

I wonder why it is that companies like DigitalOcean and RamNode are able to provide far better performance for a fraction of the price.

GCE seems poorly engineered if the profit margins are indeed low (normal, that is).

I'm aware of one similar company that used to fit sixty $20/mo VPS in 1U. They were working on a couple hundred in 2 or 3U when I left (harder than you'd think), but they also have a cheaper tier now because of competitors pushing VPS down. Margins on VPS, even without oversubscribing RAM, are pretty decent once you get past the capital but they are decreasing in a race to the bottom just like shared hosting before it.

GCE provides significantly more reliability promises to us, from the PoV of user - As far as I know, DigitalOcean and RamNode don't provide things like no-downtime host maintenance (hell, AWS doesn't provide that)

Interestingly, the "3 cores for 1 user acing one" would explain the pricing of "preemptible" instances - which can cost as little as 1/3rd of full instance. And the main difference is that they are fully rescheduled every 24h or more often, and there's no live-migration for maintenance...

… I am now wondering if GCE runs fault-tolerant VMs. Because if it does, holy moly

Resizing a Digital Ocean VM takes me several minutes.

A GCE VM spins up in less than a minute. I've seen them come up in 20 seconds.

Except they don't. You give up reliability and uptime and not being on oversold hosts for a tiny bit more money?? So some real world benchmarks between DO and GCE. Not the same ballpark.

I am guessing having many instances helps them test their setup. And also allows for multiple environment deploys.

Several reasons. We did some POC testing before we deployed this gear and knew that we would achieve high density. Honestly, I didn't expect this much. Hard to believe that we've basically deployed every non-database app we have on this cluster and we're only 10% utilized.

Every service gets run in two environments: test and prod. Both are co-located on the same Kube cluster in different namespaces. We also don't put any datastores in Kube. That stuff still lives in OpenStack for now. Ceph can make it possible but for fast disk I/O, it's tough to beat the local SAS bus.

> Ceph can make it possible but for fast disk I/O, it's tough to beat the local SAS bus.

Personal anecdote - we have everything on-prem using a pretty standard vSphere setup, I've got a couple PostgreSQL databases that aren't even that heavily used (I mean, they top out around 20-40 tx/sec) backed by a hybrid Tegile array. Randomly throughout the day my IOWAIT starts spiking because even over 8Gb fiber channel our storage latency starts spiking because everything else is on the same storage (well over 200 full VM's), or sometimes I need to run a table scan over a 40GB table for a one-off query and the storage bottlenecks the crap out of us because only our small active set is cached on the SSD's.

You can run databases on remote storage, people obviously do it, but there's a reason why even on AWS the best practice is to use your instance disks instead of EBS volumes if you care about performance. Noisy neighbors suck.

I didn't consider latency spikes on slower resources like that, thanks for the examples.

Who rips the $1,000 processors and $200 voltage control modules out of their servers to upgrade to $2,000 processors and $250 voltage control modules, then re-plans their entire infrastructure and possibly code back around that?

Symmetry between servers has a value.

The main board, raid controller, network, and usually the storage is going to be planned out meticulously ahead of time based on the maximum load the server is going to see during it's lifetime. Often, Processor and Memory come down to "If we needed X Feature and we didn't have it" or "If we took the server down for 1 hour to upgrade it, how expensive is that?".

I misunderstood, sorry. I was talking in the context of virtual servers that can scale resources somewhat dynamically. (side note, isn't it awesome we see numbers like "512GB RAM" and don't immediately assume its not a single node in a deployment?)

I initially pictured someone picking the highest spec option they can afford when setting up a new service, rather than choosing based on actual demand of each node.

> "If we took the server down for 1 hour to upgrade it, how expensive is that?"

Putting my CI / DevOps hat on for a sec: who takes production servers down for upgrades without some level of HA to avoid downtime? ;)

post-Kubernetes era

Is it?

I think they meant "since the introduction of Kubernetes" rather than "after Kubernetes' relevance."

Yes, that's exactly what I meant. Thanks.

Yes but if you are targeting an on-prem solution toward certain customers, let's say enterprise, wouldn't you want to target the platform they are most likely to be running? Maybe that is k8s, but OpenShift and CloudFoundry seem much more enterprise-y to me.. How do you choose? How do you NOT choose?

If running your app on Cloud Foundry is an irreversible architectural decision, then those of us who work on it messed up.

CF is meant to provide a platform, not an API. The whole point is to liberate the developer from caring about how the app is brought to life so that they can focus on building the app. For operators the point is that no badly-behaved app can bring down production for everyone, so they can lower the drawbridge and let developers move faster.

I imagine that my counterparts at Red Hat would say the same about OpenShift. It's there to let you go faster.

Disclosure: I work for Pivotal, the majority donor of engineering to Cloud Foundry.

I feel you're underestimating the effort required to support an "on-prem" application across platforms.

I'm happy to re-estimate.

What am I meant to be estimating, exactly?

Do you have any public docs on how you will integrate with your f5's?

Nothing yet but it basically borrows a pattern that we use for nearly all of our Kube-related systems: descriptive configuration files bundled by the developer with their individual applications. So, alongside their code, their Dockerfile, their Jenkinsfile, their Kube specs, 'and monitoring spec, the dev adds a file that describes the load balancing. "This app needs a privatenet-only VIP on port ___, speaking protocol ___. For the health check do this ____. Load balance using ___ algorithm." This file (yaml) would be parsed by our build pipeline and fed to a service that manages the load balancing. This service initiates "watches" on Kube services and nodes and configures the service appropriately on the F5 via the LB's REST API, pointing the pool at the NodePorts of a subset of Kubernetes nodes.

We would probably build this around Kube's Ingress API. We're still a few backlogged projects away from addressing this internally. For now, we manage VIPs by hand. About 1-3 new VIP requests a week.

And you can just as easily run k8s in the cloud if needed with minimal transition costs. It truly is the ultimate definition of no lock-in!

Reply on news.ycombinator.com