Hacker News Reader

> If a server is running at 100% CPU but 30% of that is spent performing background work, the amount of slack available for a sudden surge in demand is 30%,

In this context, the first half of that sentence is usually interpreted as something like "every 10 days, the computer is capable of doing 10^8 tasks, and there arrives 7×10^7 time-critical tasks and 3×10^7 background tasks."

As you can see, if the background tasks eventually need to get done, demanding more of this system doesn't work because it would not be able to finish what it's supposed to. 100 % utilisation leaves no slack, no matter what kind of tasks they are.

Your proposal only works if

- Utilisation is less than 100 % when looking at a longer time frame (then we can do fewer background tasks during high load but catch up on the backlog when load is lower);

- You are able to spin up new workers in response to increased load (this is the same as utilisation being lower than 100 %); or

- The background tasks aren't actually demand at all, but just things that are nice to do opportunistically. (And then again, utilisation is lower than 100 % even if it doesn't seem this way.)

I was assuming that it's option 2 (you can spin up new workers within 10 days). If you're using cloud compute this is almost always true.

If you're building an on-prem cluster, I'm assuming you either spin up cloud workers for the extra load, have extra servers that are fully powered off and can be booted in a few minutes, or just physically order and install new hardware. Amazon can ship you a computer in 2 days, so it's not inconceivable to design an on-prem infrastructure to allow significant scaling with short notice.

Option 3 is sorta always true as well, since there's always compute load at your company that's lower priority, and can be shed in an emergency (for example, your CI system).

In any case, I suspect the common pattern of scaling is that most companies grow in a smooth enough way that they can predict demand in future weeks within a couple percent, so you can just run at 98% CPU instead of 100%

The idea that you can run an on-premise cluster and just supplement it with cloud workers is a nice idea in theory. But the reality is that this never happens.

Because the two main reasons that a company would be running an on-premise cluster is (a) security and (b) cost. Both of which conflict with running a hybrid model because it is extremely complex and expensive to do so.

Perhaps today this is true.

In 2007 I was working for a webhostet / nascent iaas provider, and we had workloads doing exactly that.

Specific example would be Celtic FC who had a baseline of dedicated, but would scale into our VMs during events, i.e. uefa cup games.

Reaching into on-prem hardware from the cloud is a substantial engineering effort, and one that can land you in the news if you do it wrong.

Good points.

For security concerns, you would probably be selective about which services and workloads you move to cloud compute, possibly preferring to offload internal services like CI before customer data.

Regarding cost, are you implying that there's large monetary cost to preserving the ability to spin up cloud workers in an emergency, or just that once you do it's expensive per day to keep them running? Presumably if you're running an on-prem cluster for cost reasons, you would work to scale it up quickly after an unexpected sustained demand spike, and the cloud supplement would be temporary. If you're getting "unexpected" spikes every week, you need to work harder at forecasting.

In any case, I suspect today that people running on-prem clusters provision them with the intent of being able to handle a load growth and keep weekly average utilization at <60% during the spike. Investing in building infrastructure that keeps working smoothly when weekly average load is 90-100% means you can provision a smaller cluster either way.

“Never” is way too strong.

This technique is called “cloud bursting” and is done often enough to have docs on Azure: https://azure.microsoft.com/en-us/resources/cloud-computing-...

More generally, hybrid on/off prem (or multi-cloud) is a top level feature of GCP’s Anthos.

You’re not wrong about your two main reasons, and it’s true that it’s hard to wire up. But you definitely can wire up hybrid workloads in some cases.

The obvious one would be stateless batch compute like image/video processing; “I prefer to re-encode my videos on prem but can burst to the cloud if I get a traffic spike”. This might end up being cheaper than either of all-on-prem with overcapacity, or all off-prem with auto scaling.

I agree it’s not likely to be useful for something where security is your reason for going on-prem.

Yep. For example, I use control theory to keep my services at just single percents below maximum throughput achievable on the server. Then I use other tricks (like batching processing) to make the application MORE efficient as the traffic increases.

The end effect is I can just back off 1-5% off the maximum throughput and keep the service there running happily.

I would like to use this occasion to point out that all the discussion about unused CPU is at this time completely pointless.

Most services I have seen waste ORDERS of magnitude by being inefficient. Rather than focusing on trying to saturate the CPU and other resources it is almost always better to just make your application more efficient. That last 30% should be a cherry on top.

How do you use control theory?

You know how many systems have "performance" configuration? I use a controller that monitors the state of the system and changes these parameters in real time to regulate system to stay within desired state when the environment of the system changes.

As a very simplified example, imagine a backend service that is being called by external customers and does not control how those customers are calling the service. I can add a delay to each response and I can have even something as simple as PID controller regulate the CPU usage by changing the dalay. Larger delay will usually cause the clients to slow down requests (requests being usually a result of previous request completing). This is simple and naive example but this is more or less what I do.

(Of course, in reality, it is much better to just have a backpressure mechanism and whenever possible you should use one rather than try to work around HTTP inadequacy. But you can't always do it, especially if you have a public API.)

I also typically have lots of other controllers. For example something that regulates memory usage by limiting transactions in flight or something that regulates latency as seen by priority clients or database replication rate/delay, or error rates or a bunch of other parameters.

I also routinely take care of babysitting downstream systems like databases or other APIs. I may have a regulator that will automatically start backing off certain types of traffic as a response to increasing error rates or latencies in a downstream system. All this because those downstream systems are usually shit and not designed to deal with overload and it is easier for me to deal with this proactively than do what everybody else does -- keep bugging those people to fix their issues when their evidently don't know how.

This is extremely interesting!

I have been trying to move away from dumb rate limiting to a more holistic approach that allows us to make smarter decisions with traffic. Your overview made me intrigued.

Do you have any references you like to use? I am looking at the Wikipedia page, but it's so removed from practical aspects.

I don't. There simply isn't any tooling or literature to speak of. I have some experience using control engineering in my electronics projects and that's how I came up with the idea to use it for backend systems. I have researched and developed everything myself. I have used "Modern Control Engineering" by Katsuhiko Ogata, but really, mostly I just learned from the Internet.

My initial motivation was to remove configuration. I have found, historically, that giving people options to configure very complex software more often than not results in problems, especially after original developers leave. More often than not these new people will not understand the implications or interactions between various settings and this will just cause problems. So my aim became to remove any options from the software and make sure it can perform autonomously and recover from wide range of, possibly unknown, situations. Which is exactly what control engineering is about if you think about it!

If one day you'll write a blog post / article about what you're doing, it'd be interesting to read :- )

(What if you start collecting email addresses to people who want to read such an article? And if one day you write one, then you can email them? — My email is in my profile, if you'd like to add it to such a list)

Dunno... most people I meet seem to be put off by my software development ideas. I stick to them because they seem to work very well even if it initially creates a lot of friction between me, the team and the management.

Where to start... I think test driven development and unit testing is not giving promised value and instead wastes time and makes software more difficult to refactor and I think functional end to end testing to be much more effective and cost effective. I think code reviews are bad because they don't deliver on promised value and individual craftsmanship (peoples ability to deliver on their own) and pair programming are better. I think microservices to be a wrong approach for 99.9% projects and fixed a bunch of projects by rolling the software into monolyths. I believe bugs can only be truly reduced by taking responsibility for writing correct code in the first place and anything afterwards is expensive and not effective (you can only remove bugs that manifest themselves, everything else stays). I don't compile/run my code multiple times a day -- I write it all in one go, sometimes for weeks, then run it. If it works it means I know what I am doing and if it doesn't -- it is the failure of my process. Where most devs just fix the bug and restart the app I will start an investigation into why my process failed and how I need to fix it -- NTSB-style. I believe that nobody understand what Agile is and the way it is applied is damaging to software industry. I don't believe in linear development progress -- I design my apps top down and at the same time program them bottom up until top down and bottom up meet together. I structure my development process around rewriting the software -- I write the first version and then I will refactor/rewrite to remove any unnecessary complexity until I am happy with it. There is no working software for a long time and then suddenly it is complete. And when it is complete there is no more testing stages, bugs to fix -- it is truly complete.

So you see, I am probably too alien a developer to give advice to general population of developers.

And when I do talk about my ideas it usually ends in flame wars or drowns being downvoted to hell because people tend to downvote anything and everything that does not confirm their existing worldviews.

Batching (and sorting and merging) are things our predecessors in the 1950s and 1960s (and before that, in the card era) had to do to run anything at all. These days they are things that we may do to make sluggish systems snappy.

Beware of I/O, memory usage and cache eviction.

A low-priority background job can issue I/O requests that result in high I/O latencies for everything else (by e.g. making lot of random seeks).

This is very easy to get when using spinning rust.

If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the number of cores allocated to foreground vs background tasks (within less than your latency SLO). That means each core is only doing one kind of work for many milliseconds in a row so CPU caches remain hot, and I/O can be sorted into separate low and high priority queues. IO blocking is much less of a concern with NVMe drvies anyway, since they're intrinsically so parallel.

Memory usage is still a concern, since you do need to have the code and in-memory data for both workloads loaded on some number of your servers in order to rapidly change what tasks they're doing. Physical machines have so much memory today (compared to the code size or working set of most programs) that I suspect this isn't actually a blocker in practice, and it's not necessarily required for every machine in the cluster to have both workloads resident as long as some of them do.

This definitely feels like diving into an alternate universe of cluster management, and I'm not sure how easily Linux gives you tools to do things like this.

A long time ago, I was surprised to discover that a mainframe always runs at 100% cpu usage. Its OS makes sure that multiple VMs, each of which has a long list of real-time and batch jobs, neatly prioritized, keep it well fed. An interactive user need an answer now, but a clean-up job can literally take weeks to run. When I looked amazed, the operators told me ‘This thing is so expensive- it cannot be wasted’

> it would be really nice if load balancers used realtime performance metrics to balance traffic at a millisecond level

With Kubernetes this is trivial. Horizontal Pod Autoscaling integrates with Prometheus so can spin up a new instance of your service based on whatever custom metrics you like.

And it is proven to work without requiring some "creative solution" that will be buggier, less secure and inevitably less maintained than something that is industry standard.

Prometheus+Kubernetes needs in the order of 30-60s to scale up, not a good match with GP's "millisecond level" ask.

(scrape interval is 15s, pod creation requires pulling images)

> (scrape interval is 15s, pod creation requires pulling images)

You can reduce the scrape interval and depending on the cluster provider the image pulling should be extremely fast once cached but indeed it would be on the order of many seconds rather than milliseconds.

I don't think you understood what I was suggesting here, I'm referring to the load balancing system not auto scaling. Imagine a cluster with two nodes, that gets three requests. A round robin load balancer will distribute the first and third to node 1, and the second to node 2. However, if request 1 requires a lot of work for some reason (either it's an intrinsically complicated request or triggers gc or something), that means node 1 ends up heavily loaded and node 2 is underutilized. A smarter load balancer could notice that node 1 is still busy, and redirect request 3 to the node that has capacity available at the instant it comes in.

Given large enough numbers of small requests, it should all average out in the long run, but that requires over provisioning by enough that servers can still handle their "fair share" of requests even while running GC or otherwise dealing with unusually difficult requests. Realtime smarter load balancing should require less overhead.