Hacker News Reader

Comment by hayley-patton | Hacker News Reader

Did a devops elective last semester; I was surprised to hear that sleeping to wait for the system to come up was "common in the industry". 30 seconds was the magic number, which was somehow always either mostly a waste or not enough when I tried to use it.

For similar reasons, we similarly can't keep making sleep-sort faster by scaling the sleeps down.

View on HN

I've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so.

I've been endlessly frustrated by how prevalent "just add a <x> second sleep" is in the Kubernetes world.

Examples:

If you're using IRSA to provide pods an AWS IAM Role, it can take some time on first-run for the role to become usable. So your app has to just wait until it can successfully talk to AWS.

If you're using the AWS Elastic Load Balancer controller, you need to add sleeps during pre-stop for a pod. The controller has to see the shutdown event, then start target deregistration on the ELB Target Group. That's a minimum of 30 seconds, during which you can be getting new connections.

Service meshes on Kubernetes add some fun on both startup AND shutdown.

Linkerd, for instance, will only inject it's linkerd-proxy container into your pod after startup. So it's entirely possible for your main application container to start, begin talking to the world and then a moment later all your connections get reset when Linkerd-proxy takes over the pod's networking configuration.

You can fix this by injecting into all of your containers a linkerd-await utility that replaces the container entrypoint. It will wait until linkerd starts before starting your regular entrypoint. It's basically a more advanced version of sleep.

Shutdown is just as fun - linkerd-proxy will immediately terminate, dropping all network connectivity for all the containers in your pod. There's a fix for this, too, which involves the above linkerd-await and another option.

With systemd having standardized so much stuff on sd_notify for a basic liveness check, it's frustrating that container setups can't piggy-back more on that.

Systemd's socket activation mechanism also got born from the "Sleep is not a synchronisation primitive; talking to a socket is!" and the container world just ignored it whilst it's a great fit!

Actually it's a generalisation of what OP is doing here. Move the listener _out_ of the program and you never have to know if a service is up or not. Connecting to a service just blocks until it's up.

> It's basically a more advanced version of sleep.

Is then the waiting for the availability/existence of anything a "more advanced version of sleep"?

Is a readiness probe a sleep?

I don't follow at all.

Polling relies on sleep. Setting the polling time matters a great deal, because you'll almost always take at least one poll interval to discover that something has come up, and on average every polled step in your startup process adds (poll time/2) to your total start-up time. If you have a bunch of steps that require polling, that adds up fast.

The problem is often relying too much on cold-starts; these polls amortize well in a real system, where you can afford (and want to afford!)

Sleeping on error is a valid pattern when combined with retries. Consider if that would be a long-starting external process that sets up the listener (or even a remote server spinning up). I.e. something we have no control over and that has no ability to signal us it's ready. In such case, sleeping a little then trying to connect (and repeating) would be simple, effective and economical solution.

Sleeping on error is a valid pattern when combined with retries.

Agreed. Just note that it can sometimes (often?) be a good idea to add a small stochastic element to the sleep time, and/or do something similar to the exponential back-off[1] approach that Ethernet (among other things) uses.

[1]: https://en.wikipedia.org/wiki/Exponential_backoff

I've never personaly seen any benefit from exponential backoff... constant backoff + jitters + short timeouts has always worked well for me.

If anything, people underestimating the ramp up of exponential back off(or not capping it) has left me waiting much longer than necessary

the amount of times something (Discord is the first example that comes to mind) starts waiting 30 seconds between retries because I wasn't connected to the network has been super absurd.

You ideally want both exponential back off plus some random delay (to prevent thundering herd).

That was "more sophisticated" and I was given a sample script without a loop. (I got Terraform to wait in a retry loop, which was apparently too complex.)

Yeah for the record I'm not talking about delay loops, I'm talking about hardcoded sleeps in test code that cause random failures when unspoken invariants change. DevOps is a unique blend of hellfire that is almost unfair to compare to everything else in the industry. I'm glad I don't work in DevOps anymore.

> we similarly can't keep making sleep-sort faster by scaling the sleeps down.

Technically, we can, at least to a certain limit - as long as the total time needed to schedule the sleeps is less than the shortest sleep duration, sleepsort will still work. Though we're just offloading the sorting to the scheduler, which has to implement the sorting algorithm anyway...

This bleeds into the Init Wars (sorrows, prayers), but there was this idea for a while that every service was going to be directly queryable by the run control system so that there would never be a need for magic sleep numbers. The result fell rather short of the ideal (particularly on the teardown side of things) but it did at least help admins get a firmer handle on exactly where we were (and still are, for the most part) using magic waits of "long enough".

Way back I worked on a project to used virtualized routers inside labs for a bunch of use cases. The idea was to remove the need for building after building filled with equipment that just sat there idling 99% of the time.

The biggest headache we had was to figure out when everything was up. There simply was no way to detect this programmatically, so the only way would have been to alter the code to give some kind of signal on the console like "System is up." similar to a linux box showing the login prompt. So I resorted to wait loops too.

We ended up canceling the project.

There have been several developments in the past decade or so that center on the philosophy of "make it the client's problem" (hello, Wayland) which has simultaneously led to much more robust and maintainable centralized systems with much more fragmented and unpredictable peripheral clients. I'm still not sure whether I regret this or not.

I ended up writing a function that would sleep and retry according to the Fibonacci sequence up to a given total elapsed time, so you could set it to a max timeout of 60s, for example, but if the system came up in 8 seconds, you'd be waiting not much longer than that (11 seconds, technically).

I do this too occasionally — Fibonacci is easy to implement, and it's an exponential backoff (with a lower exponent than 2, which is also easy to implement ;) )

I found the following line in a codebase once, which made me laugh:

``` interval = interval*1.6 # fiboroughly ```

I think the sleep N > check for availability paradigm is pretty useful.

The alternatives have a ton more complexity for not much benefit.

Reply on news.ycombinator.com