Hacker News Reader

If your alert is going off at 5:00 am and self-resolving at 5:01, and it happens more than once, the prudent course would be to fix the underlying problem or modify the alarming configuration. Nobody benefits from alarms that don’t require action.

Hire in complimentary timezones so no one has to be on call at night

Unless you're hiring 3-4x the engineers, then this doesn't scale. I would much rather occasionally be paged at night than have to be on call all or even most of the time.

Even if that "on call" was only 9-5, getting distracted and losing half a day to tracking down the latest flaky test or owner for a failing dependency really adds up and makes it significantly more difficult to do my day job. When I'm oncall, I'm oncall 24/7 for a week (30 minute SLO) and the expectation is that the amount of non-oncall tasks I'm able to get done that week is significantly impacted. I'm fortunate to work on a team that acknowledges that reality; I know many people, even at large software companies who should know better, whose management is far less reasonable.

It's true but most people do not have the luxury of being able to focus on a single task each day until done. If that is a requirement for you I'm not gonna knock it but I feel it is rare to get.

We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC

Plenty of good devs open to competitive offers down here in APAC.

I joined a US company as part of a follow the sun model. Although ironically I seem to support people who are up at midnight in Europe more than in my area of the world.

Happy to have a chat, see if I can offer any useful feedback :)

Hmm, living in APAC, it seems like companies in the US are either never hiring for remote positions overseas, or offer laughably low salaries.

Yeah, I wish I had more power here.

We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC.

Pay more. It literally solves all recruitment problems.

Gee, I wish I could control that.

Salaries in the EU are not what they were and have improved dramatically. I've found most companies to be wildly off the mark when making offers. Perhaps you need to recalibrate.

It’s true, and it’s sad to see. A long time high performer on our team based in Germany makes a pittance compared to more Junior US based folks. Alas, I cannot do anything about it.

My current on call setup that page would count as one hour worked of my weekly 40.

Ideally I'd be paid extra as well (I get about an hours pay a week as an on call allowance). What makes it work is that I'm the senior dev on the project that I'm on call for - if it hits the fan it really is my problem. And more accurately, in the last two years I've been called out twice. So that extra hour a week of pay... it's just free money.

> it's just free money.

No. The downside of being on-call is not the number of times you get paged at 3am. The real downside is that if you are on call 24/7 and you have to react to an alert within the next 15min that means:

- you cannot go out for dinner

- you cannot go out for a run

- you cannot go and pick up your kids (if the ride takes more than 15 min)

- etc.

The only thing you can do is stay at home with your laptop next to you, or wherever you go, take your laptop with you. That's being on call, and it's NOT free money.

If actually getting paged is rare them just bring your laptop to dinner, that's what I always used to do. Sure, it might be that time you are unlucky and have to work through dinner which sucks a lot. But it shouldn't happen much.

Running is an issue, unless you don't mind a sweaty back to carry your laptop, or just run loops nearish to your house with your phone on you.

What I usually did when picking up kids (close by, about 30 min round trip, YMMV) was to ack the page from my phone and log in when I got back home.

Yes it sucks to be tethered to a laptop. But if after-hours pages are rare, and you aren't on call more than ~25% of the time, for me personally it's acceptable

That's not how my on call setup works. I need my phone and my laptop, and I need to be able to respond within half an hour or so. The changes I have to make are in code that then goes through CI and a load test before hitting production, so there's no "you have 10 seconds" stuff.

We don't have anyone on call in the sense you mean, and I've never seen it in the wild. If they want someone right there all the time they've paid them to be in the office where they have proper access to critical systems. When I worked in a medical environment that was what we did, and those people got paid very well for doing overnight "tech support" ... they were bored enough that answering random IT questions seemed worth while for them.

That’s a strange setup. When I was the only developer on a service the only guarantee I could give was that I’d try to make my way to a computer as soon as possible if I got a call, but I didn’t bring my laptop with me all the time.

A lot of stuff can be resolved from your phone though.

After reading a bunch of the other messages here, I think the reason it was ok for me was that I was always in a position to fix the root cause. I wouldn’t ever get the same alert twice if I did my job correctly.

Reminds me of my first job: on call didn't change much at night because the system was so twitchy (as in, SMS every 15 minutes that then autoresolved) that I just put my phone on silent and ignored it over night.

Of course we also had a night shift guy who would actually call you if he got stuck, so hell if I knew why the SMS system existed. (That role lasted about a year at most - night shift for tech is stupid).

Someone once had the bright idea of forwarding several international toll-free numbers directly to a developers phone on a rotational basis. All to save the internal costs of the in-house operations team. We only ever received spam calls and wrong numbers at odd hours. It was a nightmare and total productivity killer.