Hacker News Reader

For instance at Google, taking vacation in workday puts an OOO event on your Google calendar. Then the oncall scheduler takes that into account along with your oncall history when it schedules new rotations. You only have to get someone to cover if you are taking time off in the near future, and any surplus or deficit will be fixed by the scheduler going forward.

It's kind of wild that an expensive product with tons of funding and employees is worse than some scripts cooked up by some SREs.

I think outside some FAANG firms with strong engineering management, the purchase of these tools is a top-down affair.

So as cynical as my statement may seem- " It's almost like they wanted it to be easier to just not take time off?" .. user ergonomics just doesn't enter into the conversation at all, because the users are internal devs.. who cares.

I also worked at an org that gave each of 50+ large customers a dedicated slack channel with.. wait for it.. the entire internal 200+ member engineering org in channel with them. You can imagine the mayhem as each customer figured out how to get attention (@here/@channel/@theguythatfixedthislasttime) and "hack the phone tree" as I used to call it.

This also meant the every one of our 200+ engineers was a member of 50+ customer channels which were filled with all levels of noise that 95% of the time had nothing to do with their app/team/function, and could/should be ignored.

> the purchase of these tools is a top-down affair

This is absolutely it. Pager Duty don't sell on-call tools to engineers, they sell the idea of having an on-call rota to CTOs, the tools are an implementation detail.

Bottom up engineering tools take more time to start with, but almost always cost less in the long run, contribute to a good engineering culture, and build a sense of ownership.

That isn't universally true. At my medium sized company, the people on call are the ones responsible for choosing an on-call system. But we've rotated through most of the main options, because we haven't really been happy with any of them.

I am in management. I evaluated most of the on-duty tools that are available, at the cost of a couple of days of my personal time. At least they all have free trials. PageDuty, Opsgenie, Splunk On-Call (ex VictorOps), AlertOps, Squadcast, TaskCall, etc. etc.

They are all surprisingly hopeless. For example, none of them have SCIM integration to make managing your teams in them automatic. All of them have clunky calendar overrides. None of them seem to integrate well with Outlook / Google Calendar, particularly none take into account holidays. Many have no Terraform provider to manage them, and the ones that do are clunky at best, and hard to set-up/manage. OIDC is hit-and-miss. For example, for PagerDuty you need to call up their support team and get them to manually tweak something that's not in the UI settings to get OIDC for Azure AD sign-ins to work.

It's not that management is apathetic. We genuinely don't want to engineers spending their time working around vendor inadequacies and lashing this stuff together with barely-maintained scripting that they resent having to write in the first place. Why would anyone want that? Given there's seemingly no product out there that lets you avoid that, what should we do? When they're all rubbish, you either choose PagerDuty because everyone does, or Opsgenie as a protest vote, because at least both have Terraform providers and plug-ins for other things like Slack and Sentry, etc.

My point is not that management is bad & doesn't care, but that management has different priorities & cares about different things than the people using the app.

All of us have talked on this thread about all the things PD is bad at, but few have come to the defense with a list of things its great at.

The sales pitch, I am sure, to management is more along the lines of - metrics, dashboards, reports, "accountability", response times, yada yada.. and because management write the checks, I'm sure thats an area they do deliver.

It's much much much easier to cook up a solution for internal use than to create a product for external use that you can sell and that caters to your customers whims and that reasonably easy to use without requiring direct access to the engineering team for support.

You're right, but for $173mm in funding, you'd think you could build a product for external users that's at least feature parity with a "cooked up" internal project.

And a $3B market cap, unbelievably.

What's kind of incredible too is they actually lose money still. The losses are growing faster than the revenue. They've only had, what, 14 years to figure this out? You wonder what their staff of 1000ish are working on.

Also, forgot this gem: "On January 21, 2023 PagerDuty CEO Tejada's layoff memo was criticized for insensitivity for inappropriately quoting Martin Luther King, announcing promotions of executives, and tone deafness." (from wiki)

I’m glad to hear I’m not the only one. I constantly felt like I was missing something. A tool this mature and popular should have been a little more polished by now.

It felt like a tool that was sold to CTOs, not users.

As a recipient of pages, I abhorred PagerDuty more than Slack, Teams, email or even phone calls.

At least my phone notifications & do not disturb mode can be tuned around some of those other apps. And if a support guy or teammate calls me on holiday, I can berate them such that they remember to check the holiday calendar / out-of-office status next time. No such luck with PagerDuty.

If your org is not mature enough to put in the time to have a single pane of glass / view of the world / alert state.. you shouldn't be buying any of these tools.

I tried to make managing the incident escalation policies as simple as possible. It currently works with a simple drag&drop mechanism. The way PagerDuty is doing this also seemed very cumbersome to me. Of course All Quiet is currently missing holiday / off hours features. I'm still figuring out what's the best approach there.

Regarding your last paragraph - can you elaborate on this? I tried to make All Quiet be integratable with basically every system by Email and Webhook integrations. Do you see this as a problem?

Thanks for your feedback in general!

I think the problem with integration is probably a man-hours thing. A lot of shops have a hodgepodge of 3rd party tools which are not necessarily the latest&greatest Cloud/FAANG darlings - Geneos, Jira, ServiceNow, ControlM, Autosys, Tidal.

These tools can vary by industry & org size, so maybe finding some traction in a particular slice so that you can really serve that niche can give you more traction, I dunno.

It just has to be dead simple to integrate, and it's worth considering some sort of professional services/customer success engineering layer.

The problem I've seen is that in every (financial services) org I have worked, the teams responsible for integrations to these types of tools are really operations type teams that do a little development in their spare time. So it never gets done well, or completely.

To me, the only benefit of PagerDuty type tool is when it becomes the single point of escalation. If its just one of many, the product will not be sticky within a given org.

Re: cost & holidays/hours

On pricing - I don't think it matters for small orgs the way it does for big orgs. Let's say today I am in a role where I am effectively the CTO of a 5ish person startup, with only 3 technical. So whether I spend $200/year or $1500/year, I don't really care. Both are more than $0, and require me to review a contract, do a POC, have a lawyer or someone review things, etc.

My internal time to do all those things is more like $10-30k, easily.

Now, do I decide it's worth my time knowing that my devs are going to basically turn off their phones / go to do-not-disturb-mode still because it's sense of working hours is non-existent (making it worse than PagerDuty & Slack notification tuning, both of which I already despise)?

'Taking a holiday became like a 12 step program - email boss in advance, put in HR system request, cross check your PagerDuty schedule, negotiate a trade with another teammate for PagerDuty rotation, decline meetings, update your outlook calendar out-of-office, set your slack status & notifications, and re-remind everyone the week before you go. It's almost like they wanted it to be easier to just not take time off?'

Most of this has nothing to do with pager duty, but instead is around the companies process.

The problem is top-down CTO purchasing of these magic bullet tools w/o budgeting/planning integration.

Instead becoming "the place to look" it becomes "the Nth place to look" for alerts. Without thoughtful planning, its entirely possible to be in worse shape for having purchased it.

I’m not sure why PD is so expensive, but I think why it’s just known over competitors is just the sheer number of integrations it supports.

> decline meetings, update your outlook calendar out-of-office

When I am adding OOO to my Google Calendar, there's an option to auto-decline meetings. Isn't it like that in Outlook?