Datadog’s $5M Outage

> I’ll use Central European Summer Time (CEST) for time stamps, as the majority of Datadog’s engineering team is in this timezone. I’m also indicating US Eastern time (EST) for reference.

I can convert from UTC to my timezone (TZ) in my head, since logs are usually UTC. I know other engineers can do this, too. What I cannot do is convert from a random TZ to my TZ without looking it up. Clearly, the author knew this, which is why EST was added for an American audience.

Note the Datadog post-mortem is UTC [0], but the status page isn't [1]; even the EU one is ET [2].

So why are time stamps not being in UTC a thing?

Reasons I can think of:

* Status pages and post-mortems are not for engineers, but for management?

* Certain US-based companies use ET (i.e. EST or EDT) or PT (i.e. PST or PDT), and everybody else has cargo-culted it.

[0] https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...

[1] https://status.datadoghq.com/history

[2] https://status.datadoghq.eu/history

The style of the article is overly verbose to me, but the underlying incident is fascinating. Most of my workloads run on Windows Server, and having a monthly requirement to reboot due to updates is definitely annoying, but also a great way to force you to carefully orchestrate distributed system cold-starts from day one...

> This incident was costly, at $5M in lost revenue directly as a result of the outage, as revealed in a later earnings call. This was due to usage-based billing

No SLA whatsoever? Send Datadog your data, they lose it all, no discount or penalty but at least they don't charge you for storing it?

https://news.ycombinator.com/item?id=35971320 https://news.ycombinator.com/item?id=35963879

We’ve recently signed up with datadog. I think their platform is good, but recently I’ve seen posts on a few platforms stating that they are known for up charging significantly and without warning. Wondering if anyone has experience with this and if we should have went with someone else

TLDR: Ubuntu unattended-upgrades and the systemd-networkd `ManageForeignRoutes=true` [1] default broke all Kubernetes networking globally, across multiple regions and cloud providers simultaneously, at 06:xx UTC. Recovery was difficult and slow due to the scale, particularly with cloud auto-scaling behavior kinda making things worse when all nodes failed simultaneously.

Please test how `systemctl restart systemd-networkd` affects your custom networking setup. It is quite likely to break.

DataDog also published their own post-mortem: [2]

[1] https://www.freedesktop.org/software/systemd/man/networkd.co...

[2] https://www.datadoghq.com/blog/2023-03-08-multiregion-infras...