I've been the victim of this, a producer consumer system with a single index file using some variant of a heap. For each incoming request the producer; locked the file, read the heap from the file, added the value to the heap, then wrote it back and unlocked it. For each consumer thread, the thread did the reverse. It was setup with threads created by incoming requests, and a threadpool of 128 threads for the consumer.

This worked seamlessly until one day, at 3am PST, a flood of requests came in and the index file grew to ~4mb, and everything ground to a halt.

The legend of the architecture was a senior engineer did this on a weekend, then a month later left the company.

I think, you are misreading it.

It isn't about making an architectural design decision / change at 3am, but someone has to live with the consequences of such a design decision and fix something in some way or another at 3am in the morning.

I.e. you could go with a clustered db setup which is supposedly HA, or a classical single instance DB...

The point here is, whoever has to get up at 3am to fix a potential catastrophic failure in the setup should have a say in it.

It’s a variant of the old “Would you bet your career on this?”:

https://dilbert.com/strip/1992-09-06

I agree, if you design it right, which isn't easy.