Hacker News Reader

Comment by ridgeguy | Hacker News Reader

I've read that many military submarines have an "I've sunk" buoy. It detaches from the boat under commanded deployment, after some elapsed time without reset, or when external pressure rises way beyond crush depth. They're positively buoyant. They have radio beacons to report the event when they rise to the surface.

edit: No cites, sorry.

View on HN

https://en.wikipedia.org/wiki/Rescue_buoy_(submarine)

> As it may be difficult to manually release the buoy, or the compartment where the buoy controls are located may have been flooded, the buoys were arranged with automatic releases, in the event of a fire or internal flooding. Such automatic sensors proved unreliable and buoys were sometimes released unexpectedly. Accidental release of a buoy would have been a hazard during wartime operations, or even during exercises. There is some indication that unreliable buoys were welded into place. This may have been a factor in incidents such as the Kursk sinking, where the buoy was not released and it was difficult to locate the wreck.

I see so many parallels to bad devops patterns I’ve seen. You build your incident response protocols around a metric alarm (bouy being released). However the alarm is noisy so people either ignore the alarm or suppress it (welding the bouy in place). And then when an actual accident occurs your response is ineffective.

> And then when an actual accident occurs your response is ineffective.

Nothing can be actual human care and attention. Everything can break and no automated system made by careful people is exempt from needing that.

I'm reminded that late into development of a Pixar movie servers crashed, work was lost, and the automated back up system failed. The only thing which saved it was that an artist took a copy of the project home to work on it.

Reply on news.ycombinator.com