Comment by shykes | Hacker News Reader

Comment by shykes | Hacker News Reader

Parent Full thread

Hi, I work at Docker. Here is my reply on the github thread: https://github.com/docker/docker/issues/23203#issuecomment-2...

I am copying it below:

<<< Hi everyone. I work at Docker.

First, my apologies for the outage. I consider our package infrastructure as critical infrastructure, both for the free and commercial versions of Docker. It's true that we offer better support for the commercial version (it's one if its features), but that should not apply to fundamental things like being able to download your packages.

The team is working on the issue and will continue to give updates here. We are taking this seriously.

Some of you pointed out that the response time and use of communication channels seem inadequate, for example the @dockerststus bot has not mentioned the issue when it was detected. I share the opinion but I don't know the full story yet; the post-mortem will tell us for sure what went wrong. At the moment the team is focusing on fixing the issue and I don't want to distract them from that.

Once the post-mortem identifies what went wrong, we will take appropriate corrective action. I suspect part of it will be better coordination between core engineers and infrastructure engineers (2 distinct groups within Docker).

Thanks and sorry again for the inconvenience. >>>

Hi! I work at Docker as well; and happy to tell that we found the cause of this issue, and managed to resolve it.

Be sure to clean your apt cache before trying again (apt-get clean && apt-get update) if you're still seeing this.

Have you got a link to a post mortem?

  > At the moment the team is focusing on fixing the issue
  > and I don't want to distract them from that.

That might be ok for feature teams, but for infrastructure tools/services, it's very frustrating for users (devs) to be kept in the dark on the progress of the fix.

At work, the incident response starts with identifying Investigators (to find and fix the problem) and a Communicator (to update channel topics, send the outage email & periodic updates, field first-line questions about the incident, and to contact those most affected by the incident so they don't get surprised/try to fix it themselves). The person who starts the incident is the Coordinator, who assigns the roles, escalates if more help is needed, tries to unblock investigations, and turns facts from the investigators into status updates for the communicator.

i will provide an opposing viewpoint which i'm sure many people do not agree with.

if a service i use is down, all i want is an acknowledgement and that "we are working on it right now with high priority". i want all available resources to be fixing the problem.

my anxiety over powerlessness in relying on others during a crisis manifests in other ways, like figuring out why i'm at the mercy of this thing in the first place, and putting alternatives in place.

but during the crisis i'll just go do something else for an hour and then read the post mortem when it comes out.

Updates that are communicated are not the details of what's happening in the investigation, but things that are expected to be useful to users/clients. They are things such as the estimated time for problem resolution, updates on the scope of the problem (e.g. "this is an instrumentation problem" vs "this is an actual outage") or mitigation steps that could be applied by the clients. It is often very useful to know such things during an outage.

i don't think anyone, anywhere would have given you an accurate estimated figure of nearly 5 hours to fix this problem.

furthermore, even if you somehow could divine the future, telling the customer that you think an outage will last over half a working day is going to turn an extremely shitty situation into something even worse.

sometimes the resources available are suitable for different problems.

I consider high quality communicators to be a significant resource, and the problem to solve is to make sure everyone affected has the right information to make their next decision to mitigate the issue.

These communicators might not know how to solve the actual technical problem, and it'd be like putting monkeys on a typewriter to tell them to "all hands in fixing a configuration issue with the web server" for example.

I'd say apply the right people to the right problems.

Keeping the affected users informed is part of 'fixing the problem'.

Sometimes the problem is even something that some users will be able to work around if they know what it is.

I agree with you and we have a similar process at Docker. Part of what went wrong in this particular case is precisely that our infrastructure incident response process did not kick in. I am guessing that one conclusion of the post-mortem will be "make sure to handle open-source package distribution infrastructure like the rest of our infrastructure, including incident response checklist".

I think this is fine. If you use a 3rd party repository, you can almost expect it to fail. If you rely on repositories so heavily, run Apt Cacher or mirror them on s3 or the like, its pretty simple. This is a pretty common practice in any professional environment to contain all your dependencies - binaries, debs etc locally mirrored... Can't wait for dockerhub to go down now... :troll:

Thanks Solomon for the update and for bringing your wonderful product into the world!

Reply on news.ycombinator.com