Docker Hub Maintenance | Hacker News Reader

Docker Hub Maintenance

success.docker.com

42 pointstlwaddington18 comments

"Given the nature of this update and the need to ensure the highest level of security, we have provided limited advanced notice. We understand that the maintenance process impacts our customers, and we apologize in advance for any inconvenience this may cause." -- This is a ridiculous statement. Security does not come from opaque statements made at the last moment.

I fail to understand how updating a "cloud native" service requires downtime at all, much less this sort of outage.

>I fail to understand how updating a "cloud native" service requires downtime at all, much less this sort of outage.

Probably something to do with DB consistency.

Migration possibly that’s locking auth tables?

Translation: In the aftermath of the hack last week they found additional major security issues that require downtime to fix. They are limiting the advanced notice to give attackers less time to try to find them before they can be fixed.

Quoting from the page:

"Q: Is Docker experiencing a security incident?

A: No, this is a scheduled update and a proactive step we are taking to provide the best possible customer experience and highest level of security."

scheduled when...

We just scheduled it right now.

I don't think they're saying it's more secure because they didn't tell anyone.

In a prior job, we discovered some automation left ports open that could theoretically be accessed by customers, thus a malicious actor could theoretically have loaded software onto a privileged instance. So we had to treat all devices as potentially compromised. That meant running automation to tear them down and rebuild. This took days.

We were able to do this without visible outage, but the devices affected were mostly doing NAT; if you have a more central component affected, you do need downtime to back it up (that's the read-only section, don't let customers write data that doesn't get backed up) and then rebuild and put it all back together.

Also, it's just easier to get it right with downtime. And they're probably padding these estimates to handle the inevitable things that go wrong during maintenance.

24 hours notice for up to 8 hours of downtime (“read only”) seems quite abrupt.

Apparently security-related.

"Given the nature of this update and the need to ensure the highest level of security, we have provided limited advanced notice."

I suppose my point was more along the lines of “what else could there possibly be to justify this that you haven’t screwed up already?”

I guess come Sunday/Monday we’ll see...

They nearly-erased the page in the last few minutes..

Here's the original from time of submission: https://web.archive.org/web/20190503141604/https://success.d...

Now as of my comment the entire body of the page is:

Docker Hub Maintenance - May 2019

In early May, 2019, Docker Inc. will perform maintenace on the Docker Hub.

This article details actions to be taken, timeline, and any current status/issues/recommendations.

I guess this weekend I'll be setting up my own registry. Gives me an opportunity to do something I've been meaning to do for awhile anyway.

Setting up a registry is easy and actually pretty fun. You can even run it in a docker container (https://hub.docker.com/_/registry)! The complicated/non-trivial part is getting a front-end set up if you want more than the command line. I would recommend most people at least keep backups of their important images in a private registry. It really doesn't take much to keep one running, and you're covered if Docker Hub has an outage.

For most people's needs, command line access to the registry is more than enough though. There are a couple of options out there already for a registry web UI, but I've never had the chance to use any.

i actually open sourced a tool kbld (https://get-kbld.io) a couple of weeks ago that includes export/import images functionality (keeps same digests across registries): https://github.com/k14s/kbld/blob/master/docs/packaging.md. didn't think of it as a image backup tool till i read your comment.

If this is related to security, I think users deserve to know what's up.

As someone working in security, I'm fairly distraught that I still don't know exactly what happened last week. The Docker post-mortem[1] states:

> On Thursday, April 25th, 2019, we discovered unauthorized access to a single Docker Hub database storing a subset of non-financial user data. Upon discovery, we acted quickly to intervene and secure the site.

> During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as GitHub and Bitbucket tokens for Docker autobuilds.

> There was a brief period of unauthorized access to a Docker Hub database. During this time some sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of users as well as GitHub and Bitbucket tokens for Docker autobuilds. All these tokens have been revoked.

To the best of my knowledge, the above excerpts are the entirety of information about the incident that Docker has officially released. This is not nearly the level of detail that any security-conscious customer cares about, and in my opinion, it's an egregious violation of trust to not give more information.

For those that may not be working in security day-in, day-out, here's what I mean:

If this incident was, at its core, a DockerHub error that exposed a database to the Internet without proper authentication in place -- and then someone stumbled upon it -- that's an embarrassing mistake, but these things happen. I'd feel comfortable that Docker understands the extent of the breach, and that we can continue to use these products with some degree of confidence.

If, however, the breach was -- and we're going to the other end of the spectrum here -- a state-backed APT that blew 0days to breach the DockerHub network, then the threat model is significantly different. Did Docker bring in incident response professionals? What were the attackers targeting? How confident can we be that they didn't pivot to somewhere else on the Docker infrastructure undetected?

I know that Docker is likely trying to save face, and that the former scenario is significantly more likely than the later -- but guessing about whether or not the breach was severe is a ridiculous situation to be in for major organizations that use the DockerHub service.

In case anyone was wondering, I wasn't personally or professionally impacted by the breach, but we performed full credential rotation anyway. If the breach was more severe than described (e.g., persistent access was established), then that probably wouldn't do much good... but it's better than just assuming that "only 5% of users could have been impacted," and doing nothing.

I'm very unhappy with Docker's communication of this breach/misconfiguration/incident/whatever it may have actually been. I really hope they release more information (perhaps in the form of a real post-mortem) so that DockerHub users can better understand the risks of using Docker products. In my opinion, it's incredibly irresponsible of Docker to keep this information to themselves.

1: https://success.docker.com/article/docker-hub-user-notificat...

A bit of a tangent: as someone studying cybersecurity at a university in NYC and dealing with a range of topics: virtualization, forensics, incident response, defense, offense, CTFs, bounty programs etc. etc. And at the pace I am going at it (evening classes), it will take a few years to finish, and even then I might not start in the security industry right away.

I would like to ask, in your opinion, what is a good way to stay up to date, simulating as close as possible to real-world experience through hands-on self-teaching methods ? And what areas would you concentrate on personally, considering how vast the field is.

I honestly think that following current events -- whether that means technical news via HN, or more mainstream events covered by traditional media -- is the best thing you can do to stay up-to-date on what people are thinking about. Take analysis with a grain of salt. Look at breaches and ask yourself how they could have been prevented (or perpetrated!)

Attend community events and conferences. There are local "Bsides" conferences in most cities, and the big conferences (DEF CON, Black Hat, ShmooCon, etc.) are great too. I haven't been to Summercon or Hushcon East, but I've heard that they're both great NYC conferences. When you're at these things, don't just go to the talks (you can see them on YouTube later), but talk to people in the hallways/bars/parties and find out what kind of challenges they're facing.

Attend local meetups. See what people are talking about. Be involved with the community.

If you do these things, I can guarantee you'll stay current -- and hopefully, it will keep your interest in security going strong :)

Reply on news.ycombinator.com