Hacker News Reader

Comment by m_b | Hacker News Reader

For a bit of context: Matrix.org infrastructure has been hacked a second time in 24h, after restoring everything they went down again, story developing here: https://twitter.com/matrixdotorg/status/1116304867683905537

View on HN

The hacker is now doing a post-mortem in the GitHub issues of the project: https://github.com/matrix-org/matrix.org/issues

This is gold...

> I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out.

> There I was, just going about my business, looking for ways I could get higher levels of access and explore your network more, when I stumbled across GPG keys that were used for signing your debian packages. It gave me many nefarious ideas. I would recommend that you don't keep any signing keys on production hosts, and instead do all of your signing in a secure environment.

Why would they do this? It's pure negligence. I don't even sign anything important and still worry about my keys.

I have been asked twice or more why I insisted on not using a Continuous Integration environment for publishing some software releases that are installed by third-parties.

My team was automating the infrastructure to build internal software and naturally they wanted to be able to simplify things.

The idea that was proposed to me was the following: once I push a new version tag to GitHub, the deployment CI server is going to build and release it as an unstable version.

Some important detail here: I use the same key to sign packages regardless if they are released as unstable or stable. That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub, hypothetically they would be able to eventually gain access to consumers machines (basically, developers) when the consumers update it after getting a notification telling them a new version is available. No way I'd allow this to happen, but I would not be surprised if most people just took this as an acceptable risk.

Depending on your threat model I think that signing packages directly from your CI is acceptable, assuming that your CI runs is a reasonably isolated environment (e.g. on your company's LAN) and people who are able to trigger a release are correctly vetted.

If I understand the parent comment correctly they were somehow shipping the release signing key on their production environment which is a whole other level of bad.

> That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub

You have to define what the signature means.

IMHO it is fine for it to mean "this software was built on our build server from a well-defined state of the source code, which is only changable by our employees and contractors, and for which we have the full change log". So I deploy the code signing key to build servers, which is the only place where it is used.

I'm interested in what alternative meaning you would give to a signature. I have considered the possibility of tying it to the QA processes, but then a build can only be signed after checking it manually, which is problematic when many signatures are needed at multiple packaging layers (exe/dll, msi, setup.exe).

The problem is when a malicious package is produced, either because a flaw was introduced in the code, or because a dev machine was compromised, or _when the CI machine sad compromised_; the malicious package will be signed as if it were legit.

One middle point between automated and manual signing is, as usual, key rotation: have the signing keys expire in a short duration of time (say 2 weeks) and manually push them every week, so that the window of attack is as small as possible.

What does a key rotation solve? Either your build server is compromised or it's not.

You add another stage.

1. A release candidate X is tested in a CI

2. If tests pass, the CI sends a notification to the build server. The notification is "prep for release package <hashid>"

3. Build server pulls code from the repo, matches it against "ready, CI passed" notification and builds the package/packages.

Compromise of the entire CI/dev chain would be contained as the builders act as a new pipeline entry point running in parallel of the CI using pull method. To compromise keys located on a build server one would need to either get access to it via whatever the method of remote access the server has ( which should be nearly none ) or figure out how to compromise the code running on the builders using the input from a repo that passed CI.

Unless you audit the entire codebase prior to a manual build, from a machine you know hasnt been compromised with a key you know hasnt leaked, how is a manual build different to CI/CD securitywise?

Auditing the entire database is hard and is not part of my concerns why I rather not to use a CI/CD in my specific case.

This is what I gain by not using the CI/CD the rest of my team uses:

* isolation: I build applications to be delivered to personal computers of software engineers (mostly), they build applications to run on our own internal servers. * my SSH client requires I have my SSH key with me, while I believe I can achieve something similar with a web-based CI/CD, the client-side certificate isn't something as "production ready" out of the box as 24 years old SSH is. * if someone manages to push malicious code to my code base, I am going to notice during manual check: yes, I manually check the diff commits to see if anything weird came up (mostly thinking about bugs). In practice, I basically check if the commit hash is the same as the one I just pushed (usually containing the release notes). If it is, I build. Otherwise, I check what is going on (most likely, I forgot to checkout the tag).

You can say that this doesn't rule out my machine from being compromised, and I must agree... However, besides being very unlikely that I am a target of such a complex attack, I try to do my best to have a secure development environment.

If I were a high profile target, I would just use a spare safe machine to use in deployments (I believe Linus Torvalds use something like this to vet the security of the Linux kernel but I couldn't find the reference).

Another gem:

RRREEEEEEEE> I noticed you missed a doctype in your html page. In order for web browsers to know what type of html to render you should include a doctype. Thanks!

matrixnotorg> @RRREEEEEEEE Thank you, I will consider that for the next release

Edit: it got deleted

Wait, did Github delete matrixnotorg's profile or did matrixnotorg?

If Github deleted that profile, I don't really see that as being very hacker-friendly.

Although 'hacker' is often used as a positive term on HN, breaking into a company's production server is clearly illegal activity and should not be condoned. If Github deleted the account, they are simply acting in accordance to published TOS & policy.

If the attacker placed sensitive information on Github, that would indeed warrant a deletion of the account. However from what I saw from the archives, the attacker merely published details about Matrix.org's infrastructure and its vulnerabilities. Is that something that's against Github's ToS?

hilarious...

I can only think of the despair I would be in if I was in Aragorn's shoes

Arathorn (Aragorn is son of Arathorn).

Does being a parent prepare you to handle emergency situations like this better?

> Escalation could have been avoided if developers only had the access they absolutely required and did not have root access to all of the servers. I would like to take a moment to thank whichever developer forwarded their agent to Flywheel.

I'd feel so small if I were this developer right now :-|

A couple of his issues appear to have to do with the use of SSH. An Ops-guy whom I worked with had setup a bastion host with ip whitelisting that automatically shut down after 1 hour. He didn't like it as his credo was "if you're using SSH when using a cloud provider you're probably doing something wrong"; meaning to say you should automate and be able to recreate any infra at all times with logs accessible without the need for SSH. I never forgot that.

Did you work with Rich Adams?

https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-...

no, but I wouldn't be surprised if my colleague got the idea from him. It was around the same time (end of 2014 I think).

I have just checked quickly the comments and post-mortem, and I start wondering - it seems that the attack itself would be not really possible if Matrix would not be open source (as this would restrict access to the sensitive data)? Is that right?

This is not right at all, the hack was due to an outdated Jenkins instance and could have happened regardless of what other software was running on the infrastructure.

Absolutely not. There's a zero reason for development infrastructure ( which includes Jenkins ) to have any connection to production outside a well defined "transfer this tarball" to a deploy staging server path during a known deploy window which cannot be controlled via development credentials.

They closed the threads and deleted all the comments, but luckily the issues page was archived beforehand:

https://web.archive.org/web/20190412090123/https://github.co...

Luckily yeah. So much wisdom in those comments. /s

I am highly skeptical when people taking about "rebuilding [the whole] infrastructure" in a few hours. Even more so when restoring all data from breached systems and before a thorough incident analysis. Show me the org which can just pull that off.

It's quite common in enterprise to have a RTO of a few hours, and RPO of a few minutes, even for infrastructure with terrabytes of data. Of course, many moneys are paid for being able to do that.

This is doable with proper IaC implementation, and if your org does not have RPO/RTO on lock they're doing it wrong.

Events like Matrix experienced now do not lead to panicked frenzy when this is in place.

It is certainly doable, but I doubt that most people have IaC which is complete, reproducible and tested enough. And the data migration from the breached host still means some risk.

Reply on news.ycombinator.com