Critical security updates for Git, Subversion and Mercurial | Hacker News Reader

Critical security updates for Git, Subversion and Mercurial

marc.info

366 pointsmnw21cam125 comments

It's interesting to me that the fix was pattern matching the ssh hostname and banning a starting hyphen, rather than (say) passing "--" to ssh to signal the end of the intentional options so a hostname of "-oProxyCommand=whatever" is interpreted by ssh properly (as a hostname which can't be reached, instead of as a rogue argument).

I thought this was a fairly well known way to pass arbitrary strings to commands and ensure they aren't interpreted as options (for commands which honor "--", like ssh does).

We discussed that, but it wasn't clear that doing so was portable. It works for OpenSSH. It doesn't for PuTTY. We don't know what other implementations people might have as `ssh` on their systems.

It seems a little risky to assume separate clients/platforms will always interpret command line arguments the same way. At some point when a client breaks compatibility you have to implement methods to support the one-offs, or drop support. So for portability's sake, why not start writing those methods now, and have a generic universal method to fall back on for untested clients/versions?

Or is that really unnecessary?

How would a program like git figure out what flavor of "ssh" program is on the system? Keep in mind that it may change while a long-running git command is running.

Dropbear[0] doesn't seem to support "--" either.

[0] https://matt.ucc.asn.au/dropbear/dropbear.html

What happens if you do the -- fix and then use PuTTY? If it just fails then I'd argue that's OK, PuTTY will get updated as a result of the failure, no?

Why do you think Lennart Poettering is hated so much?

+100000

Someone needs to teach the next generation about ABI compatibility...

Ordinarily I'd agree, but a starting hyphen isn't valid in a domain name anyway, so it's harmless. RFC 1035: "The labels [a grammar production rule for domain names] must follow the rules for ARPANET host names. They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen. There are also some restrictions on the length. Labels must be 63 characters or less."

(And by the way, neither is a leading dot, which I use to my advantage to define hostname aliases in ~/.ssh/config which I can be sure will never clash with anything.)

> They must start with a letter

By the RFCs, that's true; but if you think it's true in practice, then 4chan.org would like a word with you.

Right... but the no-starting-hyphen rule seems to be more universally accepted.

...

...Oh, fuck me. I just found https://archive.fo/T0WcV ... (archive.is link because Firefox won't open it directly)

I think it's not so much Firefox as your OS - if you're on a GNU platform, glibc will refuse to resolve a hostname that starts with a hyphen (it won't even attempt the resolution), but macOS's libc resolver is totally fine with it. I forget what Windows does.

Chrome and Firefox on Windows 10 both open http://-emmawatson.tumblr.com/ without trouble.

My chrome will not open that website (gives a DNS resolution error).

Chrome: 61.0.3163.31 (Official Build) beta (64-bit) Windows: Microsoft Windows 10 Pro 10.0.15063 Build 15063

My DNS is pointed at my router which is forwarding requests to Google's 8.8.8.8 DNS servers.

Doesn't work on chrome for Android, DNS error.

Decade old glibc issue:

gethostbyname() cant resolve names starting/ending with "-"

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/144431

I ran across this a couple years ago because Tumblr let you use a leading - for your subdomain... I remember being so shocked reading this thread, but clearly nobody on that discussion had heard of Tumblr at the time. Very surreal.

Huh? I just opened it in Firefox Nightly...

https://tools.ietf.org/html/rfc1123#page-13

      The syntax of a legal Internet host name was specified in RFC-952
      [DNS:4].  One aspect of host name syntax is hereby changed: the
      restriction on the first character is relaxed to allow either a
      letter or a digit.  Host software MUST support this more liberal
      syntax.

Hmm.. that's from 1989. 3Com would have still been a fairly respected network hardware vendor.

I have a strange feeling '3com.com' would have been a consideration when this change was proposed.

doesn't say it must not accept anything other than a letter or a digit and so, to cover future leniency changes, i would not be surprised to see poetic-license taken such that they allow damn near anything in an unreliable way.

Kudos to Chocolatey on Windows, they immediately updated their Git package [0] to v2.14.1, so a simple `choco upgrade -y git` gets me up to date. If only life on Windows had always been this hassle free.

[0]: https://chocolatey.org/packages/git

Getting off topic fast, but I always found Chocolatey to be needlessly complicated and messy.

I was very happy when I found http://scoop.sh. It's simpler and slightly less powerful, and most notably it just runs official installers in the background instead of copying binaries over from the source and packing them up in a special way.

(It's also at the latest got already)

>Getting off topic fast, but I always found Chocolatey to be needlessly complicated and messy.

I haven't released it yet, but Napkin solves this very issue.

Would you mind elaborating?

I really hope it's just a joke about overly opaque project naming, possibly riffing on the need for a napkin after a "Chocolatey" "Scoop" [of ice cream].

oh damn you're right, I didn't catch the humor. Thanks ~

Thanks for getting off-topic. These seem useful... until WSL becomes more mainstream.

Part of my workflow involves using ncat.exe to use git over a socks/ssh proxy. A real pain. I noticed scoop has nc but not ncat.

I find `bash -c blablabla` works great for invoking the occasional Linux commands without moving to WSL entirely.

(Actually I used to run all devtools, eg git and webpack I etc etc on WSL entirely but it still has too many quirks. Moved back to plain Windows recently and I'm happier)

Being on MS-DOS since 3.3, and being introduced to UNIX via Xenix, I have found out through the years that the best way is just to use the native CLI of each platform.

For stuff that needs to run everywhere and I don't feel like writing two versions of scripts, I just use Python.

Maybe I needed to configure something, but I tried Chocolatey a little while ago on a fresh install of Windows and it seemed to take forever to do anything. Maybe I'm just spoiled by how quickly apt update and apt upgrade go, but it would hang for so long that I thought chocolatey had frozen.

antivirus?

I don't think so. All I've got is Windows Defender and seeing as how that's a default program in 8 and 10, I'd expect a little more effort on Chocolatey's part to notifier users about that if it was the cause.

I don't really get it. I did a fresh install of it a few minutes ago, gave the command to install 7zip. Then it downloaded the file, said it was installing, and it just sat there for 10-15 minutes before I got fed up and hit CTRL+C to cancel it. Then it kicked back into gear and magically finished the install somehow?

Ditto for brew - I updated to 2.14.1 from there this morning.

How did you do this ? I just did a brew update && brew upgrade and still have git 2.13.0. I'm not an experienced mac user. EDIT: I followed directives given [here](https://medium.com/@katopz/how-to-upgrade-git-ff00ea12be18). I had git installed in /usr/local. I had to use the commands brew install git and brew link --overwrite git. I don't know if this was the correct way to do it.

I really wish Chocolatey was more officially adopted by packagers. Often packages on there are outdated or just plain don't work... but, when they do work, it radically changes how software is installed on Windows.

I tried Chocolatey a while ago. Does it still install things all over the place by default? I with it could have some sort of install prefix, I'd use it then.

Edit: it seems to have one.

Yes, you need to know the command line switch for each package/installer separately to install in the proper place.

They have a switch to configure it once and have their tool figure out the right argument for each installer by itself, but it's only for paid users (~$96/year)

This is one benefit of scoop.sh - it's designed to isolate programs under your user profile, and not mess things up. A bit like using stow/xstow on gnu/bsd.

Does this vulnerability affect Windows? I don't even think it has an `ssh:` protocol handler.

Natively Windows doesn't come with ssh no, but last I used it Git For Windows bundles a bunch of MinGW components, including ssh, which it calls out to.

Yep, I'm using that ssh too, it's really nice to have native ssh in Windows.

Git for Windows also ships with ssh-pageant that can use Pageant as key agent. Really useful if someone is using smartcard certificates (PIV, not GPG based).

Chocolatey is one of the best thinks that happened to Windows recently. It just makes my life so much simpler.

wow, chocolatey.org looks awesome. Thanks for the link!

Report from the person who discovered the vulnerability: http://blog.recurity-labs.com/2017-08-10/scm-vulns

Just a note for everyone on Ubuntu, the fixed version for Ubuntu 16.04 is git v2.7.4-0ubuntu1.2 [0].

[0]: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2...

Does anybody understand Ubuntu releases?

The update this morning for Ubuntu 17.04 installs git 2.11.0

.. NOT the patched 2.11.3 NOR the latest 2.14.1

Why isn't Ubuntu fully up to date?

Almost all linux distros that use the concept of a "release" will stay on the version numbers they originally released for. Security fixes are backported but no other functionality is imported, thereby keeping the feature set fixed to the version numbers in the original release.

If you are maintaining or developing any kind of server and services where you need an uptime guarantee above 0%, you will appreciate this. Otherwise you will need staff on standby 24/7 being ready to develop fixes for breaking backwards compatibility changes. Imagine if you had to deal with funky errors like these every minute of the day - https://gitlab.com/gitlab-org/gitlab-ce/issues/36028

For Ubuntu, you should follow their "usn" security notices and you would stay informed: https://usn.ubuntu.com/usn/

For Debian, likewise there is a "debian-security-announce" mailing list https://lists.debian.org/debian-security-announce/

Also for Debian, you can check up on the status for known vulnerabilities on their security tracker. For example, for this particular git exploit, you could look up https://security-tracker.debian.org/tracker/CVE-2017-8386

They backport patches and lie about the version number, to maintain the illusion of stability. It really pisses off the PHP community.

https://github.com/ircmaxell/password_compat/issues/90

I can't speak for Ubuntu, but when I was FreeBSD Security Officer we would regularly backport patches because importing an entire new release would regularly break existing functionality or even add new security vulnerabilities. It annoyed the heck out of vulnerability scanning tools, but I decided that giving users a system which didn't randomly break when they applied security patches was far more important.

That was years ago, IIRC.

PHP has gotten better about "no BC breaks in patch versions" over the years, but the Debian/Ubuntu teams still insist on making people effectively run e.g. 7.1.8 while the version indicator says 7.1.1.

It makes feature detection a nightmare.

Ubuntu developer here. Huh? No, we don't. Can you provide a more specific and current example to make sure I don't misunderstand you?

For example, the current package for Ubuntu 16.04 LTS is 7.0.22-0ubuntu0.16.04.1. How does this mismatch 7.0.22 from the PHP upstream project?

Okay, maybe Ubuntu stopped being stupid and only Debian is still guilty of this?

This used to be a huge problem. I've since convinced [most PHP devs I talk to] to stop using distro-provided packages, in favor of deb.sury.org.

Ubuntu and Debian's PHP packages are largely the same.

> I've since convinced [most PHP devs I talk to] to stop using distro-provided packages, in favor of deb.sury.org.

You do realise that Ondřej Surý (of sury.org) is the primary Debian PHP maintainer? The point of his repositories (AIUI) is to allow users to mix and match PHP versions. The downside is that he's one person with (AFAIK) a bus factor of 1 when it comes to security updates. That's irresponsible to use in production.

In contrast, Debian and Ubuntu's PHP packages, essentially provided by the same person, has in addition teams (both in Debian and Ubuntu) who can pitch in when required.

AFAIK neither Debian nor Ubuntu lie about version numbers. They have stable versions that they backport security patches to, but they don't change the version number. That issue you link to doesn't contradict this.

The version is actually 2.11.0-2ubuntu0.2, with the fix backported, as can be seen in the changelog:

http://changelogs.ubuntu.com/changelogs/pool/main/g/git/git_...

It's somewhat confusing if you run `git --version` and it reports back 2.11.0, despite being fully patched against this vulnerability.

Thanks for the info!

I've added the git-core PPA now which installs 2.14.1 .

I just don't understand this screwing with version numbers and not being on the latest code.

The point of having stables releases is that you have a consistent set of versions of packages including git, otherwise how is it stable?

Yes, it makes a lot of sense to ignore the real world and pretend that software will always be stable.

Why don't you just run Debian Sid then? You'll get new packages every day and get all the fun of chasing changing features all the time :D

Came here to ask the same thing, running 17.04. I just did an update && dist-upgrade, still have git-2.11.0.

Edit: The package is actually 2.11.0-2 which contains the security patch.

This question is brought up every time this happens. Ubuntu uses stable (old) versions and does not upgrade to latest (unstable) releases very often. However, they do back port and apply security patches to these old versions.

Point in case: 16.04 TLS is still on 2.7.4 but it is fully patched and secure.

Yes, you're right. I was expecting to see the update by checking "git --version" and seeing 2.11.0-1 or something like that, but it's not visible that way.

Indeed if I check with "apt-get show git", the package is on version 2.11.0-2, and then I have to browse to the package web page at https://packages.ubuntu.com/zesty/git and click on the changelog and finally I get to the update information, which clearly contains the text, "SECURITY UPDATE: Arbitrary code execution on clients through malicious ssh URLs."

So it was patched as expected, it just wasn't easy for me to see that without going through a few extra steps.

Try "apt changelog git" on the box.

Or you can read the changelog on the filesystem at...

/usr/share/doc/git/changelog.Debian.gz

This practice is seriously bad.

It means that one group of developers is busy improving and fixing packages, and another different group is cherry-picking commits in order to maintain the illusion of stability (as mentioned above).

That second group has to duplicate all the testing work done by the first group, and additionally ensure that there are no new problems introduced.

It's very vulnerable to human error and adds a lot of unnecessary work.

It's better to have a competent "second group" do the backport job rather than every single user of a distro having their rug pulled out under them every time they patch. New versions of software can quickly break your environment. For example, https://gitlab.com/gitlab-org/gitlab-ce/issues/36028 . If you have to re-test every script and web service to ensure compatibility with the daily churn of vulnerability fixes you would get nothing done and you could almost never be sure your service is stable.

I agree, this is a problem, but it's not being solved by this practice.

The example you gave is specifically about a client application of git that wasn't updated to work with the current version of git - probably because of some twisted logic that they didn't have to worry about it until some future date.

It's an example of how things break if you upgrade to new versions of software. If you avoid going to new versions of software, and instead just backport the essential security fixes, there will be much fewer surprises like this.

Imagine you had a daily backup script that uses git to commit and push to a remote server perhaps managed by someone else. You then upgrade to the latest and greatest version of git to try to patch this vulnerability, and surprise, your backup script fails! Hope you weren't planning on spending the rest of the day getting the guys running the git server to implement a workaround, who might have already left for the weekend, leaving you with no backups or having to manually try to fix your git to work with that server again.

Well then, I guess with that line of reasoning we should just turn off the computers and go live in the woods. There's obviously too much risk in doing anything.

That's up to you. I find the current system of stable releases to work just right.

Yeah, well, that's just, like, your opinion, man.

I see it differently. Where do new bugs and vulnerabilities come from? When the main developers add features or make changes to existing features that go beyond fixing bugs.

From the point of view of many server administrators, using the latest versions of everything is inherently risky. What they want to use is a stable, solid version that has all the latest security fixes.

It's unlikely that these opposing viewpoints will ever be reconciled.

It's important for package developers to be aware of other software that depends on their interfaces or functionality.

Some cases will slip through sometimes, but over a couple of releases these should be gone.

>> Where do new bugs and vulnerabilities come from? When the main developers add features or make changes to existing features that go beyond fixing bugs.

Do you have stats for that?

Semantic versioning was supposed to be the fix for that, but as Rich Hickey has pointed out, that is also broken.

Everyone is their own server admin these days. We all want "a stable, solid version that has all the latest security fixes" but it's difficult accept that that might be impossible.

No stats needed. Just go through the bug list for any software. How many of the bugs were introduced by the people who maintain stable packages?

I agree with this sentiment a lot of the time but consider this:

https://bugs.kde.org/show_bug.cgi?id=382911

it is a duplicate of

https://bugs.kde.org/show_bug.cgi?id=378854

which is fixed by https://cgit.kde.org/konversation.git/commit/?id=783dc0f595e...

but upstream for some reason does not think this is important enough to tag and bag into a release.

as much hate as Fedora gets, rdieter has taken the time to backport the changes that we need in Fedora.

https://bodhi.fedoraproject.org/updates/FEDORA-2017-9c3e2138...

Ideally, we'd rather see upstream release this fix but we've all read https://news.ycombinator.com/item?id=14051106 and I don't want to be too impatient with software that upstream has graciously provided for free (in both senses of the word!)

I guess my point is backporting might make sense if there are small changes we can make to enhance release. However, I agree that there is too much duplication of effort going on.

I've seen that bug!

And, I'm not too surprised, KDE has a broken development and maintenance process. I'm sure they think they're doing a great job - but that's because they don't even see or receive most bug reports.

> And, I'm not too surprised, KDE has a broken development and maintenance process. I'm sure they think they're doing a great job - but that's because they don't even see or receive most bug reports.

KDE actually fixed the bug but it hasn't gone to a tagged release after two months. Maybe they think this small change doesn't warrant a release? I asked if KDE neon can pick it up but they (understandably) seem to see this as feature creep. KDE applications means something different for the neon folks (only the things they ship with neon from what I understand).

I agree. I have no doubt we are all trying to do our best. =)

You're welcome to use Debian testing, or the development release of Ubuntu. Both update packages as they are released from upstream developers.

Thanks, but that means I'm doing the testing. I wish I could do that - maybe someday.

I don't follow. What do you want exactly? The newest stuff pushed straight to users, or not?

A "ssh://..." URL can result in a "ssh" command line with a hostname that begins with a dash "-", which would cause the "ssh" command to instead (mis)treat it as an option.

It's a shame, because the Git dispatching code ought to be able to invoke the ssh command via

  ssh -p 22 -etc -etc -- <hostname>

to prevent interpreting options in <hostname>, thus defusing the in-band signalling causing this. But I suppose it can't depend on every ssh implementation understanding this "--" POSIX utility syntax guideline.

> it can't depend on every ssh implementation understanding this

Indeed, from a comment[1] above:

> We discussed that, but it wasn't clear that doing so was portable. It works for OpenSSH. It doesn't for PuTTY. We don't know what other implementations people might have as `ssh` on their systems.

[1]https://news.ycombinator.com/item?id=14989578

How is this any different than a SQL-injection attack?

Seems to me like git shouldn't be using the shell in order to invoke ssh at all.

I haven't checked git, but even if git uses exec*() directly and skips the shell (which I suspect it does), the problem still exists.

The issue isn't one of shell quoting, but one of ssh treating the argument list ["ssh", "-oProxyCommand=something", "remote-cmd"] not as ["ssh", hostname, command] but as ["ssh", option, hostname].

There are C libraries for SSH, like libssh, that would avoid these type of vulnerabilities.

https://www.libssh.org/

One downside of libraries like libssh is that they don't behave the same way as your regular ssh command. So thing you've configured like host aliases, proxy commands, etc, don't just work out of the box (and in some cases may not even be supported at all).

True. It sounds like there is room for an ORM-like sanitizer for SSH that is responsible for translating a C API to sanitized commands. I have no clue what to search to find if that already exists.

It almost seems like ad-hoc text formats like command line argument syntaxes are difficult to use correctly.

libgit2 depends on libssh2 actually.

Exactly. It's an option injection attack. There's no shell involved.

Got it.

Seems to me the the use of "=" notation for CLI args would be rife with situations like this.

Yikes...

[also, still going through the code, but it looks like they are setting up for a call to exec*()]

Yes, which is why "--" is the standard way to avoid this stuff, and why people should generally avoid writing their own options processor and instead use getopt() or getopt_long(), which provide consistent, standard semantics.

In this case, it's not the shell that's the problem, it's ssh's command line argument parsing that you need to be careful with.

Not just SSH's library parsing, but every SSH implementation's argument parsing.

Wow, git's url bugs always seem to become easily exploitable due to .gitmodules.

I found CVE-2015-7545 a few years ago, a malicious URL using the ext:: scheme could cause code execution. It was only easily exploitable because you can ask the client to fetch any URL you want via git submodules. (This vulneriblity was fixed, and since then the entire ext url scheme was disabled by default.)

This may be of interest to people using git on Windows and have malwarebytes installed.

I tried to install a new version and found I could not as there was another version that was present. Git did not appear anywhere as a program to uninstall. I tried to delete it (from an admin account) and it failed with an access denied - and no other information.

The solution was to use LockHunter (https://lockhunter.com/) which informed me that malwarebytes was the program preventing me from deleting it. Using LockHunter, I removed the lock and successfully removed the old version of git.

At the time of writing this, no official binary release for Mac[0], you have to build from source.

[0]: https://git-scm.com/download/mac

Looks like it's already available through homebrew, if you use that.

Confirm, homebrew correctly updated & upgraded to git 2.14.1

  brew update && brew upgrade

Or if you are on a fresh install of OSX or don't have git installed

    brew update && brew install git

If you just want to upgrade git, brew update && brew upgrade git

Where's the communication from Git itself about this? When I Google "critical git security update" I see this item, and stories from a few other outlets, but no notification from Git.

Their release note for this release just says "This release forward-ports the fix for 'ssh://...' URL from Git v2.7.6".

Nodejs has a page where they outline their procedure (https://nodejs.org/en/security/), with a list you can subscribe to. Does Git have something like that and I'm just missing it?

> * In the same spirit, a repository name that begins with a dash "-" is also forbidden now.

This will negatively affect https://github.com/nasser/---

Arch Linux has the new package in testing: https://www.archlinux.org/packages/testing/i686/git/

Fix has landed for Debian Jessie/Stretch in the security repo

https://security-tracker.debian.org/tracker/CVE-2017-1000117

A lot of the time people use git to get source code to build and run on their computers. You can probably hide a nefarious code in many places, where it would be less noticeable, too, like pre-generated configure/Makefile.in scripts. I don't think anyone reads those. Another reason to welcome those new simpler configure/build systems like meson/ninja.

This was already submitted earlier today (https://news.ycombinator.com/item?id=14984044).

It's actually quite easy to reproduce (the RCE aspect comes from the ProxyCommand and LocalCommand SSH options that you can set from the SSH command-line).

For MacOS users simply use: `brew install git` Then restart terminal instance.

on Ubuntu,

use this PPA https://launchpad.net/~git-core/+archive/ubuntu/ppa

why PPA? the official release is already patched

Thanks so much for pointing to this.

Is there any site where as security vulnerabilities are disclosed, you can see each what platform/contexts are affected, which have updated, and how to do that update?

Couple that with an API and notification system and you've got yourself a good service.

My hands are a little full at this moment but feel free to shoot me an email if you're interested in doing something I like this. Seems like a good case for automation.

Maybe AppCanary? https://appcanary.com

I wonder how can I do it on OSX without brew or port?

The next security release by Apple may include it. It's not uncommon to see open source library/utility updates in their change logs.

eg: https://support.apple.com/en-us/HT207922

I wonder how many "git client libraries" in various languages are also impacted by this.

I'm not familiar with any that exec ssh. Certainly libgit2 does not, nor do the language bindings on top of it. So libgit2, LibGit2Sharp, Rugged, Jagged, NodeGit are unaffected.

FYI, Cloudron has pushed out updates for Gitlab, Gogs (from their slack).

How can I do it on OSX without brew or port?

I'm not sure where you got your git from (did it come with macos?) but I'd highly recommend you use something like brew.

Not just for git, but for any tools you use honestly. MacOS very rarely patches this kind of thing, and in situations like this you need to be able to move fast.

> MacOS very rarely patches this kind of thing

I think it's more accurate to say that Apple often takes their time patching these vulnerabilities. CVE‑2016‑2324 and CVE‑2016‑2315 were announced in March 2016 but Apple didn't release a patched version with Xcode until May 2016. Are there any major security vulnerabilities in git Apple hasn't yet released a patch for, other than this one?

But yeah, Homebrew gets the patches out much faster. It's probably the right move to prefer using Homebrew's git to system git.

Compile manually from source?

Reply on news.ycombinator.com