Security vulnerability in MySQL ubuntu | Hacker News Reader

Security vulnerability in MySQL ubuntu

seclists.org

125 pointsmjschultz109 comments

This is a vulnerability in the authentication scheme used in the MySQL wire protocol, meaning attackers need to be able to connect to your MySQL database directly to exploit it. Attackers should never, ever be able to connect directly to your MySQL database directly. If you can connect to your MySQL instance directly from your Macbook in your living room, fix it right now.

While your advice may be prudent, that does not excuse poor security.

―Attackers should never, ever be able to connect directly to your MySQL database directly.

mySQLgame[1] demonstrates that domain logic can be successfully implemented within a publicly accessible database[2]. It would be better to reword your statement to:

Minimise the attack surface by preventing unnecessary access

There are times where public access to a database server make perfect sense. It is the reason why database servers implement TLS client certificate verification, Role Based Access Control (RBAC) and other security features which a typical application (with domain logic at the application layer) has no hope of implementing correctly.

[1] http://mysqlgame.com

[2] http://martinfowler.com/articles/dblogic.html

I would generally avoid building applications that assume clients are going to speak directly to the database. We see a couple of them every year (it's a common pattern in enterprise applications) and they tend to be horrorshows.

In any case, the typical web app deployed by HN readers has no business having an exposed MySQL port.

Actually that's exactly what we do with LedgerSMB. The database is the final enforcer of security matters and the authoritative voice in authentication (the web app logs into the db with user-supplied credentials). A lot of traditional middleware functions are pushed into the db (as stored procedures) where these can be reasonably represented as set operations against the database. As an ERP application, this means that this includes just about everything except generating printable invoices or HTML.

This works well. There are some things we are still working on improving, but it's getting there. Moreover it means multiple clients on multiple codebases are possible and we don't have to worry as much about the implications of what happens when someone writes a secondary client to hit the database.

At the same time, we use PostgreSQL, which I have a bit more confidence in than MySQL. Of course I would still suggest limiting access only to those IP address ranges where that is necessary.

Oh god you poor man. We took a look at using SMB Ledger (the parent of LedgerSMB) years back and after peeling back the covers it just looked horrific both from a security and general code quality perspective. This was around the time of the fork, and I couldn't help feel that SMB Ledger had it's work cut out. Hopefully you guys have been able to get through to the other side.

Well, here's the status so far.

1) Going to a db-centric security model has allowed us not to trust the SQL-Ledger code (good thing too).

2) We are refactoring/removing SQL-Ledger code as quickly as possible. As of 1.3 this means payment logic, reconciliation logic, contact management logic and more. 1.4 will hopefully rip out and replace all search and reporting functions. It will take us a few more years to get the codebase where we want it though.

3) The bad thing about bad code is that bad code is contagious. When you spend a lot of your time debugging bad code it is very hard to write good code. Most of what we wrote for 1.3 will need to be rewritten again. I am pretty happy with the code we are writing for 1.4.....

I think we have come through the worst of it. Pace of development is speeding up which is a good sign and much more of the application is subjected to unit tests.

As a side note, when we first added unit tests to the number rounding tests for the code we inherited from SQL-Ledger, there were failures. We replaced that logic very quickly.

Yeah, it was pretty bad... Now its getting better.

Edit: Also it seems to me the worst never seems so bad when you are in it. I don't think that I could see how many problems we had from this until I am here, half way from 1.3 to 1.4, asking why 1.3 took five years to release (we beat Perl 6 and HURD though, I guess Duke Nukem Forever in fact beat us date-wise by a couple months).

Looking back at the customers of mine who have had problems, or projects that went way over budget or took too long because of difficulties here, I can see how much that hurt us. At the time though, it was just one of those keep working on it kind of things.

Either use iptables to lock it down to a particular IP or set of IPs. Or setup ssh tunnelling to expose a remote mysql server as a local port. You should assume mysql is insecure and needs protecting.

You're just swapping one exposed service for another in that case. Ie, OpenSSH instead of MySQL.

On other other hand, I'd trust OpenSSH more than MySQL.

Well, you're reducing the exposed services from two (mysql + ssh), to one (ssh). Which is always a good idea.

Also agreed. I'd trust OpenSSH to do security better than MySQL.

Only if you make the assumption that SSH is already exposed. Which most of the time, it doesn't need to be.

Also: SSH is considerably more battle-tested in this configuration. There's a lot to be said for being aligned with how the developers imagine a program is used

Having OpenSSH running is pretty much essential for any machines you remote administer. Some things you can do: disable password auth, use public keys. Disallow root logins. Listen on a non-standard port. Configure a hardened "jumphost" as your interface between your machines and the outside world.

The counter argument being that if you expose lots of MySQL servers directly to the Internet, you end up with a product much more suitable to being exposed to the Internet.

With this logic, let's not secure any software that you cannot connect to it directly.

Before we forked LedgerSMB, the SQL-Ledger author's attitude was that unless you can exploit the software without logging in it doesn't count. It's only accounting software anyway so why would anyone want to break in? Not only that but timestamps were perfectly acceptable as session id's and they didn't even have to be stored on the server, just checked to see if they were recent.

Thus began years of efforts on our part of security fixes, which I would not have started except that I had customers to support.

What "logic" is this? Where are people getting the idea that I'm instructing them not to patch MySQL?

What is the best way to enable replication without access to the database? ssh tunnel?

Vpn, gre, whatever flavour you like. That's for replication between sites though. Internally there's nothing wrong with direct connection.

Best? Probably an infrastructure VPN across databases. Have you database listen on the MPLS link and limit access to the service based on IP addresses and strong authentication.

Cheapest? SSH tunnel or openvpn point to point link.

"Never" ... You are aware of the existence and mass use of shared web hosting systems right?

Right. By this logic MySQL doesn't need authentication, never mind privileges.

Honestly? In most real-world applications, MySQL authentication and privileges have absolutely no effect on security.

Again, you're completely ignoring shared hosting systems. You have a very strange understanding of what "real-world applications" means when it comes to MySQL.

One term for the applications you're thinking about is "certified pre-owned". Michel Zalewski just made a handy logo for those people to slap on the bottom of their front page:

http://lcamtuf.blogspot.com/2012/06/this-page-is-now-certifi...

"In most real-world applications, MySQL authentication and privileges have absolutely no effect on security."

To me, that makes as much sense as:

"In most real-world applications, file permissions have absolutely no effect on security."

Or:

"In most real-world applications, running services under isolated uids instead of running everything as root has absolutely no effect on security."

It's a bold claim, but one made with no evidence to back it up.

Don't use shared web hosting.

If ever a comment deserved down voting into oblivion, this would be it.

Your advice is for everyone that is providing and using shared web hosting, to stop it?

If you're using it for a real application, I'd move.

Sorry if that makes you feel bad.

I'm not using shared hosting, no. Although I did work for a company that provided it a few years back. I'm just trying to figure out if you're actually being serious with your claim that everyone with a database backed website should be running it from isolated servers? If you actually understand the implications of this or if you're just making nonsensical off the cuff remarks? I wonder how many tens or hundreds of millions of servers that would add to the Internet.

Of course I'm being serious. Don't run serious applications from the shared MySQL databases at shared hosting providers. You seem shocked that I'm saying this, but we work with a lot of very young startups and I have never met one running their app off a Dreamhost-style shared server.

Tens or hundreds of millions of database servers? That's hyperbolic.

Do you really think that we've come close to eliminating all the vulnerabilities inside a MySQL session, post-authentication? Because what you're arguing is effectively that application owners should trust that MySQL is resilient against attackers who can get an authenticated handle to their own database and run nearly arbitrary SQL statements against it. You think all the code in the MySQL query parser, the planner, and the various storage backends have been fully audited? This is a project that didn't even get authentication right.

From a security perspective it's good advice. I've never seen a shared web hosting provider whose boxes couldn't be owned by an account that could drop code and execute. Local priv escalation holes seem to be a dime a dozen and you don't know if they've even attempted to lock down their users' data.

Now, do you really care if some community bulletin board's database gets owned? Probably not. But I wouldn't run a shopping cart on a shared hoster.

"Now, do you really care if some community bulletin board's database gets owned? Probably not. But I wouldn't run a shopping cart on a shared hoster."

I agree. tptacek is the one who disagrees with you. He thinks there is "never" a good reason to use shared hosting.

If you don't really care about security --- and I agree there are times when you shouldn't --- then by all means use the cheapest possible hosting option available to you. But if you care even a little about security, avoid shared hosting.

I really don't understand Mike Cardwell's objection; I don't think what I'm saying is controversial at all. I actually thought I was making a relatively banal point.

I'll just leave the following here. Maybe you can figure out from it what my point was:

You: Attackers should never, ever be able to connect directly to your MySQL database directly

Me: "Never" ... You are aware of the existence and mass use of shared web hosting systems right?

I think you're having trouble with the intended target of the word "you" in my comment. I'm not writing to people hosting Minecraft forums on Dreamhost.

Perhaps you should have been more careful with your wording. It demonstrated a lack of understanding of real world configurations and requirements and implied that if you're doing it that way, you're doing it wrong. I'd guess that most websites live in shared hosting systems.

EDIT: You could have just replied to my original comment agreeing with me that shared hosting systems work that way, and that it's ok for certain types of site. It would have made more sense than your comment "Don't use shared web hosting."

I'm really not sure what you're hoping to have me concede here. If you operate the kind of application that people on HN tend to operate, you should avoid shared hosting. I work with and enjoy talking to people who are serious about running applications, and I provide advice to people who are at least somewhat serious about security.

If you don't fit either of those molds, I don't think any less of you, but I'm not going to tailor my advice to you either.

It really sounds like you're just looking for something to be indignant about. I don't know you or anything about you, so I had no expectation that you were that kind of person. Consider addressing your objections to the thread, instead of aiming them at me, if you'd like to avoid that appearance. For truly, I do not care whether you like shared hosting or your friends are struggling indie shared hosting operators. That's not relevant to me even a little.

A less personal way to frame your objection, rather than "Are you commenting just so you can be downvoted to oblivion", would be to write a comment that starts with the words "There is another side to this that readers should consider..." and go from there.

You've already conceded the point by completely rewriting your claim.

"If you operate the kind of application that people on HN tend to operate, you should avoid shared hosting."

Is a far cry from:

"Attackers should never, ever be able to connect directly to your MySQL database directly"

That comment was about as useful as:

"Attackers should never, ever be able to enter the data center where your servers are hosted"

metasploit module:

https://github.com/rapid7/metasploit-framework/blob/master/m...

I agree with the sentiment, but just to be clear: There are valid reasons to allow MySQL to talk to the wider internet, the most important one being off-site replication.

If you do need your mysql to be exposed to the internet you can use bind_address=x.x.x.x or some kind of firewall.

From the mysql commit: Date: 2012-04-06 09:04:07 UTC

That's two months ago. Looking at the changelog (http://dev.mysql.com/doc/refman/5.1/en/news-5-1-63.html), they piled in a bunch of other changes like "use less disk space". This should have gone out pronto. I feel it's not the kind of thing you sit on until your next quarterly release is scheduled.

[oh wait, this is worse. mysql 5.1.63 was actually released a month ago. But they only now tell us what the security bug was? Meanwhile the bad people have had a month to diff sources? Double unhappy.]

wow.....

What ever happened to public disclosure as soon as a patch is released?

I have been bit twice by problems relating to difficulties getting security fixes/announcements out in time. In my defence I was trying to patch a difficult codebase and this just took a long time. For example, there was a full SQL injection audit that took us two months to complete early on, and there was an issue of XSRF vulnerabilities which could not be effectively patched in a production release.

But waiting a month to announce the security issue after the release was out strikes me as hard to justify.

It looks like this is being tracked in Ubuntu here: https://bugs.launchpad.net/bugs/1011371

Unfortunately Oracle's stewardship of MySQL appears to be a closed model. There is no public access to their bug tracker, and distributions struggle to keep up with security updates because the details of their fixes in the source are not published. The future of MySQL appears to be in one of the MySQL forks.

See https://lists.ubuntu.com/archives/ubuntu-server/2012-Februar... and https://lists.ubuntu.com/archives/ubuntu-server/2012-Februar... for details.

What about http://bugs.mysql.com/?

One-liner: $ for i in `seq 1 512`; do echo 'select @@version;' | mysql -h 127.0.0.1 -u root mysql --password=X 2>/dev/null && break; done

Via HDMOORE on twitter

This is also important in other environments, for instance shared hosting where you may connect to localhost, or places where you may have given non-admin shell access to a developer (assuming they could not connect to mysql root user).

This is a serious vulnerability. Especially since the latest ubuntu seems to be affected(I'm on mint 13, and it is)

See Ready shodanhq query for latest mysql version:

http://www.shodanhq.com/search?q=port%3A3306+5.5.22-0ubuntu1

This vulnerability could also assist with local privilege escalation. If an attacker managers to use a vulnerability in a web application to execute code as a web user account, they can likely access the MySQL server instance via the loopback interface (perhaps a Unix domain socket too?).

Any chance that this vulnerability is linked with the recent hashed password lists disclosure ?

I've been trying this on lots of our customers' boxes and can't exploit it - no matter how many times I've tried I always get turned away when retrying root's password, e.g. trying "while true; do mysql -u root mysql --password=baha; done" does not yield access on any of:

Debian lenny 32-bit 5.0.51a-24+lenny5

Debian lenny 64-bit 5.0.51a-24+lenny5

Debian lenny 64-bit 5.1.51-1-log

Debian squeeze 64-bit 5.1.49-3-log

Debian squeeze 32-bit 5.1.61-0+squeeze1

Debian squeeze 64-bit 5.1.61-0+squeeze1

Ubuntu lucid 64-bit 5.1.62-0ubuntu0.10.04.1

So I'm not inclined to think it's as bad as made out by the simple exploit above.

I would imagine you have to try a different password each time.

Why would you imagine that? The bug is that what password you provide doesn't matter.

That wasn't how I read it.

It sounds like they were casting the result of a memcmp to a char. A char only has a range of -128 to 127. The resulting overflow means that an arbitrary password hash has a 1/255 chance of landing on 0, but you still have to try a bunch to hit one.

This is incorrect. You do not need to try a different password each time.

The password is combined with a random value on each attempt, the hash will change each time.

memcmp will probably return always the same value for the same parameters, which may not trigger the error. Trying different values may yield a value whose mod is 0.

It works exactly as written on my completely up to date 64 bit Ubuntu 12.04 laptop running "5.5.22-0ubuntu1"

Whether a particular build of MySQL or MariaDB is vulnerable, depends on how and where it was built. A prerequisite is a memcmp() that can return an arbitrary integer (outside of -128..127 range). To my knowledge gcc builtin memcmp is safe, BSD libc memcmp is safe. Linux glibc sse-optimized memcmp is not safe, but gcc usually uses the inlined builtin version.

How do you know, how the Ubuntu devs compiled their mysql server?

Try to connect to MySQL as root with a made-up password several hundred times. If you successfully connect, the bug is present and you know how it was compiled.

where is the perl one-liner :P?

perl -e 'use DBI; for($i=0;$i<4096;$i++){DBI->connect("dbi:mysql:", "root", "nope", {PrintError=>0}) and die "Vulnerable!";}'

Hehe, ok ok.

Well, the Ubuntu part came from HD Moore [1]. I haven't been able to confirm it on my Ubuntu 12.04 virtual machine instance though, nor does my virtual machine appear to trigger the bug using the CVE-2012-2122 checker [2].

But, that is just my single VM instance and I would assume HD Moore knows what he is doing.

[1] http://pastie.org/private/903voijkkz8nmde3yqj4rw

[2] http://pastie.org/4064638

I can confirm. Ubuntu 12.04 LTS (64 bit).

The C script is returning vulnerable on both my local machine (precise) and my vps (Ubuntu 11.10).

However tests of trying to brute force the root password using the mysql one liners in this thread have failed every time.

Both machines allow local access only so I assume I'm safe.

Yep, if you're blocking remote hosts to authenticate on 3306 (or any other port you're running mysqld on) you're safe. The attacking host can't authenticate itself, so it's unable to exploit this bug.

For anyone running phpMyAdmin, make sure to lock it down.

Here's a guide for limiting access to it by IP address via the apache config for Ubuntu users:

http://mixeduperic.com/ubuntu/how-to-restrict-phpmyadmin-ip-...

I'd love to see the code; quite how they are not comparing a memcmp to 0 would be interesting to see...

There is a check_scramble function which returns a my_bool, presumably a char typedef. That function itself directly returns the result of memcmp. If your memcmp implementation returns the full range of int values (allowed), 1/256 of them will have a 0 low order byte, which will then compare equal to zero when the check_scramble call is tested.

This is why real C programmers use int as their bool type. :)

> This is why real C programmers use int as their bool type. :)

It has the same problem if a wider type (e.g. long) is assigned to it. Use _Bool instead.

I was at first tempted to complain about useless pedantry, but you're right. Even if some of us still like to party like it's 1989. I should have said the real lesson is don't reimplement standard C types, poorly.

Here you can see a copy of the MariaDB source code and the file in question https://github.com/atcurtis/mariadb/blob/master/sql/password...

Here is the fixed version of the MySQL code: https://bazaar.launchpad.net/~mysql/mysql-server/5.1/view/35... (line 534 didn't have the test()) previously)

yeap, strange; I'd have thought the obvious natural code would have been:

    return 0 == memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE)

As in, return a bool on whether it matched.

I mean, as I read it, the function returns true if they don't match?

Has anyone managed to actually repro this. I've tried it on a wide variety of systems I run and no repro. Just looking for anecdotal data on how many systems are affected. To me it doesn't seem like a high percentage.

Here is the reference from the Ubuntu CVE tracker: https://bugs.launchpad.net/bugs/cve/2012-2122. So yes it looks like it is confirmed.

Also, the associated bug report: https://bugs.launchpad.net/ubuntu/+source/mysql-5.5/+bug/101....

I can reproduce it on my Ubuntu laptop (amd64 precise), so it definitely exists.

I am trying to make it work on my Ubuntu 12.04 VBox VM. I have mysqld Ver 5.5.22-0ubuntu1, which is the supposedly affected version. I can not reproduce it.

I thought at first it might be because I do not have a root password (it's a VM) so trying to use any password at all returns a failure before the faulty code is reached. Then I tried to login as debian-sys-maint and some other users that do have passwords, and it still did not work.

I couldn't get it to reproduce on a VM (VirtualBox) either. I'm wondering if the SSE-optimized version of the glibc doesn't work the same way in a VM as it does on host hardware (i.e. SSE instructions are virtualized to some degree). Since the hardware doesn't support it, the library falls back to a version that won't trigger the bug.

(Again, this is just an unconfirmed theory.)

while true; do mysql -u root mysql --password=fail; done

So if I try logging in with phpmyadmin 256 times, will I succeed?

Maybe.

If you roll a dice 6 times, will one result be a 6?

Probably.

The probably of NOT hitting a 6 value even after six throws is:

(5/6)^6 = 0.335

For the original question:

(255/256)^256 = 0.367

so yes, you'll succeed in 63.3% of the cases.

<?php while(true) if(mysql_connect("localhost", "root", "password")) exit("connected with password: password");

I am sad to say it worked on my Ubuntu server :(

It seems that SSE4 extensions are needed to be vulnerable, otherwise memcmp() is doing classical computations.

So you need a 64 bits system, and be sure that you are not using a virtualisation system which does clear SSSE4 flag in /proc/cpuinfo (VirtualBox does).

Can anyone view the MySQL bug report linked to from the email? I get 'access denied' each time I try.

"Because the protocol uses random strings, the probability of hitting this bug is about 1/256."

Why 1/256?

Because when the return value is truncated to a char, the last two bytes in the int would have to be 0x00, which is 1/256 assuming the return value is a completley random int.

Doesn't surprise me.

Ubuntu always ship fucked up, broken, shitty MySQL versions.

Look at the one that is current HEAD on 10.04 LTS. It's got so much broken stuff in it, we had to move everything to a spare windows machine where we could stick a later version on without screwing up the machine (DBs for: team city, jira, crucible).

Why migrate to a completely different environment? I'd recommend just installing some version of percona server (mySQL flavour) instead. You get both better software and upstream version instead of a repackaged one.

It was what was there as a piece of duct tape for the minute.

I'm in the process of binning it all and moving to Debian which is actually trustworthy...

except that they try to optimize randomness in security libraries every now and then.

Any more details? Not had any problems on that side of things yet.

http://digitaloffense.net/tools/debian-openssl/

Thanks for this - interesting read :)

Also http://lists.debian.org/debian-devel/2012/04/msg00528.html (but that's just some internal governance tool)

Thanks for link :)

Why would you screw up the machine if you just installed the latest deb?

libmysqlclient dependencies.

It's squarely a MySQL bug, you can't put any blame on Ubuntu here.

It is but they've fixed it in MySQL and Ubuntu have pinned the HEAD mysql build to the broken one and have pretty much stopped supporting it. LTS my arse.

Is the bug limited to ubuntu?

The bug is most definitely in mysql. Its manifestation depends on compiler/library, and it appears ubuntu is the most prominent afflicted platform, but there could be others. Relying on the bug being limited is very thin ice.

Just tried the various one-liners mentioned in the comments on a hardy (8.04) release using mysql 5.0.51a and could not get in.

This is a slicehost box, so I'm assuming that can be extrapolated to mean that anyone using ubuntu on slicehost is probably safe.

Bad assumption. Very, very bad assumption.

It depends what version they are using. My local dev machine (using Precise) is vulnerable, so Slicehost customers running a more recent version than Hardy might be affected.

That's a horrid assumption, is verifiably wrong and what in God's name are you doing on Hardy? You just screamed "ignore me" three times in two sentences.

I'm not the poster, but Hardy is an LTS release that is still supported in server configuration. It's quite a bit newer than Centos/RHEL 5, which is widely deployed on servers (potentially the most popular platform for existing deployments, still).

Reply on news.ycombinator.com