How we discovered a database leak in one of the biggest Swiss hosting providers | Hacker News Reader

How we discovered a database leak in one of the biggest Swiss hosting providers

security.infoteam.ch

133 pointsredsec61 comments

A little update on the service Security Guardian after the publication of this post.

Thanks to Hacker News and its incredible community, there have been a massive number of new users. We are working on adding more resources to the infrastructure to make the scans quicker. For now, it is possible that some of you have to wait some hours before receiving the first results.

Thanks for trying our new product, we hope to improve it with your feedbacks.

> Then, they blocked the IP address of our scanner so we could not scan their server anymore.

Long time lurker, made an account just to ask this: What are your comments on this unprofessional reaction from the Swiss?

They should release the name of the hoster. Not so much for internet points but to warn the customers. Who knows what information could have leaked from all those DBs, and I doubt that the hosted sent out a report on that incident.

100% agreed.

For whoever was wondering who this provider is: according to whois-nslookup-mxtoolbox_arin_lookup, the server hosting infoteam.ch is provided by metanet (metanet.ch)

Not trying to ruin their business, but they should consider handling issues like this one properly.

Maybe I’m wrong but the PTR of the IP seems to be in a spam blacklist:

https://apility.io/search?q=infoteam.ch

https://apility.io/search?q=80.74.143.113

https://apility.io/search?q=dynco.ch

This thing happens sometimes, specially if you use a shared hosting. Recently using a well known cloud provider the Public IP address assigned was in a spam blacklist! It’s a good idea to have a look at your IP and domains frequently.

(Disclaimer: I’m the founder of the tool used for the lookup in the blacklists)

Sorry, but to avoid problem for now, we prefer to keep the provider anonymous, however we can assure you that it is not metanet.ch.

My mistake, and I am sorry. Feel free to downvote my above post into oblivion.

You should probably mention that at the top of your blog post. bipson is probably not the only one coming to this (apparently) wrong conclusion based on your current hoster.

The moral of the story should have been -- change your hosting provider the minute they commit such a blunder.

Indeed. Like they say, fool me once, shame on you; fool me twice, shame on me.

However how do you go about picking a new provider? How do you know that anyone else is any better?

nine.ch is better (disclaimer - I work for them ;) )

The first time might have been forgivable, but with the second incident it was definitely time to change provider.

The conclusion for what happened is obvious: the host "patched" what the attacker had done, but not the vulnerability that gave them access. Thus, the attackers re-used the same attack later to re-gain access.

Can't test the product they try to promote because emails with a `+` in them are not valid.

Email validation regexes are so annoying. Everyone ought to just use .+@.+ as their validation regex and not be more strict than that.

Beyond that just queue and try to deliver the email. Tell the user than an email should arrive shortly and that if it doesn’t they should check their spam folder and that they should check that they gave the correct email address. When you say this you repeat the email address that the user gave you (escaped for XSS of course).

I think some people “validate” against a strict pattern to keep their users from mistyping, but really there are so many ways to make a typo and still match those regexes that IMO it’s pointless to use a complicated regex and 80% of the times those regexes end up rejecting actually valid (though unusual) email addresses.

I think for a lot of developers the reason they do this is that they’ve learned that they should validate data and so they decide to validate email and to do so they either copy-paste some random-ass regex off the internet or they write their own broken regexes.

All your regex should do is to ensure that there is an @ in the address and that there is something before and something after. This keeps people from mistakenly entering say for example their phone number because they didn’t read what the field was for.

To prevent people from making your machine send your emails where it should not, such as to root@localhost of your server or elsewhere on your local network (don’t know why anyone would and also it wouldn’t be a big issue, just a tiny bit annoying), is a server configuration concern. Specifically, a concern of configuration of the email server software and of your firewalls.

User presses sign up -> Send then to registration form, they fill in their details which you validate lightly client side, they submit -> You validate lightly server-side and either send them back to the form or on to the next step -> You tell them “Thank you, your registration is now complete. An email should arrive in your inbox shortly. If it does not, please check your spam folder and also control that you entered your email address correctly. The email address you gave us was somebody@example.com.”

I could see validating the domain after the @, but yes, delving into the username portion of an email address... that way madness lies.

Sorry, but .+@.+ isn't going to cut it if you want to confidently accept deliverable email addresses. Regex valid, but not email valid:

codetrotter@example

code@trotter@example.com

code trotter@example.com

codetrotter@example..com

codetrotter@example.com.

.codetrotter@example.com

My company runs a website that has elderly people signing up for newsletters. The client is paranoid about not getting every last drop of possible data, and raises hell for every email address that isn't deliverable. There are LOTS of ways to easily ruin a simple email address.

An email address being syntactically valid is no guarantee that the mailbox even exists or is correctly mapped to the person at the keyboard!

codetrotter@example is a valid email

I should have been more specific: an email address that is routable over the internet. Where's the TLD on that?

What section of what standard says a TLD can't have an email server on it? Is AAA not allowed to host an email server on `aaa`[1], and have the email `sales@aaa`?

[1]: And "aaa" is a valid TLD; see the full list: https://www.iana.org/domains/root/db ; now, perhaps it is required to at least have a second level domain, but that's what I'm asking: is an MX record invalid on a TLD?

It’s actually a valid email and I’ve seen examples of such emails, but can’t remember the exact specifics.

There is a other email that the regexp lets pass and that’s still not valid:

codetrotter@example.com

It conforms to the expectated format and could be a valid email, but it’s actually not because no such user exists.

An email might also exist, but not accept mail from you.

The given email address might exist, by could belong to another user.

There’s a million things that can go wrong and you’ll have a very hard time catching them.

The only way to identify if an email is valid and accepts mail is to actually send an email there.

You can, as a help for the user identify odd looking email addresses and flag them in the UI (“this looks unusual, are you sure”), but generally speaking, chances are high that any strict validation will reject real world addresses while not catching all errors.

Good points. Why do regex tests at all if emails could fail in any number of ways?

If you're going to test for at least 3 characters with a @ in the middle, you probably should implement some other simple rules to have a snowball's chance on the internet:

only one @

no spaces

has TLD (guarantee at least one period after @, and something else after, no consecutive periods)

can't begin or end with a period

A (very simple) regex text might exclude some randomly-entered garbage or inadvertantly invalid address. Even then, the scope and reliability of such tests is minuscule.

Your “has TLD” test is already wrong: localpart@tld (example@de) is an odd, but valid address.

If you want to "confidently accept deliverable email addresses" the only solution is to send a test message out with a link that the recipient has to visit to validate that the email address exists. Otherwise what about regex and email valid addresses that still don't exist?

It appears you're missing the main point of the parent post.

You cannot validate an email address.

You can make a basic excruciatingly simple test for proper form. And should probably limit your checks to that.

For all else, attempt to use the email address provided for validation within your onboarding loop with a sufficiently unique verification URL or code. If that succeeds, the address is ... still not absolutely certainly valid, as it may have gone to a third party who proceeded to verify it. But at least it delivered to somebody.

See: https://hackernoon.com/the-100-correct-way-to-validate-email...

> emails with a `+` in them are not valid

Wrong: https://en.wikipedia.org/wiki/Email_address#Syntax

(I know this is of little help if your email app doesn't allow them. I just want to point out that the standard does allow them.)

I think what Mashimo means is that Infoteam.ch thinks that `+` is invalid when it isn't. Sadly they're far from the only ones whose email validation code won't accept the plus character, or many other legal characters in the local part of an email address.

Their vulnerability scanner is basically an on-demand DOS attack. Tried it on my site and almost brought it down

Perhaps your site is just really slow?

It's not super fast, but it's not really slow. A single web server doing 50 req/s, and they pushed that up to 100 req/s. The limit is somewhere around 75.

Good. That also tells you that your server probably needs some DOS mitigation. Because if their service almost takes it down, so can a trivial nmap sweep.

Yes, my site does - but that's not the point - it's a text field on a website anyone can use to hit any site without effort.

It's also supposed to be a helpful service, which generally implies it's not gonna behave in a way some would consider to be malicious, but it has no rate limits. If I saw that hit my logs i'd consider it to be an attack, not a friendly vulnerability scan.

This is a nice anecdote to promote their new Guardian service.

For the longest time a "clean" MySQL install would set up an no-password superuser for presumably dev convenience. I don't know if they changed that (it's been a while since I last installed MySQL) but if not, this could simply be a security hole by design, with the maintainers simply not paying attention to their install script flags.

I thought it was no password, but only available via localhost?

All that requires to be exploited "on" localhost is some PHP script interpreting unsanitized user uploads (uploading a php script that has an image file extension's a pretty famous example) on any of a thousand customer sites. You don't want that MySQL user to exist, ever =/

Sounds like they may have had a deploy script which ran again a week later or something like that :/

(Also overriding scrolling is not cool)

I second the scrolling part. Between the massive fixed header and the overridden scrolling the page is unusable to me. I am able to get rid of the first with a little bookmarklet - the second one I haven't figured out yet.

"If you use this network security appliance, you can stop all traffic that doesn't match the profile of your normal traffic from leaving your network."

"That's expensive, and complicated! We'll just do regular audits and be fine."

[some time later]

"Someone exfiltrated all our data using mysqldump!"

= /

currently infoteam.ch seems to be hosted on METANET (metanet.ch). Is there anyone who can deny or confirm that this is the provider they don't want to mention?

source: nslookup infoteam.ch; whois 80.74.143.113

Just in case anyone is reading here first and because this could really hurt the business: The infoteam.ch people have stated above[1] that the provider was not Metanet.

(I do think, however, that it would be important to know.)

[1]: https://news.ycombinator.com/item?id=16529273

> "Hopefully, we had ‘only’ read access and could not write or delete anything"

Sounds a lot like feigned ignorance about the nature of the root user. Not entirely sure if it would help them in a court of law. They should probably anonymized the whole thing better to be completely on the safe side (not a lawyer though).

That reads more like an attempt to make a light-hearted quip suggesting the exposure was limited. Say, more like: "OK, so we could directly access the database, but surely nobody would provide a full read/write administrative account publicly accessible and passwordless.... Oh."

Given a few other hints at English not being the first tongue (or possibly even the fourth, CH having four official co-equal languages) of the writer(s) in question, that could just be awkwardness from a non-native speaker. Their English is vastly superior to my German, French, Italian, or Romanche.

I'm getting a "Something went wrong. Please retry in a moment." error when trying to submit a domain to Security Guardian (tried different domains and email addresses). I assume you're being hugged to death.

Moral of the story. For my sake:

Check the access logs.

Regularly.

Why, are you bored? They can just access the db from an internal host that already has connections and exfiltrate from there. That's part of why people retrieve data using holes in web apps - it looks like a spike in normal traffic.

I'm sharing my takeaway after reading this. Access logs are fun to some people ;)

:s i can't even get my mysql to get me to be allowed to login root without password >.< that takes a special kind of negligence.... and really, how long was it there before they developed a new product and tested it on themselves? :/ seems logical, especially for a security service provider that with the lack of such product still this would be noticed?

that besides pitching their own product for an issue any similar natured scan would pick up i'd say it smells like marketing department at work more than chinese hackers or shitty service provider.... >.>

i doubt they would have left a passwordless root on their mysql, or didnt they check the initial setup they were given by the provider before taking it in use?

If you read the story you would see that the database host is a shared host (as in hundreds of other clients of the Webhost have accounts on it) and that the error is likely the result of a persistent hack. As in there is a vulnerability where someone can get access to the server and create a passwordless root account, so that they can siphon the data out.

Once that account is deleted, a new passwordless root account is created by the attacker in order to continue access.

tl;dr, portscanned a server, found an open MySQL port with a weak password.

Not just weak, but no password at all, for the root user.

And the problem returned a week after the 'fixed' it.

Important to remember that these “one of the biggest $x in $y” where $y is a country with a population under 10 million means that the statement encompasses many entities which are just a half-dozen people in a small office somewhere.

I know nothing about the particular hosting provider in question.

Security Guardian is not a he/him. It may be a translation issue ... or maybe you've achieved human-level AI and it's become self-aware? In any case, I find it interesting that your first response is that the tool might have a bug ... and the link also on HN at this moment is about the Apollo 13 mission control engineers thinking their telemetry might be at fault. This is an excellent first response and it's important to provide a way to distinguish between the two.

Ships are not she/her and yet we use that kind of language with them all the time. Silly thing to quibble about.

Given the number of failures I've seen as a direct result of security and/or availability systems, the instinct to first question your own readings is a valid one. It's a good check particularly to make sure that your tests are valid, and that you aren't going off half-cocked with an alert (alarm fatigue is a Real Thing, and attention is a scarce resource).

I've also seen, in general, far more false alerts than real ones. (Yes, Nagios, I am in fact looking at you.)

The gender-casing is a quirk of the article, and would reflect at least three of the languages official within Switzerland (German, French, and Italian are all gendered languages, I'm fairly certain Romanche is as well). It's fair to note that, though perhaps not quite to the extent you did.

They probably use a homebuilt admin panel and sw mgmt, and an update brought back in the old root vuln. They don't use cPanel or Plesk. Or the Chinese hacked it again.

Interestingly they - https://security.infoteam.ch/ - offer the very same security service, automatic security audits for their customers. Which explains their angry response the 2nd time.

Maybe I got you wrong, but infoteam.ch are the guys who wrote the blog-post, so your link is to their product they are describing, and not one by the provider? How does that again explain the angry response?

AFAIK there is no mention or link to the provider, I guess since they don't want to make enemies just yet (and lose hosting while doing so). But I am willing to speculate that they are looking for another provider just now :)

I got it wrong, not you. Thanks for the heads-up.

They didn't mention who leaked the data. I got confused. The symptom they described looks like an automatically updated system with a leak. One of our swiss customers, a big hosting provider, uses such a system and would fit.

Reply on news.ycombinator.com