Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage
newsblur.com
newsblur.com
This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models).
About two hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. (Thank goodness I did that! It'll come in handy a few hours from now).
Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world. More info on SO[1].
To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a script kiddie dropped NewsBlur's MongoDB collections, and ransomed about 250GB of data. I am now running a snapshot on that old primary, just in case it reconnects to a network and deletes everything. Once done, I'll boot it up, secondary it out, and be back in business. Let's hope my assumptions hold.
[1]: https://stackoverflow.com/questions/30383845/what-is-the-bes...
Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/
https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
Eventually a kind script set a password on redis which caused me to notice and fix this issue.
'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96
That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...)
Note: that's about 5 years old now...
I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.
And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”
If your mongoDB server is exposed to the Internet it will show up there. When that happens, it's only a matter of time until someone targets you.
You can write an alert that probes for sensitive services exposed to the Internet. In that way, if this happens again, you get an alert that you can use to detect the problem early.
Also... use authentication for your database, it doesn't take much effort to do. With a secure password.
Are they scanning networks 24/7
I’m just a noob in security so therefore learning
Abstractly and intuitively, IPv6's massiveness would seem to put an end to the interesting closed loop of address space vs backhaul capacity that has developed around v4. I can't help but wonder though - with for example some providers leasing out ginormous blocks of address space according to fairly predictable patterns (and customers just using the first v6 address that pops out - if at all), this makes me wonder if it'll be possible to steer v6 scans using a mix of statistics, machine learning, and Perl if statements :).
The other thing I'm idly curious about is how you actually scan on a regular basis. Broadly speaking about long-term viability, I guess the TL;DR probably boils down to coordination and careful nurturing of reputation similar to what the large-scale email providers maintain. But from a technical perspective, I do wonder if/how much things like peering, and BGP, and noise-cancelling routing (if you will), etc, come into the picture - and how big the links are :D
I would be very happy to coincidentally discover writeups touching on these questions anytime. Thanks for reading :)
But for now we can do this with the v4 parts that are left: https://www.youtube.com/watch?v=nX9JXI4l3-E
Also, some crazy person good-haxed a bunch of routers and modems back in 2012 and made http://census2012.sourceforge.net/paper.html (without access to fast connections and before the advent of masscan and other straightforward tools, too). Of note is that the "Unallocated" grey areas in the the analysis images are a curious illustration of how much less-full the IPv4 internet was just ~9 years ago.
https://help.shodan.io/the-basics/what-is-shodan
The scanning algorithm is mostly just this:
1. Generate a random IPv4 address
2. Select a random port from a list of ~2k ports
3. Check the random IP on the random port
4. Store the result of the check
5. GOTO 1
The above loop runs endlessly and because IPv4 is fairly small it doesn't take long to check everything.
If you have a membership (which is a one-time payment of $49 - no subscription necessary) then you can monitor up to 16 IPs. We have a lot of individuals that have configured monitoring for their home network just in case something accidentally gets exposed.
I setup alerting rules so that certain subnets or servers suddenly being exposed to certain instances of blackbox will wake people up!
I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun.
Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1].
In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out.
I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening.
1: https://www.bleepingcomputer.com/news/security/new-meow-atta...
2: https://techcrunch.com/2018/10/05/california-passes-law-that...
Not GP, just hypothesizing. This is what Travis matrixes do, for example.
But it seems that without external pressure (regulatory?), it's not getting fixed at either end.
Except with GDPR, which makes you responsible for having regular security testing, I mean, just an nmap after changing your infra, it takes 2 minutes, and yet when you don't take these 2 minutes you find a way to blame the manufacturer. I mean, don't you /know/ that bots are scanning the internet for open unauthenticated services, and still want to be taken seriously. Back in the days every kid was doing it to host warez you know.
I am also making the observation that this does not seem to get fixed by itself, rather it seems it needs regulation, which includes GDPR.
Wow, the level of entitlement around open source is appalling. People download and run shit for free and get mad that they didn’t configure it right and want the government to go after the maintainers.
I can’t wait for the day where us open source devs have to contribute patches via pseudonyms and tor because they aren’t “government compliant”.
If some philanthropist were to give out free bicycles to everyone, but it turns out that unless you tighten a bolt one of the main support bars will likely snap and could even impale you, the government would rightfully go after them regardless of any "without warranty" disclaimer or EULA.
This is already well-trodden legal ground in the physical space; it's just a matter of bringing it into the virtual space.
Open source projects seldom get popular overnight. Responsibilities grow as you get more popular, that is all...
That is the GP example. If I make 1 bike by hand and put that somewhere on the street, and someone takes it and breaks their legs, that is not my fault. Nor should it be. If I sell that 1 bike, that can be another story.
It will kill open source imho as if you, in your underwear in the attic are responsible for some code you write and put on GitHub, you won't write that code anymore because of the risk; but I don't want to find out who is right here.
Responsibilities should befall the person deploying the software. Aka if you write some software and throw in on GitHub, that should not do anything. But if I take that software and put it in my app or backend and takes people's money because of some bug, then I am the one who is responsible. And this is already the case anyway. No changes are needed.
I think we agree though as I think we are talking about the person who writes and deploys/sells the software: they should indeed be responsible and already are.
I shut it down and moved on, we don't use Mongo to date.
You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default.
Serious: Listening on localhost-only works in dev environments only. In production, it is not the norm to run the application on the same host as Mongo, especially given what a resource hog Mongo is. So, for practical purposes, listen-on-localhost is actually an obstacle is needs to be disabled first-thing anyway.
You guys know this too, because you do exactly this (and a lot more) on that Atlas thing you guys love to upsell everyone and their grandmother on.
Honestly, it is telling that this is the only defence you could provide — that you listen on localhost — and not anything _actually_ secure in prod. Must come in handy when upselling Atlas, I guess, that your the default configuration conveniently omits everything.
> Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-chec...
Snark: maybe they couldn't trust your guidelines because MongoDB the company is a known blatant liar[1].
As the first paragraph of my comment says: listen-on-localhost is untenable in production. Unless, of course, you guys seriously believe people should be running their production applications on the same host as mongo daemons. Honestly, I wouldn't be greatly surprised if you guys believe that.
> Our defaults make it difficult to ...
You have one (1) default that does that. Singular. None of your other defaults do that. And, as I've said above, that one (1) default is also useless, because it's one of the first things that need to be disabled in production anyway.
> However if you do add a MongoDB database to a public IP address we strongly encourage you to add a strong password. Better still do not expose your database on the public internet. Put it behind a firewall with auth enabled, secure it with a certificate and only allow access to named IP addresses.
Everybody knows this. You aren't adding anything new. Nobody's claiming MongoDB _cannot_ be secured. Everybody knows that it can be. The question, instead, is: why does every user of MongoDB even need to make it secure?!
I doubt you can answer that honestly, but plenty of us suspect we know it anyway: because MongoDB Inc. "cares" a lot more about developer experience, than it does about their data.
You're awfully arrogant for someone who has no clue in how to properly architect systems.
If you're using Kubernetes then it's very common to have a service mesh in a Production environment to enforce certain safeguards e.g. mutual TLS and provide circuit breaking, auditing, logging etc. In which case MongoDB would be running on localhost.
If you're not using Kubernetes then it's also common to have some form of middleware to achieve the same as above e.g. HAProxy, F5. Again, in which case MongoDB would be running on localhost.
Look, I've done my fair share of fronting services (including DBs) using TLS/SSH proxies and load balancers. When I needed them. But the question isn't if any of that can be done or needs be done. The question is: Why does MongoDB, which has all of these security support built-in, not enable them out of the box?
And your answer to that is ... throw more stuff on top of it?! Are you seriously claiming that every production user of MongoDB should take on so much software surface area just to fix the broken defaults Mongo ships with?! This is worse than what even MongoDB Inc. does; at least they document how to enable security (lol, what a concept) and merely automatically blame the user for all lapses.
And no, k8s service meshes and LBs in front of DBs aren't nearly as common in production as you're claiming.
Everyone has put some sort of middleware between their applications and databases.
Your claim that no one is running databases on localhost is simply your ignorance.
Are you seriously claiming that the only production users of DBs are "Fortune 10, banks, telcos"?! Or are you claiming that because those guys do something a certain way, everyone else must also do it like that?! This is a weird variation of the Argument to Authority, and even more flawed than the original.
> Everyone has put some sort of middleware between their applications and databases.
Really? "Everyone"? Or are you just generalising the state of the entire industry based off of your limited experience with a small number of players in it?
> Your claim that no one is running databases on localhost
I made no such claim. I said it's "not the norm" and that it's "untenable", not that "no one does it". It doesn't matter that there are a few examples you've seen that do; they're not representative of the entire industry.
On the other hand, for the vast majority of the industry who run databases in production, my claims hold.
The vast majority of the industry, that does not include "Fortune 10, banks, telcos".
Docker, as we know will open exposed container ports to the world, that shouldn't really be the baseline though for not having your instance compromised in less time than it takes to enter an iptables rule correctly, or read the guidelines.
I'm not trying to place blame, it was an exploratory endeavour anyway, but Meow existed because security in Mongo is a guideline and not a rule
As someone who builds secure software solutions for a living it doesn't thrill me that security is often an "optional extra" (looking at you elastic).
If I asked our customers/users the same question you just asked me, and then followed it up with "You must have compromised...", I'd be in hot water.
A combination of factors contributed to us choosing not to use Mongo at the time, if we have such a need again, it will be considered without prejudice.
The first time I commented in this whole discussion was _in reply to_ your "you must have compromised ...".
Only if you choose to explicitly expose them.
In which case the fault is entirely with you.
The userspace proxy used by Docker Swarm is still trashing my L3 client IPs though, yet another docker networking hassle.
Ultimately solely relying on an L3 firewall for access control is a bad idea, though. Your services should all have authentication turned on, even if they are only bound to localhost, for reasons which at this moment must be obvious.
I've commented here in the past about my feelings on Docker's attitude towards real issues raised by users, the only way I can describe it succinctly is "contempt".
There's some serious flaws, with tickets that are the best part a decade old now, one simply can't rely on them to do anything.
Stop using Docker or solve the issue yourself is my advice.
The root cause is that IP packets going to the containers are not going through the INPUT chain of the "filter" table (they go through FORWARD), while various firewall projects like ufw or firewalld only provide convenient management of the INPUT chain, and, worse, don't event provide a good way to express the notion of "packets going to external port 27017 and then forwarded" (i.e. the equivalent of the --ctorigdst option provided by raw iptables).
Here are some options for container engines:
a) Use only a userspace proxy (which is what Docker does when configured with {"iptables": false}). This way, there is no packet forwarding, so packets go through the INPUT chain, just as expected by high-level firewall packages like ufw or firewalld. The major downside (which makes this method useless e.g. for containerized mail servers) is that the information about the source IP is lost, and there is no good way to fix this. I guess TPROXY can help here, but nobody uses it.
b) Use slirp4netns (which is what Podman does when running rootless). It has a really nice mode (available via "podman run --network=slirp4netns:port_handler=slirp4netns") where on the host side, there is only a userspace process listening (so that packets go through the INPUT chain), but inside the container, packets going out of tap0 have the correct source IP. The downside (actually a Podman limitation) is that you can't set up multiple containers communicating over internal IPs.
I would say that I am not really in favor of options (a) and (b) because of the overhead created by the proxy or by slirp4netns. If port forwarding can be done in the kernel (and it can, the only missing piece is --ctorigdst in high-level firewalls), it should be done in the kernel.
c) Document the situation better (e.g. I don't see the --ctorigdst option mentioned at all on https://docs.docker.com/network/iptables/), shift the blame to firewall authors so that they start creating a duplicate of each INPUT rule also in the FORWARD chain with --ctorigdst added as necessary.
d) Provide usable primitives (similar to Network Policies in Kubernetes) to control the firewall for containers, so that ufw or firewalld is not needed.
* Install slirp4netns
* DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER="slirp4netns"
* Upgrade to rootless docker 20.10.7 or higher (was bugged in 20.10.6. Yup.. bad.)
*** WARNING YOUR FIREWALL ISN'T WORKING!! RUN AGAIN WITH --my_firewall_is_broken_and_I_accept_the_risks OPTION TO CONTINUE!!! SEE THIS FOR MORE INFO: <hyperlink to docs> ***
nbset:PRIMARY> show dbs
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
admin 0.000GB
local 16.471GB
newsblur 0.718GB
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
switched to db READ__ME_TO_RECOVER_YOUR_DATA
nbset:PRIMARY> show collections
README
system.profile
nbset:PRIMARY> db.README.find()
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }Your adversary was unsophisticated but this incident was your fault.
Heavy fine yes but not arrest AFAIK. Anyway this is a script programed to scary the target.
Do you even store personal data inside that database?
Newsblur is an American org. GDPR is a foreign law that has no relevance to American firms lol.
<insert Saruman "you have no power here" meme>
I couldn't agree more with the spirit of your comment, but sadly the reality may be somewhat more nuanced:
GDPR in the USA https://www.cookiebot.com/en/gdpr-usa/
"The GDPR has extra-territorial scope, which means that websites outside of the EU that process data of people inside the EU are obligated to comply with the GDPR. ... In fact, the very first GDPR enforcement was against a Canadian company... being a website in the US does not exempt you from GPDR compliance and the territorial distance will not protect you from its enforcement either."
Reminded me of:
CISA amendment would allow US to jail foreigners for crimes committed abroad https://www.theguardian.com/technology/2015/oct/22/cybersecu...
https://www.axios.com/china-hong-kong-law-global-activism-ff...
In reality, a US business with no EU presence only has to follow US laws. The only "enforcement" power the EU has would be to order the website to be blocked in the EU, and I'm pretty sure they can't even do that.
Have you contacted the relevant authorities?
https://www.shodan.io/search?query=READ__ME_TO_RECOVER_YOUR_...
Really? If they ask 100K dollars they are probably not going to be paid. So, just hijack 100 servers and assure that they will pay (since 1K dollars is "not" much if you are running a business).
The data is already leaked, let your users know what was leaked and recover from there.
See also: 80% of orgs that paid the ransom were hit again https://news.ycombinator.com/item?id=27552611
I paid Samuel and entrusted him with my data. Not too much, but enough for it to matter. When faced with a massive leak like this, he downplays everything, calls the hacker a "script kiddie" and calls this "good practice for what will be the first of many sleepless nights", looking at it only from a "service disruption" perspective.
So far we've gotten no indication of what's been leaked, if it contains deleted feeds, or what he's doing to prevent the data from being leaked by the hacker, if anything. He's been solely focused on restoring the service and ignoring the leak. Compared to not having access to an RSS reader for any random period of time, the leak is orders of magnitude more serious to me and I'd wager to most of Newsblur customers.
I honestly don't care if paying a ransom or interacting with the hacker makes him more likely to be targeted in the future, his duty towards his customers was to keep their private data private and not only he failed at that, but he doesn't even seem to register that as his main priority. As far as I'm aware, if he allows the data to leak publicly, then there's no "recovering from there", he's not getting any more of my money.
I'm saying whoever is ransoming the data already has the data, the data is out of Newsblur's control, therefore the data is already leaked.
The data leak is past tense. It has already happened, not will happen. No amount of money will undo that. If that means they've lost you as a customer, that's how it is.
What we now need to know is what data was leaked?
[1]: which to be fair Newsblur, they are, but if a script kiddie hacks you using something so basic as a missing firewall rule.. Arguably not knowing Docker's quirks but using it anyway is the same damn thing as what script kiddies do. Sys kiddie if you will.
[2]: Why is that cause for celebration? Do you not have backups?
The attacker has offered to not publish if they are paid. Their word probably isn't worth much, but $1,000 seems like an affordable sum for a business to gamble on them being honest about it. And if Newsblur doesn't fix their security problems they'll be targeted again either way.
As someone who has a decade of data in Newsblur, if there's any chance that an affordable ransom will keep my data from spreading further I want Samuel to take it.
Do you know if they simply encrypted the data in place or if they succeeded in exfiltrating a full copy?
Not saying it's a good practice but it's a common pattern I've seen.
It’s in the name, so it doesn’t seem worth disputing that at least at the time when the term was coined- it did have to do with age
> or even skill
Skill is literally the defining feature.
What exactly do you mean by “it’s simply a hacker term” anyway? Are words just sounds we make with our mouths?
1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes.
2. If you can afford another server/a hosting provider with VPC-like features, don't put your databases on the open internet. Run them in a private network (RFC 1918), behind a NAT and a load balancer/entrypoint that only routes to your internet-serving applications. Allow only those application servers to hit your databases. If you had done this, the attacker wouldn't have noticed your mistake, because all they could hit was your public server.
3. Keep regular backups. Oh, and test them! If you don't test your backups, you don't have backups - you have archives. In the GitLab data loss incident [0], they had 3 methods of data backup, but all of them failed. A regular test would have discovered this. Don't make that mistake.
Good on you for sharing the events as they happen. I think people tend to be much more forgiving in response to openness. Don't freak out, and write a public postmortem when you're done.
[0]: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...
1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners
2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ...
I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.
You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.
An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."
Maybe there is no excuse, but literally every company I've worked for (including Fortune 500s) has been missing at least one item from your list. So "industry best practice" means committing less time and money than it would take to implement all those things properly (rightly or wrongly) and you need to triage and prioritise.
You see this "authN/authZ above all else" line of thinking in Google's security design [0], with their ubiquitous login wall. For their employees, that login wall has extra hardening - you can't do regular password resets, and you need to possess a physical security key (which acts basically as a scaled-down single-purpose HSM) and pass a suite of posture checks on the device (proportional to the sensitivity of the protected resource) to pass through it.
Then they put this login wall in front of everything, even internal services, and that tends to be OK because the system "fails closed," and the only way to access their protected resources is via physical safe rooms deep in the Google offices in which elevated privileges may be obtained.
Putting all your systems on the Internet forces you to get the incentive structure right, and that can be useful in companies where "private networks" can serve as justification to weaken security - if your login page is Internet-exposed, then you simply have no choice but to make it strong enough to withstand the chaos of the Internet. There's significant merit in that.
[0]: https://sre.google/books/building-secure-reliable-systems/
And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.
This is in vogue but it’s wrong because it presents this as an either/or.
In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone).
There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges.
People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.
I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it.
In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public.
This is easily, done on something like AWS.
In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control.
This holds true whether we're talking RDS DBs, something you've spun in a K8 cluster, or something you're running on a vanilla compute box.
Assume you will screw up at some point and open up a port that shouldn't be. Ask yourself, "what's the impact here?" If you're OK, even with a fat fingered port, great.
Assume you will have a dev deploy a DB without a password. "What's the impact here?" If you're ok, even without a password, great!
That isn't to say that the above two scenarios are something you should tolerate, but automation can help detect these sorts of issues and make it easy for you to resolve them. While that automation is running, you want to make sure you're not going to get owned.
Kudos to the OP for sharing their story. Always lots to learn on this front, and we can always get better at our cloud operations.
You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working.
(Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).
@conesus It's just not worth using discount hosting providers for this exact reason. Use a M(icrosoft) A(mazon) G(oogle) cloud, yes it costs more.... But your time is worth it!
I mean, if you can reliably run Kafka or Postgres in containers, what's sufficiently different about MongoDB?
Or are you talking about using Docker itself as the container host as opposed to K8s or ECS etc.?
There are reasons why running a high performance database instance in containers is problematic, but security is not one of them - not any more so than application containers.
You just need to know what you are doing, it's not a black art.
What's the difference between a container and a instance in a cloud service?
Have we suddenly stopped teaching the basics?
Everyone seems to want to worry about “nation state actors” and getting with novel 0 days when in fact missing the basic low hanging fruit is likely to result in far more damage
I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?
Computer security is hard enough without loaded footguns like these lying around.
Defaults are often insecure but maximise interoperability or general usability. Look at Windows !
The defaults should be secure with explicit unlock steps for those that know their environment well enough that they can explicitly relax some restrictions.
Such drastic changes to the security model should only be one after explicit instruction.
The number of companies that operate without access controls between servers on the same segment is unfortunately quite large, database security controls are - again - more often than not left at their default setting and those too are quite often insecure.
Defense in depth always has limited depth, though I totally agree that running a database without access controls is not the way to go.
The main problem is lack of basic security.
At least they’re being honest up front. but yes it does show a naivety and inexperience of basic security.
You can then expose ports to a specific IP and use UFW to allow it.
Much cleaner than any UFW-docker hacks out there, and more secure.
We have a strict policy that servers must not have a routable IP address at all, and must have an external firewall applied. It's a good idea in any case, but turns out it's absolutely necessary with Docker.
With many people doing this, it is kind of surprising that it took so long for a vulnerable server to be discovered.
[0] https://zmap.io/
As others commented, scanning the whole Internet is even not a problem so scanning a "limited" part where you are likely to see these services pop up is even less of a problem.
I think the takeaway is that you cannot hide in the masses on the Internet anymore, 10-20 years ago you could throw up a insecure server and it could be fine for a long time.
Nowadays you must assume someone will find and try to login to your service, even if you put it on a non-standard port.
Also, if it's a HTTPS service take note they when you get a certificate you will be announcing that domain to the whole world and publish it to a searchable database (for example https://crt.sh/ ).
better still use MongoDB Atlas and get our best security practices baked in.
Angry at myself for not reading the docs carefully but who has time for that? :/
I'm always using this with providers that support it, since I've managed to mistakenly open ports that should be private (either by misconfigured firewall, or due to the Docker "issue").
[1] https://docs.digitalocean.com/products/networking/firewalls/
Other than that, it's commendable that you have working backups and are responding calmly and with a plan. I hope you get everything back in working order smoothly :)
[0] https://www.theverge.com/2016/3/8/11179926/facebook-account-...
the case you mention is more of a hacker feat because that exploit had to be crafted specifically for facebook. meanwhile in this case it was most probably someone who just continuously scans the IPv4 address space for open mongo instances and applies the same generic "exploit" against them if it finds some in a fully automatized process
The terms get clouded a bit - but in my definition a "script kiddie" is pretty much this: someone with not much skill(like a kid), but hands on some hacker tools/scripts - to find easy targets and feel powerful. And later on, try to make some money.
And they can make great effort in doing so - but they remain script kiddies. They don't really know how to hack.
Whether this was just a "script kiddie", I doubt. More a professional ransomware gang. But what op probably meant was, it was not a targeted attack.
https://www.imperva.com/blog/ransomware-attacks-on-mysql-and... https://www.itproportal.com/news/ransomware-attacks-on-mongo... https://security.stackexchange.com/questions/237048/mongo-db...
Everybody falls for that, I mean, look at the BTC these guys made, it's crazy! Anyway, Docker uses the DOCKER-USER firewall chain:
https://docs.docker.com/network/iptables/
Example:
https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/...
People should really test their firewalls after setting it up.
Another thing, instead of using Ansible+Docker and exposing ports like that, use Ansible+Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate.
https://docs.ansible.com/ansible/latest/collections/communit...
And DO still have this in motd for newly created droplets:
Welcome to DigitalOcean's 1-Click Docker Droplet.
To keep this Droplet secure, the UFW firewall is enabled.
All ports are BLOCKED except 22 (SSH), 2375 (Docker) and 2376 (Docker).
Though it links to https://do.co/3j6j3po and it mentions ufw problem:
> Note: The default firewall for the Docker One-Click is UFW, which is a front end to iptables. However, Docker modifies iptables directly to set up communication to and from containers. This means that UFW won’t give you a full picture of the firewall settings. You can override this behavior in Docker by adding --iptables=false to the Docker daemon.
This is just what another poster commented on, sacrificing security for ease of use.
DigitalOcean Support Thursday, March 15, 2018 9:53 PM
Hello,
Thank you very much for bringing this to our attention.
I will create an internal escalation to our images team to review this. :)
[...]I’ve been got by this too.
1. you put your DB in a server which is exposed to the internet.
2. you have no VIP/NAT in front of your systems.
3. you rely in iptables , while knowing some automatic system is manipulating it.
3 hours? I wonder it took so long. I expect this infrastructure will be a script kiddies party room within a few minutes.
1. I need to be able to connect to my DB from anywhere.
2. No idea what that even means.
3. Don't know. Never even touched the firewall.
I have a PW on my DB and that's it. Why do I need more than that?
Understandable and reasonable. there are ways to achieve this without exposing the DB but it take some more effort.
> 3. Don't know. Never even touched the firewall.
Firewalls are good, because they add an extra layer on security. Personally I believe, there should be firewall on most servers (firewall should be the default). But firewalls aren't magic and they are just one layer.
> I have a PW on my DB and that's it. Why do I need more than that?
Because this means you have 0 margins for error. Either you PW auth works flawlessly and without bugs, or you are screwed.
Also you expose one more endpoint that can potentially be a ddos target.
- put the database in a virtual private cloud (VPC), an internal network
- setup a Virtual Private Network (VPN) also placed in the same VPC from which developers can connect to to access the internal network
- setup at least two MongoDB users, one `readWrite` user that can connect from the internal network and one administrative user that can only connect from localhost
- setup a key based SSH connection only accessible from the VPN to the MongoDB instance
- setup Security Groups (firewall) to lock all the unused ports and IP origins out
That way you'll need a VPN key, an SSH key and the MongoDB admin user's access to fully compromise the database.
So do I. So I setup all my dbs with TLS mutual auth or equivalent. Even databases that don't support TLS natively (e.g., Redis 5 and below) get a TLS/SSH port-forward setup for them at the network boundary.
> I have a PW on my DB and that's it. Why do I need more than that?
If you're not using an MITM-proof connection (e.g., TLS or SSH), and you connect to your DB from a network that has me (maybe we're in the same coffee shop, maybe I'm working in your office, or maybe I just work at an ISP between you and your server), then I have your PW.
Is this a useful distinction? Define the sharp line between a "script kiddie" and a "hacker".
You're describing a person who detected your vulnerability and acted within a 3 hour window of it appearing and closing. If I were you I'd be more focused on reflection and not on putting labels on whoever hacked you.
Yeah I had the same problem. If you are not using orchestration engines like Swarm of Kubernetes you can just avoid it by explicitly binding the port to your local interface (i.e. -p 127.0.0.1:27017:27017 instead of just -p 27017:27017)
Seems like it would be a dead simple detection system, and would be a huge value-add.
You could of course order an additional IP and do it yourself, too, but it seems like the colo doing it would be more efficient in terms of scale. (And they'd likely be more diligent in doing it, avoiding obvious DoS vectors like triggering the honeypot from AWS netblocks.)
Hetzner (my host) sends me nastygrams from their IDS when my box tries to connect to RFC1819 space (which isn't even routable via them!) when running p2p software like ipfs. You'd think if they are willing to complain to customers about zero-impact stuff like that, they'd be willing to blackhole non-customers for nonzero impact malicious traffic.
[0] https://github.com/docker-library/redis/issues/259#issuecomm...
This is crazy. Your network should have been on a private IP address space behind a firewall running static NAT exposing only ports 80 and 443 on a routable IP address. This is network architecture 101.
Had some fun using mongodb but I don't think I'll ever use it again =/
I'm now self-hosting a rss reader, but NewsBlur will always remain dear to my heart.
Disappointing response, this.
What data got leaked? Please let haveibeenpwned.com know if your system leaked emails or worse.
Actually it makes it worse, since your security is so bad that any "script kiddie" can hack into your system.
* Ansible
* Docker
* Big redesign
* New database cluster
* New firewall config
I have found great benefit to breaking problems down into smaller parts even though some times it causes some extra work.
Oh boi. I just knew(0) that default value is problematic.
Too bad this time it caused somebody real money.
- all story content from all private feeds
- any uploaded OPML files, including URLs for any private RSS feeds
- User’s twitter/facebook account info and access tokens, if the user had linked those services with their newsblur account
- all data that would be used to create a user profile page, including email address, whether the user had a public profile or not
However most personal data, such as password hashes and billing info, was stored in postgres.
On top of that, I used to use his "forward newsletters to Newsblur" feature for a long time. I've long stopped using it and deleted all the feeds with newsletters, partly because it never worked very well, but mostly because I more or less had an inkling that something like this would happen and it's just not worth it, too many email newsletters leak personal data all over the place. However, I have no clue if those were really deleted or if they stuck around in MongoDB.
Clarification what exactly the ransom is (did he just dump it locally and encrypt it? or did the hacker download it and is threatening to leak it?) would be very welcome.
would you share your alternative?
Considering Newsblur's solution relied on setting up (sender/subject) filters on your email provider, I just kept doing that, but instead of forwarding to Newsblur, I now direct them all to a separate folder.
Lost the grouping per sender, but I honestly didn't explore an alternative too much. Even if Newsblur didn't mess with the newsletters' HTML and displayed them as GMail does, it was just too much of liability to blindly forward emails to a third party service like that: many companies do obnoxious things like send transactional emails from the same address as their newsletters, or blur the line between what is bulk and targeted mail, and I'd rather not have things like emails with flight information and other random tidbits of personal data floating around in someone's MongoDB.
9:54p ET: Holy moly, when I switched to a new Mongo DB server, a hacker deleted all of NewsBlur’s mongo data and is now holding NewsBlur’s data hostage. I’m dipping into a backup from a few hours ago and will keep you all updated.
They've also given some info on twitter https://twitter.com/NewsBlur that I haven't digested yet.
Update: emailed Samuel directly
About a year ago, an app I run had grown to the point where a Linode setup wasn't adequate/cost-effective enough and I migrated it to a multi-server dedicated environment with a Redis Docker container handling queue processing and caching between the machines. I presumed the UFW rules I'd set would protect it from the outside world.
It all seemed to be working fine when I went to bed, then when I woke up in the morning, someone had found the open Redis port and had set up a replication node and was streaming all the data to themselves.
Super-luckily it wasn't handling anything sensitive (just weather data in a small farming region in Australia) but boy oh boy did it hit me just how bad it could have been if it was handling sensitive data for a lot of users.
So I feel for these guys; I'm no security expert but I've been running web apps on Linux servers for nearly 20 years and have never had a breach before, so I feel like it's a pretty easy mistake to make.
On the other hand I'm expecting to be ridiculously productive at work until Newsblur is back up.
For me also, the tooling to actually see what is happening at the network level, what DNS has been assigned, what can and can't route is not easy to identify even though I understand a reasonable amount about the theory.
Even an obvious question like, "if we are sharing a registry between development and production clusters, does that introduce a vulnerability?" doesn't have an obvious answer.
VLANs are great but again, they don't seem to exist in K8S by default and we already read that Docker was punching its own holes in firewalls anyway.
Maybe the default for all of these orchestrators should be private networks unless you specifically open them up otherwise I can see why people might recommend running DB servers on VMs with more obvious attack surfaces.
That said, software suppliers have a serious responsibility to choose sane defaults, especially for security related items. If that inconveniences the users to the point where they have to explicitly overrule the safe settings and that reduces adoption then so be it, that's a small price to pay. Failure to do so will make those suppliers accomplices in all future hacks due to their lack of respect for reality: the internet is a hostile place and anything that can end up facing the unfiltered net will eventually do just that.
Finally, we will eventually end up with a regulated internet because of all these script kiddies and other wannabe hackers, where just like in the real world you'll need a permit to operate a server, mandatory pentests and so on.
The likes of AWS already make it a bit harder to expose an insecure server to the net by checking for common configuration errors such as the one that caused this particular failure.
Has this changed recently? S3 was a huge part of data leaks a few years ago, and that's basically a managed server.
- There is an account-level S3 setting to instruct S3 to ignore public access grants in all buckets in that account (i.e. no matter how bucket is configured, public access is impossible).
- The list of S3 buckets in the S3 console homepage includes a prominent column saying if a bucket is private or not. Buckets that allow public access have a warning icon and red color in this column to make them stand out.
- When creating a new bucket, the default is "Block Public Access settings for this bucket". If you change this, it gives you a warning and asks you to check a box saying "I acknowledge that the current settings might result in this bucket and the objects within becoming public"
- When editing a bucket's access control list, if you grant public access, it asks you to acknowledge "When you grant access to the Everyone or Authenticated users group grantees, anyone in the world can access the objects in this bucket."
- Whenever you are looking at file listings or settings of a bucket with a public access rule in its ACL, the S3 console includes a prominent red "Publicly accessible" next to the bucket name in the top nav on every page
IMO, this is significantly better than how it used to be, and helps reduce or, with the account-level setting on, fully-eliminate accidentally public buckets.
The reality is that proper cross-account IAM role based access is still a little tricky to set up and difficult to test without coordinating with the other party, which means that this won't stop people looking to transfer data to some other account from making a bucket public and assuming it's OK as long as there is an obscure name for the bucket.
No. Hasn't been for years.
> I guess that’s what you get when the “conversion funnel” guys take over “engineering”
That's also what you get when you post rubbish without even bothering to check.
No matter how you twist it, it doesn't have the same meaning as the common English phrase: "enabled by default".
And mind your manners...
Insecure software defaults = bad software.
https://github.com/microsoft/AttackSurfaceAnalyzer
Interesting to see such a strong example of where tools like this could help the very next day.
Note: I haven't used this yet, just saw it and made a note.
What is the general trend for Dockerizing everything based upon? Are we not already largely running in virtualized hypervisor instances on our clouds and do folks actually run multiple contained apps on one cloud instance? Not referring to using ones cloud providers scalable Kubernetes systems, of course, as I see where Docker comes in to play in that case.
It's better to build every server for public exposure from day 1 and treat all connections as potentially hostile, even if they're coming from the internal network.
Re: "You will sooner or later..." it's super easy to test for stuff like this with sentinel - I use this and scan dev / stage in my CI pipelines with rapid7 which will SCREAM about stuff like no DB password.
I would definitely say it's more effective to test your existing layers before adding more layers, and I think the "defence in depth" concept leads people astray there. Having multiple porous layers works on a battlefield where attacks are costly; it doesn't work on the internet where if one attack gets through an outer layer then all the other attacks can immediately get through the same way and start hitting the inner layer.
$ nmap example.com
PORT STATE SERVICE
80/tcp open http
443/tcp open https
1119/tcp closed bnetgame
1935/tcp closed rtmp MAILTO="youremail@yourdomain.com"
*/30 * * * * nmap yourdomain.com | grep open > nmap.log.tmp; diff nmap.log nmap.log.tmp; mv nmap.log.tmp nmap.logWith the resources of the federal government, it shouldn't be hard to find and take down the criminals. Think of how easily the criminals exploit their victims - it is just as hard for the criminals to play defense as it is for everyone else.
don't know the first thing about how to even collect evidence to aid in prosecution.
don't know what crime has happened.
no one died.
probably from another country and "that ain't my jurisdiction"
lazy
etc.
https://sec.okta.com/articles/2020/08/crimeops-operational-a...
No money so no incentive.
Notice, however, when an "oil pipeline" had their billing software hacked everybody went apeshit.
If anything, their shutdown of the pipeline should be illegal/prosecuted, on the grounds of being limited in what you can do business wise if you're operating critical infrastructure.
True cost of each investigation is probably in the millions so investigating hacking of random small company is probably not high on the list of priorities. Evidence that can hold up in the court are hard to come by.
With all the talk about Russian hackers hacking Dem Party, elections, etc, there are zero people in the jail and zero evidence presented that it was even Russians. So it's hard.
You can remove a lot of threats by just blocking every country you have no desire to reach people in. And if major hosting and cloud providers were restricted similarly such that foreign actors can't just rent US servers to stage attacks from...
I'm aware this is an unpopular opinion amongst tech crowds, but it's impossible to maintain order in an environment where some people have to obey the law and some people don't.
It doesn’t solve the problem, but it’s low hanging fruit, and a few checkboxes (or lines of Terraform) if you’re at a cloud provider and using the usual primitives.
ACM also blocked a lot of Indonesian IP addresses (because "infiltrated by sci-hub" according to their support, but since carried-grade NAT is ubiquitous here you end up blocking vast swaths of the country).
All in all, it's really annoying.
Yeah, you're going to stop various bad actors with this – no denying that – but you're also going to block your own nationals who happen to be abroad. Especially for a lot of critical services (banks, gov't, etc.) it's pretty much a non-starter.
That's why this approach is only really complete if supported at a national level. Cloud providers in the US shouldn't be allowing foreign actors to rent them, for instance.
It's possible if US law enforcement bothers working with their counterparts in "unfriendly" countries (I doubt they even reach out to them).
Case in point: 10 years ago a bunch of Russian scammers scammed Americans out of thousands of dollars pretending to be a valid mail order bride business. As far as I remember, because US police did nothing ("it's Russia! we can't do anything"), one of the Americans had to contact Russian police directly. A criminal case was opened and Russian police eventually busted a whole network of scammers. I know because they were based in my city, there was TV coverage. Those scammers targeted only foreigners so our police wasn't even aware they existed, because no one filed anything.
It may be problematic, but Russia are part of Interpol, so surely a record of the crime should have at least been made.
Sounds like apathetic policing.
The resources are whatever Congress and the President makes them. The budget is typically over $2 trillion. Dedicating overwhelming resources - billions of dollars, far beyond any hacker group - to the purpose would be a rounding error. The potential access to expertise and equipment is far beyond any hacker group. The legal power is enormous - the government can access, through cooperation and compulsion, so many systems that the hackers would have to try to access at great expense. What major company is going to resist cooperating against these people? They may enthusiastically participate. Just think of the potential of backbone operators, cloud providers, etc. etc. cooperating.
> zero evidence presented that it was even Russians
I believe that's false.
> (jurisdiction issues)
It's a problem, but not impossible. The US has enormous power in the world if they want to use it.
couple that with the literal marketplaces of hacked Windows computers that you can remote desktop into by country, postal code and bandwidth metrics, the best case is that you end up framing someone
No one gives them a pass. Law enforcement, however, requires a method of enforcement. You have anonymous emails request Bitcoin to anonymous wallet, what do you "enforce" your laws on?
https://docs.mongodb.com/manual/release-notes/3.6-compatibil...
I've never quite understood the opposition to just shipping mongodb with authentication on by default. What sort of use-case does it solve by not requiring it, and is it worth all the bad publicity every time this crops up in a new exploit report?
Or not, maybe their best potential customers should continue to get burned publicly in incidents that have a direct line to their poor decisions.
In conjunction with this, prudence would dictate that you enable authentication as well. In this case, it seems that reliance was placed on Docker to maintain iptables settings to disallow connections from untrusted IPs and that iptables setting was reset.
As always, defense in depth is a good strategy; authentication in addition to firewall rules would have prevented this.
Edit: they/(you || your employer). I know it gets tedious, but calling out your conflicts of interest can save everyone a lot of time.
Sorry about your prior experience. I think early versions assumed a systems knowledge that was at odds with the idea that anyone could just start using a database without any prior database knowledge.
The regularity of ransomware has apparently made the expense “ordinary” therefore now tax deductible. Shrug.
[0] already being signaled: https://www.reuters.com/article/us-treasury-cyber-idUSKBN26M...
Top managers will deny everything, but FBI will start investigations, the will be a whistleblower, and eventually FBI will offer a deal to some middle manager to testify in court that top managers knew that "consultants" were actually hackers.
This will be enough to greatly discourage stakeholders to participate in paying to these consultants even indirectly.
Maybe it won't work perfectly, but at least it will make paying more risky, more expensive, thus less often.
Sanctions work the same way. You can't deal with a company who is under sanctions, and if you try to use some intermediary to get around sanctions, you can still be fined at least.
Because they'll just call 911 and say "I just paid ransom under cover"?
You know this is why incompetent lawmakers terrify me. They believe they can just be "tough" and everyone just falls in line instead of trying the infinite loopholes you leave open, each of which is a better outcome for them than the draconian path drawn by the law in spirit.
If you make payment illegal, not only you're punishing the victim, you're adding a STRONG INCENTIVE for them to keep the whole thing a SECRET.
You're shooting your own foot.
Because some whistleblower will leak it (or just some person who is seeking their 15 minutes of fame).
Or because there's a transaction to China by a company which never worked with China. Like if you paid to a security consultant company based in China, and that consultant company was registered a week ago, and the website was down for a week, that is a reason to start investigation.
Oh, by the way, a law may mandate disclosure of a ransom request like within three days. So even if the FBI couldn't prove the consultants were fake, the company can be still be fined for not disclosing the request.
> STRONG INCENTIVE for them to keep the whole thing a SECRET
I suspect companies which pay ransom now don't exactly shout about it on every corner. Only those who refuse to pay do so.
> You're shooting your own foot.
How it could be worse than it is now?
Hacks in many ways incentivize companies to invest in their security.
Also, if you prohibit paying ransom nothing will probably change. Hackers will continue to steal data, they will just sell it like they always did.
Sure they will target now companies that have more valuable data, but the big picture won't change.
One possible solution to rape is that women walk around with a bomb and if someone tries to rape them they kill themselves and the attacker. Rapists can still be destructive, but at least they will have less incentive to participate in such activities.