Comment by cetra3 | Hacker News Reader

Comment by cetra3 | Hacker News Reader

Parent Full thread

This whole salt-stack incident could've been handled a lot better by salt themselves:

- the notification was a week ago to a small mailing list, which is tucked away on their site

- no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam)

- no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or anything on hn.

- they only blogged about it on their official site yesterday, way after damage had been done

- one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability

- the patch was released late Thursday early Friday depending on your timezone, giving attackers the weekend head start

- the official salt docker images were only patched yesterday

- You can't get a patch for older versions without filling out a form and supplying details

- Ubuntu and other repositories are still vulnerable

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous.

Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

> has network access to the salt masters port, which should never occur

You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not.

People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched.

> the ones that were compromised by this issue have deeper security issues to deal with

Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't.

(For the record, I've already been working for a few months on a move to Ansible for non-security reasons)

> People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet.

I question that that is a legitimate reason to expose it to the internet.

Defense in depth is a thing and putting the keys to the kingdom at layer 0 doesn’t seem wise even if a vpn or bastion doesn’t offer perfect protection.

Read the sentence after the ones you quoted. The internet connected salt master is used to provision accepted hosts in to the tunneled (VPN) network where the real master lives.

You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.

Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.

Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure.

Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history!

So yes, it was made for sysadmins. :)

> It's how I bootstrap random VMs and bare metal from the internet.

What's stopping you from bootstrapping a bastion server through a simpler script and then bootstrapping with Salt from there?

That is exactly what the internet connected Salt master does. It bootstraps enough control that I can get the tunnels and keys properly configured, and the other 95% takes place once it is switched to a protected Salt master.

Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.

+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.

that’s just plain negligent. of course you use network isolation as one of your defenses in depth.

I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)

If they wrote software which should never be visible to the internet, they should have made that clearer.

It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.

If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't.

> Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet

Is this SSH access or is this access to the salt master from minions? Or just access in general?

Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?

SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly.

I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.

> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability

While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.

You have clearly never worked at a large enough OLD corporation.

One week is nothing compared to what it would take to upgrade your configuration management system.

That sounds like an old corporation problem. The world should not pay the price of old corporations inflexibility.

If someone hacks your system you certainly wont have a week to respond. The longer a vendor sits on a vuln, the more likely it is to leak or to be rediscovered by a malicious party.

Weird that something with this title got buried here on HN: https://news.ycombinator.com/item?id=23041528

It was posted, just no traction on it (eg. https://news.ycombinator.com/item?id=22972100 posted 11 days ago). But yeah supprising the lack of posts about it.

> - Ubuntu and other repositories are still vulnerable

isn't really salt's problem though.. same could be said for relying on any distro-provided package

Reply on news.ycombinator.com