Hacker News Reader

Comment by formersaltuser | Hacker News Reader

SaltStack has a long history of home brew protocols, internally written encryption, security issues, and bugs. I'm not surprised they would end up being used as a vector for attacks.

View on HN

See also the F-secure timeline where the salt team's gpg key had been expired for years, salt had a lack of clear communication, and general sloth in response

https://labs.f-secure.com/advisories/saltstack-authorization...

Exhibit A: CVE-2013-2228.

As described inthe relevant entry [0] on Debian's Security Tracker:

> SaltStack RSA Key Generation allows remote users to decrypt communications

The fix [1]:

  - gen = RSA.gen_key(keysize, 1, callback=lambda x, y, z: None)
  + gen = RSA.gen_key(keysize, 65537, callback=lambda x, y, z: None)

---

[0]: https://security-tracker.debian.org/tracker/CVE-2013-2228

[1]: https://github.com/saltstack/salt/commit/e8ce66cf688b43aeb3e...

Jesus that's some rookie stuff right there

They also built their own low-level network protocol based on 0mq. Back when I tried it, it had constant keepalive issues behind NAT and would randomly lose messages (that was in 2014, probably better now).

It wasn't any better as of last year, and its constant connectivity issues were part of the reason I abandoned it.

This is rookie stuff, but it’s a recurring security issue where crypto libraries have APIs littered with footguns. The “lambda x, y, z: None” just strikes me as cargo culting copy paste code - I’m willing to bet there is other software with the same exploit as they all copied the same answer on StackOverflow

The second parameter in the gen_key function is the RSA public exponent. I believe this means the ciphertext will be the same as the padded plaintext.

Here is the documentation from the RSA.gen_key() function.

    def gen_key(bits, e, callback=keygen_callback):
        # type: (int, int, Callable) -> RSA
        """
        Generate an RSA key pair.

        :param bits: Key length, in bits.

        :param e: The RSA public exponent.

        :param callback: A Python callable object that is invoked
                         during key generation; its usual purpose is to
                         provide visual feedback. The default callback is
                         keygen_callback.

        :return: M2Crypto.RSA.RSA object.
        """

https://gitlab.com/m2crypto/m2crypto/-/blob/master/M2Crypto/...

This is reminiscent of

    int getRandomNumber()
    {
        return 4;  // chosen by fair dice roll.
                   // guaranteed to be random.
    }

Attributing this just to Salt is deeply unfair. Ansible has had many similar issues over time, also homebrew encryption etc.

Ansible is relying on SSH for security and does not use any homebrew crypto or protocols by default.

ansible-vault is homebrew crypto

it uses the standard cryptography or pycrypto backends as everything else

I don't understand this claim. ansible-vault uses AES256 which is anything but homebrew.

Salt uses AES too. The problem is it puts together standard primitives in homebrew protocols. Cryptographic protocol design is as likely to mess your system up as cryptographic primitive design.

This is why like me, you're not a cryptographer. AES256 is a cipher, its one component of a cryptosystem. Analysing cryptosystems is a complex area that does not involve vetting software for buzzwords.

I can take AES256 and make it output the image here: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation...

That's not supposed to happen. The part where homebrew diverges from cryptography is that the former involves engineers like us connecting buzzwords together to produce images like the Wikipedia article, the latter involves complex math and rigorous peer review.

I think you're just trolling

> homebrew crypto

AES-256?

https://linux.die.net/man/3/ansible.fireball

It definitely has them however.

You can use salt over SSH, but it is not the default.

Ansible doesn't have a server to expose, let alone publicly. It could be just as bad and still be safer. (So ex. Puppet would also make me nervous) Now, I suspect that Ansible is also in a better position because it's mostly SSH-based, but the architecture also inherently makes it harder to have this level of problem.

Serverless isn't a cure all, Ansible has had numerous code exec bugs due to e.g. interpreting strings coming back from remote machines or third party APIs as templates.

Some of these bugs aren't even that old

Both Ansible and Salt are underengineered, but highly convenient.

> Ansible has had numerous code exec bugs due to e.g. interpreting strings coming back from remote machines or third party APIs as templates.

Sure, and that is bad, but the exposure is still way smaller. Let's say that Ansible has a bug that allows code execution on my machine by any target server or API, and Salt has a bug that allows code execution on the master server. In that case, the salt server will be owned by script kiddies within hours and the only way to stop it is me killing it or restricting access. But at the same time, the Ansible bug can't be passively exploited without me running it, and can only be exploited by my own servers or vendors when I decide to interact with them. I don't actually expect AWS/DO/whoever to attack me, and my own servers could be compromised but that's still a much less likely jumping attack.

What about AWX/Tower? It offers both a GUI and REST interface to the playbooks.

Yep, anything that makes it a server process that listens for connections definitely takes you into the same category. Likewise, running puppet or salt in the local-only mode removes the issue from them.

And this attack payload would have hit ansible servers just as hard. Looking at the payload script if the SSH keys were not password protected it would have logged into the servers https://github.com/Aldenar/salt-malware-sources/blob/master/...

once it had root access it would have hit anything hard. But it wouldn't have been able to get root access in the first place because ansible doesn't have a port listening on the internet

What "home brew" protocols and encryption does Salt use?

Something about Salt has always rubbed me the wrong way. I'd rather stick with Puppet, Chef, or CF Engine if I was considering this tool.

I'm all about IT automation and like the tech a lot, but after running SaltStack for 5 years, I decided to transition to plain old written documentation. When you consider everything that underlies the "infrastructure as code" tech stack, it ends up being an extremely steep learning curve. I was very productive with SaltStack but turned into the only person who could write or edit our deployment scripts. It was a bad situation.

Now, nine months after dropping SaltStack, my colleagues are editing the docs I've created, and more importantly, they're also writing their own. That's a huge win. What I've lost in terms of personal productivity, I've regained in terms of wider participation in our standardization and documentation efforts.

I might slowly re-introduce IT automation technologies, maybe something more popular like Ansible or Docker, but only after making sure the rest of my team has a solid understanding of general IT automation concepts. I think we're on the right track. I had a colleague today ask me about what Python programming certification they should get, so I pointed them toward Google's Python-based IT automation course on Coursera. I supported another colleague's efforts to create a library of standard server images based on our internal deployment standards for one of our private clouds, and I'm encouraging them to expand that work to our other data centers. And so on. The team as a whole works better together, so that's where I'm staying focused.

So you went from full automation to... Copying and pasting by hand?

Are you seriously telling me that you went from infrastructure as code to doing things by hand!?

We were always doing things by hand. I was the only one doing any kind of IT automation, and I failed to account for the learning curve of my tooling when trying to expand its use. At least now, the entire team—not just me—is doing a better job of documenting what they're doing by hand.

The reality is that devops toolchains are really complex. For example, to use SaltStack the way I had things set up, it meant learning:

- SaltStack's domain-specific programming language, which amounts to writing Python in YAML

- their macro preprocessor, Jinja, which has completely different syntax and semantics than their DSL

- a programming text editor that supports YAML and Jinja

- Git and GitHub

- secrets management (and there were huge risks here)

- the general concepts of IT automation and infrastructure-as-code

Learning (never mind teaching) that tech stack is really hard. For example, the Google IT Automation with Python certification on Coursera is an 8-month-long course. That's just one bullet point on the above list, and that list doesn't include all the things I wanted to do on top of SaltStack, namely the continuous integration/continuous testing stuff, which would involve learning:

- the Chef InSpec DSL, which is based on Ruby

- branching and tagging

- release engineering

- test-driven development as a software engineering methodology

- software engineering methodology in general

We were struggling with even just writing good documentation, but there I was asking everyone to write good code. That's orders of magnitude harder. It was too much, like asking somebody to run a marathon without any training. I don't care how fit you are. That's just not going to happen.

Reply on news.ycombinator.com