Hacker News Reader

Comment by koolba | Hacker News Reader

> Those sources said the breach appears to have started when the attackers somehow gained access to the company’s Gitlab code repository, and in that repository was a token or credential that gave the bad guys access to Sisense’s Amazon S3 buckets in the cloud.

So plaintext AWS credentials checked into source control.

> Both sources said the attackers used the S3 access to copy and exfiltrate several terabytes worth of Sisense customer data, which apparently included millions of access tokens, email account passwords, and even SSL certificates.

And those credentials can access tons of other credentials, also stored in plaintext.

Even if the security policies didn’t pick this up, didn’t they at least having billing alerts for the terabytes of AWS bandwidth?

View on HN

Isn't S3 egress billed at something like $0.01/GB? Even if you assume they exfiltrated 100TB, you're only talking about maybe $1k. I can't imagine they'd have alerts fine-tuned enough to catch that without driving their on-call staff crazy.

My previous team argued with me when i said to avoid using terraform with vault with our state in S3.

I eventually got my way after 18 months and found out it was really bad - hard coded s3 access keys were floating around that could read any bucket and they had used terraform with vault and that had pulled all sorts of root level creds into the plain text state files.

Lots of other ways to control this risk but I removed vault from TF.

opentofu is solving this with proper state encryption support: https://github.com/opentofu/opentofu/issues/874

State encryption won't solve this problem, it will simply move the key management problem further down the stack. I.e. to the users who are least equipped to deal with it.

The problem is storing anything sensitive at all in terraform. Terraform is terrible for secrets.

Do you think a feature like this is tied to distancing OpenTofu from Hashicorp, e.g., AWS & friends have less incentive to do enterprise feature gating around security?

State Encryption was one of those long requested features[0] (I had it on my ideas list for years[1]) that Hashicorp didn't have much incentive to build. I don't think it has to with distancing opentofu as such, but the opentofu team prioritizing the right things that customers actually need.

[0]: https://github.com/hashicorp/terraform/issues/9556

[1]: https://github.com/captn3m0/ideas#-mars-terraform-remote-htt...

Interesting!

The 'much incentive' part is the interesting thing to me. Hashicorp did build encryption-at-rest for state into their paid remote cloud service. They explicitly recommend using the paid service instead when security is required, and instead of having both versions secure, the paid form simply disables the local one [1], drawing a strong line here between paid/secure and unpaid/insecure.

So I'm wondering if that created a disincentive, from a perspective of enterprises upselling for security & compliance. OpenTofu community members aren't relying on selling a remote management service, and when a cloud provider, more usage in general is better, and that includes security for everyone. I can be off, so not quite clear to me the more I look.

[1] https://developer.hashicorp.com/terraform/language/state/sen...

With a tool like Terraform, when is security not required?

Encryption at rest of local terraform state seems beyond the threat model & compliance needs of a pre-PMF consumer startup

It's a lot harder to be an enterprise company or b2b startup (SOC2) and brush that off in front of knowledgeable auditors. I'm guessing most would miss it and rely on self-reporting, which in turn raises how 'loud' this issue is for most OSS terraform users.

We aren't heavy terraform users, so I'm not sure of the threat modeling here. Understanding these kinds of product decisions and OSS security ramifications is a pet interest of mine, and this is a fascinating one!

Hard agree. When I started my company we were playing with Terraform but left it pretty early and before taking anything into production because of the issues with state management and security.

We moved on, but there are areas where I wouldn’t mind poking at it. I’ve been thinking OpenTofu is where I’d start afresh with TF.

Was that a fundamental problem with TerraForm or just an implementation issue?

The thing is that most HN folks, competent IT people see tools like Terraform, ansible, docker et al as neat tools that simplify and speed up annoying chores, so you get to spend your time on solving actual problems. What it means to managers is that task X has become so easy that you can hire complete idiots who would've failed without the tool but now manage to cobble something together quickly that appears to work well. Except you now have passwords and keys in repos, open s3 buckets, ...

Long lived credentials are a security red flag.

We setup our AWS organization’s policies (SCPs) to prohibit long-lived tokens. Instead access goes through SSO or OIDC.

It’s difficult to track usage behind access tokens, prevent leaks, and effectively revoke them.

Yup. TTL (leases) must become the norm for All The Things.

> didn’t they at least having billing alerts for the terabytes of AWS bandwidth?

AWS is notoriously bad about this. You can't set hard limits. Even if you set a limit, your notification won't come in until long after that limit has been hit and blown through.

At least according to general "HN hivemind experience", AWS is reasonable in handing out refunds if you got hacked.

My business has not (yet?) been compromised, but I did accidentally send my S3 bill into the stratosphere in the past 12-18 months by miscalculating API charges for some intra-AWS changes I was making with lifecycle management and bucket-to-bucket moves.

I wrote to support asking for a refund and explained my calculations. I pointed out where I had made the mistake, and asked for some relief on the bill. Honesty went a long way in that case, and they gave me a credit on the bill equal to about 70% of the total S3 spend. So they ate 70% of my mistake. That was more than fair enough; it was my error.

The amounts weren’t huge. They could have eaten the entire amount and it wouldn’t have even been a rounding error on their books, but I didn’t ask and I didn’t want to expect that.

I can’t speak for others.

> So plaintext AWS credentials checked into source control.

Could also have been keys set as variables for their CI pipelines to support IaC/etc deployment.

This feels... ...ridiculous in this day and age.

On the other hand, once they had source code access, does it matter?

People play make believe about this stuff. “The practice I feel strongly about, it’s secure. And the other guy is a moron.” The XZ crisis shows how little credential isolation matters. Maybe even at Google, you could work there and feel very strongly about their security practices, and then governments have access to everything anyway, at the front end proxies or whatever. It’s important to pay attention to security, but I think programmer resources should focus on “cheating,” not “security.”

Yeah it does, because you left the key for the place where you stored everyone else's keys out in the relative open. It reduces your defensive depth to 1, as in, 1 compromise was all it took.

For a tortured analogy, let's say you keep your guns in a gun-safe in your garage, and then hang the key for the safe in the garage. With a sign saying "gun safe key".

Not leaving the safe key out in the open doesn't guarantee your guns are impervious, of course, a sufficiently motivated and skilled burglar willing to put time and effort into breaking open the safe would be able to access your guns.

But, that takes time. And it makes noise. Both of which increase the chance of being caught.

But if the key is right there? Easy as.

Yep, they had obtained access to the repo, like the burglar had obtained access to the garage.

But if the AWS secrets weren't just hanging on the wall, then progressing from Gitlab compromise to S3 compromise would a) be harder and b) take longer, both of which increase the chance of discovery.

Just saying, if you care about the security of your guns and/or customer data, don't leave the key hanging about in plain sight as a good first step.

Make them compromise multiple things, not just one thing.

I'm desperately trying to work encryption at rest into this analogy. Um... trigger locks?

> Yep, they had obtained access to the repo, like the burglar had obtained access to the garage.

I'm not going to talk in analogies. There's no more depth at the stage that the source code is compromised, regardless of how many credentials are put where. It's sort of a matter of opinion. I mean this is what I am saying by make-believe: everything you say is, quite literally, a bunch of analogies, because there isn't any actual hard evidence to any of it, it is wood carving traditions. Of course I agree that they're good practices, but man, there are unlimited good security practices, and only limited money and time.

Another point of view is, it's easy to provide generalized, vendor-colored advice about credential security. You know, just buy AWS KMS whatever the fuck, right? It is hard to write software that accommodates arbitrary changes to authorization requirements & stories. There are a lot of commenters here giving these Sisense people a hard time, and I doubt they're stupid, more so that they are really unlucky.

I feel like the comments miss the forest for the trees. Most startups have admin roles, and if you're using something like Next.js, Elixir, Ruby on Rails or Go, your application's API methods have ad-hoc, in-source policy enforcement and definitions.

This is what happened here: the developer's GitHub credentials are an Admin role. It could happen in any scheme. Can you develop an application without an admin role? Not a SaaS. Now you should see why it's okay for Postgres to have all its source code open, but not your SaaS company's, for the purposes of security.

I'm really unsure why you're so convinced that "don't leave AWS creds in source code" is just security theatre.

You must be trolling, right? Right?! Programmer resources should not focus on security? Credential isolation doesn’t matter?

I know this says credentials were in a repo but I wonder if this has to do with their self-hosted (EE?) GitLab instance. I've found mine (CE) sort of difficult to upgrade sometimes.

Not sure how they store keys/tokens but it is sort of easy to leak to logs, and perhaps there was some exploit.

Should maybe be using cloud hosted GitLab at their level of scale, even though that option has had problems with uptime among other issues.

A past company used Sisense because the new manager came in penny wise and pound foolish. However, the other ETL/visualization products (e.g. Tableau) truly were expensive. I found Sisense's ETL / viz product to be meh.

Reply on news.ycombinator.com