Hacker News Reader

Comment by hangtime79 | Hacker News Reader

I don't work for Snowflake but I spend a lot of time working with them and their SE organisation.

When working and building demos with clients, SEs create demonstration environments on the same $400 Snowflake demo accounts anyone can. To build demos the client would grant access to that SE. The SE would take some of the data to the demo environment and then work on it. This is further confirmed by the name of the environment Hudson Rock just published.

As far as I can tell, this is a process issue of clients not expiring an ID of someone who they were sharing data with and a threat actor swiping credentials. There is nothing novel about this as there is no exploit.

Also congrats Hudson Rock you just outed a person who was taken due to having malware on their computer. This is no different then if you gave a contractor credentials and they had those swiped. Dicks.

View on HN

Just because there isn’t a “novel exploit” doesn’t mean this isn’t a big deal.

Snowflake is susceptible to their SE’s having credentials stolen. These credentials can bypass MFA. And per the article, they have no expiry. That’s strikes one, two, and three.

Snowflake’s security practices lead to a situation where their customers are either required, or at minimum encouraged, to share access to broad datasets with Snowflake employees. That’s strike four.

Yes, there is also issue here that the customers are responsible themselves for not granting too broad access, and that’s on them. But it’s also on Snowflake for not having a better system that doesn’t require this access, or at minimum not having better oversight and control over this transitive access.

Once these accounts are granted access to a customer’s data, they aren’t “demo accounts” anymore. They’re real accounts, with real, very valuable data, and they should be treated as such.

Edit to add: it is worth noting that Snowflake claims the demo account did not have access to customer data and wasn’t the source of the leak, which is in contradiction with what the attackers claim.

Indeed, the less novel the exploit the more embarassing it is.

Data stolen because of some crazy multi-exploit zero day chain. Well that is understandable, i don't blame the company.

Data stolen because no 2FA support? In 2024 that is just embarassing.

Yes, that's also what I think must have happened—missing 2FA. But it seems it's also not mandatory within Snowflake accounts also, from what I understand from the general message.

I've never used Snowflake and assumed that because you push all your data into it, it probably has 2FA enabled by default. Is it optional?

I don't quite understand how Snowflake works.

My understanding was that you had to grant storage access (e.g. S3) and compute access (e.g. EC2) from your account to Snowflake, which would then use said resources to perform queries that you issue from their hosted web UI.

In that case it would mean stealing the Snowflake demo account of a SE should not expose your data unless you forgot to revoke their access to your underlying resources.

Can someone explain if that is how it works?

No, Snowflake runs it's own storage and compute (on either AWS, GCP, or Azure depending on what you pick).

So the customer data is actually stored on Snowflakes AWS accounts?

What difference does it make what underlying storage / provider it uses then?

Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?

At the snowflake size you get custom price lists from cloud operators.

But I think there was also support for peering with client VPCs (or equivalents) which is why they support AWS, Azure, and GCP - you choose the location that is most fitting for linking with your cloud/physical workloads.

> So the customer data is actually stored on Snowflakes AWS accounts?

Yes.

> Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?

Yes. It's covered comprehensively in their docs, along with the caveats.

> What difference does it make what underlying storage / provider it uses then?

"Snowflake does not charge data ingress fees. However, a cloud storage provider might charge a data egress fee for transferring data from the provider to your Snowflake account."

unsaid: "...and you have to pay for that".

Note that when they say 'your Snowflake account' they mean our cloud account which we own, and which we run our workloads in, which we refer to as 'your' snowflake account.

Tangibly speaking, what means is that if you want to check up your billing; you go through snowflake; you can't login to a cloud console and see the actual charges the cloud vendor is charging.

> What difference does it make what underlying storage / provider it uses then?

They pass the specific underlying cloud vendor costs on to you (with, I guess, some markup, though you have no way of know what that is :)

Snowflake usually unloads data to an internal stage bucket in the same region as your snowflake account. If you use an s3 gateway endpoint getting that data is free of egress charges.

Ironically this model most resembles Teradata, which used to sell their own proprietary hardware/software combination at exorbitant rates.

Snowflake compute instance types cost about $.30-.40 an hour on EC2, so it's quite a markup.

As far as security I do believe they allow the customers to set their own storage keys, so there may be some isolation from a global breach.

This was the case in ~2019 but are you sure this is still true? I think you can “bring your own account” with Snowflake, but I’m honestly not certain because their docs aren’t exactly clear about it…

You can definitely bring your own storage (e.g. store your data in your own s3 buckets and integrate it with snowflake) using storage integrations and external tables.

See https://docs.snowflake.com/en/user-guide/data-load-s3-config...

Personally believe this is the right approach as the data resides in a location fully under the company's control. You could ditch snowflake and the data still resides in your s3 buckets for reuse with a another platform (just remove the iam permissions for snowflake).

Yes, federated queries (external tables) are supported but that is a lot slower than ingesting the data into Snowflake's storage and querying it. Since Snowflake's pricing model is based on computation time, querying external tables are usually more costly because of worse performance.

And the network ingress/egress costs are higher for cross account/region transfers.

I always wondered why snowflake doesn't just install a control plane on customers own cloud resources a la databricks. Seems like they'd be able to mitigate a lot of liability that way.

All storage/compute/networking etc. is handled snowflake side.

For various reasons, you're not getting to touch the actual DB bits.

You can, IIRC, use snowflake-hosted connectors to access external data though.

And there's a "data marketplace" of sorts where clients can publish/consume datasets.

Databricks -does- work that way, iirc.

Having spent a lot of time in the Snowflake ecosystem, a few things to understand.

Snowflake is both a platform and a database. Customers can implement any level of security within the system. For example, when Snowflake introduced OAuth functionality, they put a lot of pressure for us to implement it in our tool. We're small enough and they are a significant partner so we priortised and got it into our platform so that a customer CAN implement it if they want too.

The keyword here is CAN. As a platform with database functionality, Snowflake allows customers to implement any level of security for access. So each instance of Snowflake https://XX123456.us-east-1.snowflakecomputing.com/ security is independent. So let's answer some questions

* "Was this a production breach", No, this was not a production breach. This is not when hackers went into the Azure and were able to elevate credentials and see all accounts.

* "But doesn't Snowflake use MFA", yes and on INTERNAL systems and the ability to connect to the PROD environment of SF it would absolutely have that. However, every customer has the ability to configure a user's credentials for access in their instance https://XX123456.us-east-1.snowflakecomputing.com/. Likely, the SE was granted access to the client's instance through the use of a user name and password.

* "But why not use MFA then Snowflake", this comes back to Okta. You have to create a trust relationship between providers. That's easy when you are only talking about your own employees. They all sit in your same directory. However, to set up a separate trust relationship for another domain, well now we have to get the security team of the client involved. All this so that someone can take a little bit of data and move it into another environment in order to do a demonstration. Not likely.

* "Well that just sounds stupid, my application has SSO enabled." True, but as also a database - Snowflake has to account for entire ecosystem that don't even support SSO-OAuth. For example, two major products that do not support SSO connections to Snowflake: Azure Machine Learning and Zapier. Both only support user name / password connections to Snowflake. So if you were an SE trying to show the value of Snowflake and its integration with AzureML; the ONLY way you could do that would be through user name / password functionality.

The issue in all of this is we are so used to thinking of SaaS as just an application. Databases have to support all sorts of application weirdness. Here is my gut thought on how this went down.

* The SE was working with multiple clients. These clients in an effort to save time and not wanting to have their security team involved and try and create a trust relationship with Snowflake went ahead and created a user name and password for the SE into the client environment vs setting up SSO. Again, this is ONLY controlled by the client.

* The SE at some point had active access to multiple instances of Snowflake to their various clients; with various levels of access depending upon the skill and fastidiousness of the client when they created the account for the Snowflake SE into their account. Again, how the SE is connecting to Snowflake is the same way any other user and application are doing so. I know this for a fact as SEs have to create their own throwaway demo instances https://XX123456.us-east-1.snowflakecomputing.com/. SEs don't have access into the backend of Snowflake.

* At some point this SE had malware installed on to their computer. At this point, this person was cooked and any customer who had given them access to the environment.

As for Hudson Rocks, they can go f themselves for doxing the SE. It was quite trivial to find who this person was and based on the Snowflake announcement they no longer have a job. That person was having an awful week already, now there name is out there. So again, F Hudson for doxing. It would have been easy to blind them out of the exchange (they did the hacker), but did not do so for the victim.

(I condemn doxing in any way)

Calling the SE a "victim" is debatable. If you work environment is infected with malware, you screwed up.

"Typically, Lumma has been distributed disguised as cracked or fake popular software like VLC or ChatGPT. Recently though, threat actors have also delivered the malware through emails containing payloads in the form of attachments or links impersonating well-known companies."

I would not be suprised if this was a recruiter-attack-vector.

Ironically they pulled the article. Did they also dox the wrong person?

Agreed, mentioning the login name of the compromised account seems really unprofessional and unnecessary.

This was a really effective anti-ad for Hudson Rock.

Generally, I like the page and the openness of the API behind it. It is much more common for people to talk about haveibeenpwned as a source for leaked credentials, but the site claims to have over 20 million computer entries from log stealers ... and every computer has XX password.. But yes probably this was written in a hurry to catch the wave?

The entry seems written by someone lacking maturity.

The candor in the screencap'd chat conversation is novel, and will probably drive clicks.

But in its unedited form serves as dirty laundry, and including the language from the threat actor is both unnecessary and inappropriate.

I can't tell if the threat actor agreeing HR would have potentially helped avert this problem is a good endorsement or not.

On one hand testimony from a threat of a product's effectiveness would be good, but on the other, this is a little up close and personal of an endorsement from someone actively ransoming so many companies and putting so much data at risk.

> But in its unedited form serves as dirty laundry, and including the language from the threat actor is both unnecessary and inappropriate.

A threat actor has intent to hold a company ransom for $20 million and your first reaction is to feign offense that the word 'retarded' appeared in a chat log? These are not charm school graduates.

> The entry seems written by someone lacking maturity.

Agree with that.

You saw someone on the internet use a bad word and you felt offended?

This is the description of one of Hudson Rock's main products, "Bayonet".

"Imagine getting access to a lead-generation platform featuring hundreds of thousands of compromised companies around the world with active vulnerabilities that you can convert into customers."

I've seen and dealt with a couple of these types of companies. It's a pretty sleazy tactic, and it's low skill/effort from a technical point of view as well.

Doing some more digging, this is where the data is sourced

"Hudson Rock acquires and purchases compromised data directly from top-tier threat actors operating in closed circle hacking groups. What sets our data apart is its quality in providing high accessibility to hacker groups looking for potential targets, and the speed in which we make it available to clients compared to other threat intelligence companies. Our operational knowhow, and our boots-on-the-ground approach to cybercrime originates from the IDF's 8200 Cybercrime division, and its efforts to thwart nation-state adversaries and professional threat actors."

https://cavalier.hudsonrock.com/docs

This would imply that they are financially engaging with and supporting cyber criminals.

About as ethical as those other 8200 alums, NSO. The ethics of the IDF on full display.

Just a side effect of any apartheid regime, it is a deeply morally corrupted nation to the very core.

Generalizing over the whole nation? Hmm... Where have we seen this?..

In south africa?

Care to explain how a conflict between two national groups that are indigenous to the same land is similar to a racial system of segregation including beaches and restrooms, based on race theory with different racial classifications?

Or are you using a word that describes something different just because it evokes negative emotions?

I see you are ok with what you are... Well, no harm trying - I highly recommend you reading this: https://mo-husseini.medium.com/50-completely-true-things-1ce...

NSO wasn't founded by 8200 alumni, however companies such as the following were:

* Checkpoint

* Palo Alto Networks

* Waze

* Wiz

* Cybereason

Does your theory hold up? no, but why not generalize

They've supposedly founded >1K companies, so easy to cherry-pick for any desired conclusion, but point taken.

That screenshot where the attacker agrees that buying their services would help makes me have a very strong suspicion this attack is either faked, sponsored by or in some way Hudson Rock is in cahoots with the attackers. That screenshot is a pure 100% advertisement manufactured by Hudson Rock one way or another.

As they usually say.. I'd check whoever is running Hudson Rock's hard drive.

The original HN link redirects to this company's home page now, so the blog post was likely an ad campaign to get traffic.

This is the point where the company's domain should get banned.

I just had a quick read through a couple more posts on HR, and a lot of them end with something along the lines of "heh, should have bought protection from us", reeks like a racket.

The whole blog post reeks of extreme self-congratulation and youre right, a total scum move to expose the victim. Altogether very weak performance from Hudson Rock.

It seems somehow like an Indiehacker page, probably taken too lightheartedly. Additionally, the related pages have no real contact details.

Snowflake’s customers are the victim, not Snowflake.

By victim I referred to the Snowflake employee that had their credentials taken

Reply on news.ycombinator.com