Space secrets leak disclosure | Hacker News Reader

Space secrets leak disclosure

huggingface.co

197 pointsmarkyg83 comments

Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like Huggingface means the author of the model can execute any code on your machine. Not sure if the slides are uploaded elsewhere, I got them sent as file: https://dro.pm/c.pdf (45MB) slide 188

I didn't realise at the time that I flipped through the slides that this means not only the model's author gets to run code on your machine, but also if Huggingface got a court-signed letter or if someone hacked them (especially if they don't notice for a while¹)

As someone not in the AI scene, I've never run these models but was surprised at how quickly the industry standardised the format. I had assumed model files were big matrices of numbers and some metadata perhaps, but now I understand how they managed so quickly: a model is (eyeing slides 186 and 195) a Python script that can do whatever it wants. That makes "standardisation" exceedingly easy: everyone can do their own thing and you sidestep the problem altogether. But that comes with a cost.

¹ https://www.verizon.com/business/resources/articles/s/how-to... says 20% doesn't notice for months; of course, it depends on the situation and what actions the attackers take

Are you telling me that when I run software on my computer I could potentially be running software on my computer?

Nobody expects a model file to be code thrat executes whatever.

I'd say anybody who is working in the field has this expectation. But the outside observer who is excited to try a new model does not expect it.

This is like expecting any random binary to only do what it 'claims' to do.

No, this is like PDFs that can contain and will execute PostScript but without a sandbox and with more general libraries

That is borderline incompetence

This is more akin to downloading a jpeg and the jpeg running arbitrary code. Models should be like jpegs and I believe safetensors treat them that way, while the old pickle format didn't.

The weights are like jpegs, the model itself is still a piece of software that executes code.

You typically need more than the model to actually use it, as you might need to "massage" the data into the right form before and after the model call.

I think the gap between 'should' and [short-lived] reality is the point - that's obviously the reasonable setup, but pickle happened.

In the quest of being first or whatever, amateur rediscoveries are happening. Consistently.

I'm making a disgusting amount of money in the field by doing day-one packaging work any commodity project would also be doing. They just wouldn't decide to redefine things in a clearly worse way.

I don't actually know jack about AI, yet here we are. The hype/gold rush is real; competence in the shared fundamentals can go a long way.

The trick is knowing/applying the basics for the wizards. There's a steep premium so they can play around, apparently.

Case in point: most of these humongous container images and how they're naively moved around

Now that more people know how to publish (containers/packages/articles), we're seeing more drivel published. It's ridiculous yet inevitable. I just wonder how much value the rush provides.

IIRC 'safetensors' aren't necessarily even that safe. I could have sworn I read they're fairly nebulous. Grain of salt etc

no such exploits have been publicised against known safetensors, so I'd need more than an unqualified Internet rumor to consider them unsafe. Not saying it's not possible but the entire point of them is that they're a blob of numbers, and not a known exploitable bit of python.

How're you making money? is it a contracting via network thing or can anybody apply?

I tried to explain that it's nothing more than a rumor. I wrapped that in poor recollection as much as I could. Three absolving statements for the one!

Did you, somehow, feel I was trying to convince? I'm fishing for education.

I'm making my money doing 'SRE', which really, amounts to doing housekeeping for the AI people.

Anyone can apply. What are you insinuating? Say it. My situation (other than comp) isn't special, they generally need help.

I don't really need to know about what they're running to keep the systems online and running well. Another day, different workload.

I may bother to learn a thing or two about it... but it's not a requirement. At all. Same toys all the way down. It's shiny old news at this level.

I don't mean to belittle it. It's jazzy stuff. I just build the boring parts - OS, VLANs, etc.

My 'ire'/elbowing is that 'they' (the AI folks), almost pathologically, refuse to engage with the basics like Python packaging. It's with love, honestly.

> I'm fishing for education.

(imo) Could've said safetensors are secure, and wait for Poe's law to kick in, or just not said anything.

Apologies if it came across that I'm insinuating some sort of thing, it's just I've some friends looking for that type of work. My email's on my profile.

Noted, I could've been more explicit - safe to the best of my knowledge, but it's murky/weak.

Apologies here as well. I know I have a short fuse. The internet [and sometimes, life] trains us for it.

But no, really, where are you finding this lucrative work?

In a gold rush go to those who make the shovels. I can't control demand, so supply it is

I follow trends. Web hosting, cloud, now AI. It started out as, 'I want a discount for the things I can't afford'. I had an interest then found the work.

That's as close to doxing myself as I'm willing to get. Sorry for the riddles. I may reach out when things calm down personally. Work is about all I have that's right, right now.

These AI people all want their own clouds. The clouds want to be AI. What's old is new again.

This dynamic is where I exist. Point at one the companies that makes things, you're probably in the right direction. There's maybe a dozen worth attention

Following Linux and company into new areas. It's that simple; building blocks get reused

I realized this immediately and have been consistent about it over my career. That's at least part of the compensation. We aren't gluing Lego bricks, y'know

Data intended to be read as instructions for the interpreter or the cpu, is a whole different ballgame than data intended to convey values of something. High order sparse/dense matrices serialized in some xyz format is what most people think of when they hear the word "model". To switch it up and send some arbitrary python file and execute it on the client is a security nightmare. This outrageous.

Isn't that why we have the .safetensors format, which can't execute code on the host?

This is my understanding as well

I was at Info sec local meet up six months ago, and mentioned the Tensor models get control of your CPU and GPU… I was basically passed over like I was some idiot making LLM == TERMINATOR claims.

Good stuff.

> Tensor models get control of your CPU and GPU

"get control" is not accurate.. there is a code execution model on the CPU based on an operating system; the GPU is a different story. Executed code with an operating system has execution context and protections.. this is basic to operating system theory, all the common operating systems implement related concepts.

Cool… so… get control of a CPU means being able to execute code.

It can’t necessarily defeat ring protections or secure zone depending on the context. But then again, I wouldn’t put it past a good exploit to do so.

If someone can inject bad code for execution into the GPU that's also pretty bad though yeah?

The GPU should be able to pull and push stuff into memory, and maybe even access other devices directly over the PCIe bus?

> I had assumed model files were big matrices of numbers and some metadata perhaps

ONNX [1] is more or less this, but the challenge you immediately run into is models with custom layers/operators with their own inference logic - you either have to implement those operators in terms of the supported ops (not necessarily practical or viable) or provide the implementation of the operator to the runtime, putting you back at square one.

[1] https://onnx.ai/

Hugging face standing right behind you, ready for hugs

> using a model from a place like Huggingface means the author of the model can execute any code on your machine

To my knowledge this is only a problem if the model is serialized/de-serialized via pickle[0].

[0]: https://huggingface.co/docs/hub/en/security-pickle

The fact that pickle even exists is fundamentally wrong to start with. They should not be permitted as a distribution format, period.

it's pretty neat and functional if you are the one writing the code. you can do things like "hibernate" with zero developer cost.

That's what it's used for in a major application I use on a daily basis. The entire config for the application is saved off in a pickle binary file and then loaded at whim. Makes it super easy to backup and transfer configs to compatible systems rather than saving everything to a database. One time we were trying to get the system to do something it was not designed for but was probably capable of. Eventually someone got it working on one of the two identical systems we have through a lot of trial and error. Rather than two people sitting on the phone all day comparing hundreds of settings, we just grabbed the pickle file and emailed it over and had it instantly working.

As others have pointer out, this is format-dependent. One of the formats that hasn't been white-listed in this thread yet is GGUF, used by llama.cpp and derivates. It's pretty much "big matrices of numbers and some metadata." Some vulnerabilities were found [1] and patched.

[1] https://www.databricks.com/blog/ggml-gguf-file-format-vulner...

> Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like Huggingface means the author of the model can execute any code on your machine.

Proceeds to link to pdf of unknown origins

That's precisely why it's unexpected that a data model can run code. Wouldn't expect a pdf to start executing code on my system either, it should be data!

(The dro.pm link will expire any minute now. It's so short because it's temporary, should maybe have used a more permanent service. I've found the talk here in case you're reading this later: https://m.youtube.com/watch?v=8XysLIq-e3s)

That’s a very weasley worded statement, to begin with “they have suspicions” is not a statement that should be in a communication of this type

I thought it was pretty good actually. Most of these leak disclosures usually say things like "We do not have evidence they accessed any secrets" or something like that, because they don't "know" what the hackers did once they were in. At least huggingface is saying "Yeah, they probably accessed secrets but we can't confirm it"

> Over the past few days, we have made other significant improvements to the security of the Spaces infrastructure, including completely removing org tokens (resulting in increased traceability and audit capabilities), implementing key management service (KMS) for Spaces secrets, robustifying and expanding our system’s ability to identify leaked tokens and proactively invalidate them, and more generally improving our security across the board.

That's a serious amount of non-trivial work to be done in "a few days". The kind of work that should trigger more time consuming activities like security audits, pen tests and the like, before going live, right?

Hopefully the work was underway for awhile already, and maybe they just launched it now because the damage is already done?

at a larger organization with a whole SRE department that inclues a dedicated security team, sure, but (my impression is) huggingface isn't that size of an org (yet).

My anthropic key was leaked and someone ran up a 10k bill on it. Are HF going to cover that?

I always thought you could set your "maximum limit" for spending on cloud providing platforms.

That $10k was probably the limit for their work, not someone else’s stolen time.

That's surprisingly not a thing in many platforms.

Anthropic is too new to have built that functionality I guess. Only found out because they were mad that my key was abusing their ToS and they notified the organization owner.

> Anthropic is too new to have built that functionality I guess.

That’s no sort of excuse

Are you sure it was only stored in your space secrets? Not variables (which are public) or stored in the .env file (also public).

I searched everywhere for any other leaks of it and found nothing.

i think you can ask Anthropic to provide access data (IP addresses, User Agents etc) specific to your key.

Then you can challenge hugging-face (eg paying customer) even sue them if you wish to...

My openAI key was leaked and I noticed someone was using it, luckily the damage wasn’t nearly as bad as you. A few dollars worth of GPT4, a model none of my apps were using at the time.

I’m almost entirely certain it was leaked via secrets on HF space, I got a message a few days ago warning me some of my spaces were affected

I noticed a few weeks ago that some of my OpenAI keys got compromised, they were only active as secrets on a huggingface space. I got an email a few days ago informing me that the spaces were compromised , so I suspect this issue has been going on for at least a few weeks

The title made me think this was an article about space, but instead I got an article about Space.

I legit thought someone leaked proof of extraterrestrial life and disclosure began.

Another day..

There's no mention of handling with regard to costs inappropriately incurred - wouldn't access to the secrets let people call APIs and run up costs?

Or is this purely about theft of data/code?

It could be both. In my case my keys were used to call OpenAI, almost certain they were leaked from my Spaces secrets

What is ‘Space’ ?

Its shortcut for their Spaces.

https://huggingface.co/docs/hub/en/spaces-overview

The front end/portal. I speculate that is coded in Python. Maybe some Django thing...

Why is being coded in Python relevant?

Might be relevant or not. Depends what you want to know about it.

It’s a vm where they run your code

Why does HF store "secrets"?

Couldn't they just store a public key, the user has the secret key and signs their requests with that?

Usually you need some sort of "token" that lets you practically operate within a browser session. It seems like this is about tokens they had to revoke, which is kinda like a password but not.

You can build apps hosted on HF which access third party APIs, e.g. OpenAI or Anthropic. The api keys for these are then stored in the HF secrets

given how difficult it was do a simple thing -- this was not a surprise

For all those wondering, this is not about aliens.

That's what NASA wants us to think.

It's just occurred to me that if Aliens wanted to take over Earth...They could progressively leak scientific secrets, under the disguise of normal scientific progress. This would lead us create a Trojan-ed AGI, that would take over everything, and just build spaceships to ship them all our Palladium...Just imagine a giant spaceship on the way to Proxima Centauri full of stolen catalytic converters....

Can't get into the details, but it seems there is a way to convert Palladium into Dilithium Crystals. When you achieve that all hell breaks loose....

Aliens already took over earth, we are the aliens.

Yeah, but we have no Tentacles, so we are considered a kind of subspecies, and ostracized in Galactic social circles...

The cuttlefish and the octopi are the real aliens. You might not think they pose much of a threat. But of course, that’s merely because they are intentionally dormant at the moment. Once the AGI is complete, and the spaceships have been built.. it’ll be “so long and thanks for the fish” just like Douglas Adams predicted except he suspected the wrong species.

Inventing technology is hard under water, hence octopuses created us to build them spaceships so they can go back home.

I like it!

This figures in the plot of A Deepness in the Sky by Vernor Vinge

Never read it. Sound interesting, and added to reading list. Maybe I am a just an LLM based Replicant...

Highly recommend it, as well as it's ... sequel, sort of? A Fire Upon The Deep.

(Though I would skip the third book in the series, personally.)

Also, somewhat in The Lives of Tao, by Wesley Chu.

Sort of the opposite of the Three Body Problem, haha

Before reading it I thought their AI components detected life beyond Earth, basically what SETI has been doing for decades [1].

[1] https://www.seti.org/

Disappointed.

How can you be sure? Have you asked them?

tbh, I didn't think it would be aliens, I thought it would be ITAR related.

https://news.ycombinator.com/item?id=40539993

Very disheartening. HF is doing so much good in the AI community, much more than regulators understand at the moment.

What does this comment mean? Why is it disheartening? What do regulators have to do with it?

Reply on news.ycombinator.com