Hacker News Reader

Comment by _ea1k | Hacker News Reader

I'm surprised that things like this don't happen more often. It seems like uploading malicious binaries to npm or other registries would be fairly easy.

Then again, convincing people to play around with a model might be easier than getting them to use your library in an application.

View on HN

I've seen an alarming number of Colab notebooks that casually ask for access to your entire Google Drive and then download a few thousand Python packages (and who knows what else) and execute code.

Eh, have you tried vscode lately?

Some of the official extensions install code from random places like that.

maybe it is more like - one million transactions in good faith and then an actor or group does a Bad Faith action.. is the tolerance for 'perfect' in the way ? How can ordinary use be enabled while correctly blocking out malicious use, without crazy and ugly intrusive security engineering ?

put another way, maybe the cost of a few bad faith actors is much higher than anticipated among people doing skilled work? How is that handled in a human way?

I can guess that de-platforming based on whatever is much easier for the server side. This dynamic is how we got to the personal computer "revolution" thirty+ years ago

Model weights are data, they shouldn't contain general purpose code with access to the host environment.

Sure, any library/tool that harnesses models might contain malicious code. The models themselves should not be able to.

This is more like distributing music as .exe files instead of .flac

You'd hope, but a lot of models are still distributed as .pt files, which are Pickle — which means they can contain and execute arbitrary code on load.

Some popular applications (Automatic1111 etc.) take measures against that, but the real fix is to ignore anything that isn't safetensors.

Which, large platforms like Hugging Faces bear the responsibility to do what they can to prohibit python pickles.

I think some of the models require remote code execution to support things like custom tokenizers.

There are action models based on LLMs like Gorilla LLM or autoGPT that take a natural language input and convert that to API calls that could do all sorts of things from sending emails to performing stock trades.

To my knowledge it's still the harnesses that perform those actions, not baked into the weights (other than perhaps special tokens denoting that an action should be taken)

Reply on news.ycombinator.com