Ultralytics AI model hijacked to infect thousands with cryptominer | Hacker News Reader

Ultralytics AI model hijacked to infect thousands with cryptominer

bleepingcomputer.com

Somewhat tangentially, I really dislike that Ultralytics (and others) started slapping higher version numbers of their YOLO variants. Redmon used the numbering scheme v2 and v3 for his improvements on his original model. But Ultralytics' 11 is it's own thing with no connection to Redmon. I just think it gives a misleading impression of what the history is.

Agreed. YOLO, the midel name, was essentially hijacked.

Ultralytics also had, for at least ~a year, a language model replying to GitHub issues using their CEO's account (without any kind of disclosure). It was frequently confidently incorrect and probably wasted thousands of developer hours (because when the CEO replies to your issue with advice why wouldn't you take it at face value?!)

Looks like they've since given the bot its own account but that experience definitely soured me on the company.

(Also, there's an MIT licensed implementation of "yolov9" here: https://github.com/WongKinYiu/YOLO . Affiliated with neither Redmond nor Ultralytics as far as I know.)

>a language model replying to GitHub issues using their CEO's account (without any kind of disclosure)

What is the possible justification for this? And did they just not do any oversight at all? Did no one notice the CEO was suddenly full of shit?

The ultralytics/ultralytics repo is pretty beginner friendly (kudos to them for that) but I surmise that it therefore draws a lot of beginner level coders who can't immediately tell that the AI generated "solutions" are bs.

I guess my question was geared more towards the CEO and company. Either they didn't notice the BS, which isn't great, or the did and chose not to do anything about it, which might be worse.

I've made several contributions to their main repo and the LLM generated mush replies from various core team accounts have been a horror, derailing Issues threads and such. An excellent case study in how not to use LLMs.

The only appropriate response is to turn your own bot on them that submits pointless pull requests so their bot can reply to them with nonsense critcism.

Yep, the bot gave the complete wrong answer to something for me. Problem is that it wasn’t me that asked the question, and the person who read the answer took it at face value.

I knew from the formulaic response it was an LLM but had to fight with the other person to get them to see it. As soon as you see the question being repeated back at you in summary form as part of the answer it’s probably an LLM.

the whole "asking for permission" to assign a YOLO model variant was weird to begin with.

"gpt" is a more egregious example of using naming scheme to gather more attention than a substantial connection to the original.

however this is a simple price to pay for allowing open research without requiring to go through the conventional approval/clearance process.

Google Collab banned affected users.

I wonder how that's going to be resolved, of if Google will just do their usual and make it close to impossible to appeal and get unbanned.

I think they consult some arcane equation when they need to make a decision. Two of the most important variables are virality of the incident and whether or not anyone affected has an in at Google. Mercury in retrograde messes this equation up.

This is why you should never ever use Google Collab.

Should be easy to migrate?

It looks like they did, but the comments after seem to indicate there are still affected users https://github.com/googlecolab/colabtools/issues/4979#issuec...

Serendipitously comes one day after this story[1] was on the front page: at least one Debian maintainer failing to realize the risks of non-alphanumeric usernames. "What could go wrong?" Well, here's Git allowing branch names to contain dollar signs, backticks, etc., because "what could go wrong?"... and... well, this could.

Names are identifiers. Allowing identifiers to contain anything besides identifier characters merely opens new and weird attack vectors.

[1] https://news.ycombinator.com/item?id=42338134

This appears to be a code injection in a PR branch name, not an AI model compromise..

Yeah, I think the story that's being missed here is that GitHub allows branches named "$(curl...|bash)".

Sounds like something GitHub should be easily able to detect too.

Hopefully that's an exploit path they'll close soon, if they've not done so already.

What/where does code get evaluated from a branch name?

In Github Actions, because their runners use echo commands to print out environment variables and variables that have been declared via inputs, which in return is parsed in the UI.

So technically, all environment variables are unsanitized and this was only the first problem in a list of bugs. This bug specifically used the "pull_request" event/action because it is automatically executed without any chance of stopping it, and was using details exposed via the pull requests head.ref.

Next up: git usernames and emails that use shellcode injection names, because github probably won't introduce sanitization to all variables/inputs now.

This is a prime example why you should never ever use a shell to log arbitrary data.

Github should not disallow/filter branch names. Developers should write (and be be given the tools to) workflows that do not allow command injection.

GitHub could disallow/filter branch names. Developers will not write (and be be given the tools to) workflows that do not allow command injection.

Do you have the same reasoning for SQL/XXS injection? Should developers not write code that is resistant to SQL/XXS injection and instead rely on something like a WAF?

These protections (WAF for SQL/XSS, branch names for this) will never be enough. The code/logic must be secure, any additional layer is not enough since the actual target must be secured.

Developers will do it if its necessary, and it is. These situations are just proving it is necessary.

the garbage that is yaml "builds" is essentially impossible to secure

untyped strings, untyped strings everywhere

and they're directly executed, with untrusted user input templated in, with full release privileges

the entire thing is insane

to think pypa deprecated pgp offline signing for this...

Why not both? Git itself checks and sanitizes branch names, and GitHub should arguably match that behavior. I don't think anyone would object to safer workflows and related tooling instead of being told "here, have some bash inside YAML, now be careful!"

Of course I agree that github should follow the git spec for branch names. I meant that I don't think github should impose any additional restrictions on branch names.

The yolo models are “dumb” black box object detectors, it’s a supply chain attack, the model itself was very likely never touched

The malware was in the pypi package, so anyone doing `pip install ultralytics` would've received the virus.

The injected malware code came from the PR branch name, called by a Github action that was misconfigured.

in parlance this is fine because weights of a model is useless without the wrapper code around it. ignoring the monorepo bs a lot of ai companies are pushing (like hugging face also), this is not as misleading as calling open-weight models as open-source.

> Ultralytics tools are open-source and are used by numerous projects spanning a wide range of industries and applications.

Open source and popular doesn't necessarily mean safe.

Technically you can read the code source but no one does that and especially for each update.

The vulnerability was not in the source code of the repo. See https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj... for a deeper analysis - a great read!

It was. The vulnerability is the use of `pull_request_target` on a workflow that has access to secrets.

You are right, my choice of words was poor. The vuln is exactly as you describe, and it's the malicious payload that was not in the codebase (cache poisoning which is not detectable by reviewing the code of the repo).

This is exactly why I'm building Packj audit [1]. It detects malicious PyPI/NPM/Ruby/PHP/etc. dependencies using behavioral analysis. It uses static+dynamic code analysis to scan for indicators of compromise (e.g., spawning of shell, use of SSH keys, network communication, use of decode+eval, etc). It also checks for several metadata attributes to detect bad actors (e.g., typo squatting).

1. https://github.com/ossillate-inc/packj

Reply on news.ycombinator.com