Jellyfish GPU Rootkit | Hacker News Reader

Jellyfish GPU Rootkit

github.com

84 points_8j5038 comments

I wouldn't really call this a rootkit, it seems more like a function instrumenter. To me it looks like the hooks are not persistent and require the LD_PRELOAD environment variable to be set to function, which is very easily detectable. The only cool thing here is storing the log data in GPU memory instead of the system RAM. The actual hook logic is also still running on the CPU.

Even with the actual hook logic running on the CPU, this prevents a large amount of possible detection from various filesystem/workingmem vectors...

And forensic analysis too. I think lack of existing tools that analyze and protect GPU based anomalies is what this project highlights.

But the hook's .so file still has to be loaded from filesystem, so it should be still detectable?

And even if "hook logic" runs on the GPU, the actual function calls still have plenty of CPU parts. Check out rootkit/kit.c -- it is just plain old LD_PRELOAD rootkit, with a ton of calls running on CPU. Workingmem detector should have no problems matching this code.

I also don’t think it would work as-is on macOS due to two-level namespacing…

They also made another small project to hide executables in GPU memory on Windows: https://github.com/nwork/WIN_JELLY

It’s like the ‘you can store a virus in a jpeg and running it just requires this decoder’ “scare” from a decade back.

Don't joke. I have had security people report my stuff is insecure because there is a document upload feature and a user could upload a virus and name it virus.png and then a user could download it and rename it to virus.exe and run it.

Change the scenario to "download it and double click it, exploiting vuln in file format handler" and it's correct.

At that point you may as well ban all user generated content since one user might have an insecure OS

Well, security engineering is about tradeoffs and managing risk, you won't get far outlawing all sources of risk. The smart way to handle attachments is to scrub them somehow (re-encode, pdf2pdf etc).

That is actually a serious issue especially if you allow unicode and RTL override in the file name. Always validate input file type.

MS07-052: Code execution results in code execution

https://devblogs.microsoft.com/oldnewthing/?p=25683

That's very interesting but I have to disagree with the response. The vuln might be used iirc to bypass signature verification of the PE? It might also possibly be used to evade sandbox analysis of malware, AV heuristics and ML based approaches might also be thrown off guard.

But yes, It would have been a lot more interesting if there are cases where the executable is attacker controlled and the exploit would end up running code as the privileged parent process or some other user that has more rights.

Modifying the binary should change it's digital signature, making it invalid in the first place. Running unsigned code is running arbitrary code

No,what I meant was the header containing the signature does not get included in PE sig verification, since sig/certs are part of the header to begin with, or at least large parts of the header are not included. I vaguely remember bypass techniques that depended on this.

That's plausible and could have been implemented, by exploiting RCE vulnerabilities in jpeg decoder libraries.

Right -- I was excited that someone had figured out how to exploit the GPU to load unsigned code.

Last Commit 15 May 2015?

> Can snoop on cpu host memory via DMA

That's why we have IOMMU

You can probably still snoop on just GPU memory, scraping the screen. (Also replacing.) Big security hole, as GPUs now have fast video encoders or enough speed to OCR. Hope your driver set up framebuffer in a separated MMU area on the card. (Not in Windows they don't.)

Hopefully this cannot jump MxGPU or GVTg or GRID barrier - GPU virtualization.

Modern GPUs (AMD, Intel and NVidia) have a per process page table so the worse you can do is access memory mapped to the GPU for _your_ process. That is assuming you can guess the virtual address of those, a wrong guess and the GPU context for your process will hang and you loose everything (for the process).

So if IOMMU is disabled you would still have to be able to find a way to re-program your process GPU page table which is not map in anyway to the process (ie only accessible by the kernel driver).

So i say good luck to you sir.

What we need is a rootkit that the computer owner controls, a master of all rootkits to prevent other rootkits gaining a foothold in the first place

Hardware debugger, you can JTAG intel boards with a cheap USB cable.

Now, who put WebGL and WebCL into the browser? Just how many of GPU instructions can be made to sneakily peek/write into host's memory?

P.S. Perfect, now Google disabled the option to turn off webgl in Chrome

I think browsers are sandboxed even for GPU access. But I am now curious if persistent tracking can be done using GPU memory due to warm reboot resilience.

The sandboxing ends as soon the browser calls the GL library, and you have really no control whatsoever of what will be happening after that.

Up until very, very recently, driver coders had no idea at all that GPU instructions can be used offensively, and it is still common for buggy GPU code to crash your program/driver/os or the GPU itself.

Sandboxing is nearly useless when you let so much executable code through it.

GL library is pretty high level, so most of the functions are safe. You don’t in general specify direct GPU instructions to run, instead it is large things like “apply transform”

GL does allow uploading some code to card sometimes, in form of shaders for example, but this is fairly constrained and limited language.

Unfortunately there are some ancient but still supported old OpenGL extensions that could allow arbitrary code execution on GPU. New ones too.

E.g. uploading a fake "already compiled" shader binary in OpenGL 4 via glProgramBinary. I'd be surprised if these are actually validated. (Well more than a CRC plus length. On GCN it is CRC32. On Intel's, some hacked SHA256-alike.)

SPIR-V probably is better checked, but I wouldn't trust it farther than I can throw it. There's too many vendor and architecture specific extensions.

> I think browsers are sandboxed even for GPU access. But I am now curious if persistent tracking can be done using GPU memory due to warm reboot resilience.

You probably seen this, but in case you haven't: https://arstechnica.com/information-technology/2017/02/now-s...

I have not heard of that specific technique. Thanks.

WebGL specifies that memory is always pre-zeroed. There were implementation bugs in this in the early days though.

> Just how many of GPU instructions can be made to sneakily peek/write into host's memory?

Ideally, zero. Of course in real life this will be non-zero, but I like to believe that with the huge bounty Google offers on chrome vulns, those bugs will be fixed before they could exploit me.

(Note that the original page says "Can snoop on cpu host memory via DMA" -- this only works within the process boundary via approved APIs. So basically no extra security risks, this is exactly the same thing that CPU-based LD_PRELOAD files can do already)

how would one spoof their GPU if they wanted to?

All accelerated graphics cards (that is, all but the earliest, earliest graphics cards) have had some form of a CPU (now called a GPU) and memory on them... that is, they all can be thought of as miniature computers on your computer...

Many of those CPUs were originally constrained to only run vendor-supplied routines in vendor-supplied ROM on the graphics card -- those cards (assuming we discount the potential of Nation-State tampering) would have been more-or-less secure, by virtue of the fact that a PC supplied program couldn't interact with the graphics card's CPU (GPU).

The most secure graphics card (aside from those early ones - Hercules MDA anyone?) might be NO graphics card -- that is, you have a computer on the one hand, and an RS-232 (or other) connection to a dumb terminal, on the other...

Benefit: Any data going to or from the dumb terminal, at least could be captured and audited... And if you really wanted good auditing, you'd stay away from cursor positioning routines, like ANSI/Curses/Termcap/etc, and use straight text... but then of course you wouldn't be able to run cool programs like text emacs...<g>

Anyway, this repository is more proof (Spectre, Meltdown, Foreshadow, etc.) that that which is claimed to be secure -- might not be all that secure... which is sad, but the state of our current reality...

Which early accelerated graphics cards had ability to run code? I'd wager they had only fixed function ASICs controlled by commands from the host bus and the ROM contained code for the host CPU to run in the S3/early Matrox era.

They sometimes had ability to follow non turing complete command lists though (eg Amiga Copper programs)

Where do you see this? What you say is true in general, but this repo in particular is nothing like that. It is operating entirely using vendor providing APIs, an in particular it fully respecting process boundaries - the GPU code is only alive as long as containing process is, and only has access to the data that CPU part explicitly granted it.

Really, the only thing bad here is that some memory scan engines in antiviruses cannot monitor this. But how many memory scanning Linux antivirusesave you seen?

There is the warm reboot persistence aspect of it regarding gddr mem but yeah, this isn't some earth shattering work.just an interesting way for malicious code to be more evasive.

Reply on news.ycombinator.com