LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows | Hacker News Reader

LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows

blogs.nvidia.com

69 pointsseaal33 comments

In other news: GeForce experience continues to require an account and has creepy privacy issues

sorry bud, we cant make your games super duper fast if you dont give us your email!

email? They’re sweeping up an incredible amount of data, basically every window you open.

Do you have more info on that? Their support site[1] claims their data collection is much more constrained than that (although still seems like a lot).

1: https://nvidia.custhelp.com/app/answers/detail/a_id/3188/~/w...

Ah, you’re right! I’m basically repeating stuff I got from a somewhat recent HN thread [1] which I found shocking when I first read it, except it’s not true and I did not bother checking it because it was not unthinkable and… confirmation bias, I guess. Now, if only I could go back and still edit the original post…

Thanks for catching my bullshit, I stand corrected (and should have been wiser).

[1] https://news.ycombinator.com/item?id=37033859

This is the HN I love :)

No worries, confirmation bias comes for us all at some point!

It is less convenient (by design), but you can always manually look for new drivers and download them. Thats how I do it.

Maybe someone wrote a script that automatically scrapes the website and downloads the newest driver automatically?

TechPowerUp's nvcleanstall is a pretty lean NVidia driver installer. It fetches from TPUs servers, and installs only user-selected driver components. The screenshots linked do a good job of communicating what it does.

Has a scheduled update-check if you want that too.

https://www.techpowerup.com/nvcleanstall/

Where is AMD in all this? Like M1 Macs, their Ryzen APUs have access to the whole system memory and AI accelerators, so I would like to get a laptop with 32-64GB of RAM just to fit giant LLMs in it, instead of splurging a lot more on RTX cards with large VRAM even if I don't game, but AMD isn't even trying.

https://www.reuters.com/markets/deals/amd-acquire-ai-softwar...

Sure, but Apple laptops can run Llama right now. How many years do we have to wait till AMDs aquisition efforts materialize in products we can buy?

I dont understand, are you claiming non-Apple devices cannot run LLMs?

https://github.com/ggerganov/llama.cpp/issues/34

If you meant eGPU support, IIRC that is beta for everyone right now.

DDR4 and DDR5 are nowhere near as fast the memory in Apple's SoCs or GDDR6X found on modern GPUs.

But AMD can in theory also use the same DRR that Apple uses. There ale laptops already with LPDDR5X-8000 frequency.

Really? Wikipedia says DDR5 supports 8 GT/s and GDDR6X supports 16.

I mean I guess they go a lot wider on a GPU, but there's no reason an APU couldn't do that too.

Yeah, sure in theory they could build chips with 384 bit memory controllers for that sweet bandwidth, but something tells me they won't.

They already do that for the top end threadrippers pros. AMD Threadripper Pro 5995wx for example has 8 memory channels that's a 512-bit memory bus.

That's hardly comparable to an M1 is it?

Not to an M1 ultra. That has an 1024-bit memory bus. But a M1 max has 512-bit as well. Now a standard M1 has just a 128-bit memory bus.

I'm looking forward to CodeLlama being usable on a Dev system using some CPU offloading and only 32GB _while developing in an IDE_. Code Llama is great but using it alongside an IDE just kicks swapping into overdrive, and at least this will mitigate some of that loss on the LLM generation side (at least for the part in GPU ram).

Do you use https://github.com/xNul/code-llama-for-vscode or something else?

Haven’t found any good setup instructions for Linux or my Google skills are failing me.

I use https://github.com/oobabooga/text-generation-webui separately as I code in VS 2022, VS Code, Intellij Idea and Webstorm

I just setup vanilla llama.cpp this past weekend - good place to start as an intro IMO

Latest 545.84 released 10.17 NVIDIA drivers have impressive performance improvements.

Noticeable for gaming on a 3090? Working my way through Phantom Liberty DLC and massively enjoying it.

These types of optimizations are what I figured Carmack would be doing at Keen. He's so good at optimization and low-level hardware understanding that I thought he might be able to wring some small extra percentage out of existing hardware for AI tasks. With compute currently as expensive as it is, that would be extremely valuable.

I assume they released equivalents for Linux?

was wondering about this

It appears that it's already in there. The trick is that you have to install an extension in some apps to make it work. For instance, there's a TensorRT accelerator extension for Stable Diffusion WebUI that takes advantage of it. I'm installing it right now, though the docs leave a bit to be desired.

https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT

I got it installed and tested with an SD 1.5 model in Stable Diffusion Webui using a 4090. (SDXL models did not work.) I generated the same test prompt with and without their TensorRT acceleration. The generated images are nearly identical, with very minor, almost imperceptible differences, which is pretty standard for techniques that optimize the attention layers.

Prompt: A cat wearing pajamas Model: deliberate_v2 Size: 768x512 Batch Size: 4 Clip Skip: 2 Samler: DPM++ 2S a Karras Steps: 70 Seed: 2024828515

Generation times avg over 5 runs

================================

without TensorRT: 19.2 seconds

with TensorRT: 12.3 seconds

Benchmarks

Batch Size 1 / 2 / 4

=======================================

without TensorRT: 31.11 / 36.23 / 42.34

with TensorRT: 55.32 / 58.06 / 62.27

It's faster, but clearly it's not 4x faster. I suppose they cherrypicked benchmarks against generation techniques not using xformers or SDP Attention. Also, it appears to be limited to a max batch size of 4.

Does fix the regression that was causing poor stable diffusion inference ?

does this include wsl or just regular windows?

Reply on news.ycombinator.com