Nothing above 24GB? Not much point in upgrading, then. Memory size remains the biggest bottleneck.
I assume they will soon have a professional card announcement that includes 48GB+ cards. Assuming that the high ram cards have improvements similar to this generational leap in the gaming market, they will be in high demand.
Sure, but you dont buy a high end GPU and then buy the best one next generation... I mean some do sure but a high end GPU like this you keep for a few years...
To be fair the ability to easily purchase the cards didn't materialize either until recently. Why develop for what your consumer can't get?
But also doesn't account for poor programming. I remember a reddit post about God of War using 15gb of vram after a while of playing, probably a memory leak.
Having available vram just gives yourself headroom for newer games with higher quality textures, there was a game which name eludes me which shipped with 8k resolution textures for 4k gaming so it used like 10gb of vram? Or something along those lines.
But it can also be used for pre-caching stuff, like if you were playing WoW and the next zone was already precached so you don't need to load it from the disk.
I think that new tech Sony and MS were working on, direct storage access? Will play a bigger and better role than higher vram.
I only game at 1440p so I don't care for more than 8gb of vram.
horizon zero dawn, notably, was already VRAM-limited by 10gb cards several years ago and would run lower-quality assets on a 3080.
> Not so cheap if you're just using it for gaming
I am not even sure if there is any recent game that can utilize 24GB of a 3090.
Honestly everything VR takes up huge amounts of vram, it's not possible to run No Man's Sky at max settings with only 8gb vram, and you still might need to dial down some settings with 12gb vram.
This is with a Quest 2, I'm guessing headsets with higher resolution will be even more demanding.
It's not just double the memory, is faster memory as well. Although, less CUDA cores and also doesn't include latest NVENC encoder/decoder, nor AV1 encoder, so... Tradeoffs I guess :) For some people it will be worth it, for others no.
For what? AI shit? These are for games broski
the hardware for low-precision matrix multiplications in ML is mostly independent of the hardware that’s typically utilized for vertex and fragment shaders in computer graphics. It really only gets used for graphics if using DLSS. It’s not used at all for (integer) computations in cryptomining.
Are they for high-precision scientific simulations because it’s possible to multiply matrices on them? These cards are probably quite shit at double-precision matrix multiplications.
Yes and they make Quadros for those :)
I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.
For inference? You can’t generally shard an AI across multiple cards, but you can run multiple instances. Depends on what your trying to do.
In no case will it be a simple linear speedup, and you certainly can’t add the memory together.
Speed is generally not a serious issue:
- I'm not trying to scale to millions of users on inference; it's a dev machine
- Training is a bit of fine-tuning; I'm not building big models from scratch.
- Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run.
- It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time.
I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration.
Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models.
I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.
I hope they come out with 4090TI with more memory though