Nvidia pursues $30B custom chip opportunity with new unit

NVIDIA isn't missing the mark on the programming model and toolkit framework (PTX and forward/backward compat) either. They have a good, lean gpu design with a lot of features and a good programming model and ecosystem etc.

You're right, it's not just the matrix math, that's not rocket science, but there's a ton of little glue code around it. And you need something GPU-like for that anyway, plus a bunch of scheduler and shader-execution-reordering stuff for your tensor threads and glue code, etc. You end up with something broadly similar to a GPU anyway.

It's the ProgPOW theorem, right? That there is not some major gain to be squeezed by implementing a smaller/different machine on the instruction set. That GPUs are relatively close to some kind of computational optimum for parallel workloads (in terms of programmability/flexibility and performance).

NVIDIA's model isn't far off the global optimum imo, it's certainly in a great local minimum, and that's really true of a lot of their designs these days. It is always a little wild how everyone trivializes the idea that AMD/etc are going to catch up with some 80% solution in RT or tensor etc... like just maybe NVIDIA did the math and figured out what they think a reasonable ray performance level is, and how much they'd need to upscale, and what parts of the pipeline make sense to have accelerated by units vs emulated on shaders/etc, and there's not some massive gain to be squeezed by just putting a handful of devs on a project for a year?

Same thing for prices too. Everyone wants to assume that AMD is just choosing to follow them in gouging or whatever. The null hypothesis is that both nvidia and AMD are subject to the same industry cost trends and can’t actually do significantly better (not like 2x perf/$ or whatever), and that nvidia is in some kind of reasonable price structure after all. People are going to find that a lot of electronics prices are going to go up in the coming years. There’s no more 1600AF for $85 or 3600 for $160 either, or Radeon 7850 for $150 etc.

Not talking about original 4080 pricing etc but actually 4070 and 4060 are fairly reasonable products, and 4070 quickly fell even further below msrp. 7800xt and 7900xt and 7600xt are all fine as well. That’s about what the price increases have been since the last leasing-edge products.

Although, as mentioned, works under a cycloptic vision where the card is only doing compute tasks. I'd be interested to know if even supported cards can multitask GPU and pure compute tasks without crashing because it looks like it might be a design issue. Hard to tell with driver corruption. Maybe the testing catches that on supported cards; who knows.

[0] https://www.britannica.com/event/World-War-II/Human-and-mate...

[1] https://moneywise.com/life/lifestyle/financial-facts-about-w...

[2] https://www.whatitcosts.com/world-war-ii-cost-united-states-...

Turing and ampere as a whole were already below the price increase curve due to the trailing node. Ampere certainly was a massive attempt at passing the savings through - 3060 ti and 3080 were both really aggressively priced and were great overall products. Notoriously, 3060 ti was so aggressively priced that partners didn’t even want to make it at msrp…

And Turing is another example of the “AMD is subject to the same overall industry price trends/not able to drastically beat nvidia pricing either” too - sure, AMD leapt ahead to 7nm early with 5700xt… and this was more expensive than Nvidia’s solution with Turing, such that AMD was not able to undercut Nvidia’s pricing. Those cost increases have been eating up the density gains for a long time now.

Nvidia is really actually doing the things that keep the costs down, and ampere and Turing indeed bent the cost curve below where it would otherwise have been. It’s obviously not going to stay fixed, the era of $100 and $150 gaming gpus being serious contenders has obviously long since come to a close, and now it’s happening for $200-300 gpus too, $500-600 is where a $300 gpu was 10 years ago.

People just don’t have any way to objectively determine what “true” prices should be, and refuse to believe the true price is rising because it offends their “gamer” worldview. It has to be gouging, otherwise I don’t get a new Sony GameStation X every 2 years at the exact same price, and that can’t be possible.

https://www.pcgamer.com/amd-moores-law-aint-dead-its-just-a-...

The other cloud gaming providers need to do the same thing, and they either cost more and/or have worse hardware and/or have publisher relationships. Xbox and PS streaming use actual consoles and are much less powerful, for example, Shadow costs a lot more for worse hardware, Luna works with publishers, Stadia just gave up before ever really trying (lol), etc.

Maybe in Nvidia's case, at least they can get the GPUs at cost, and maybe use them for non gaming async workloads during off peak times? I dunno. I just hope they keep GFN going; it's completely replaced my desktop PC.

So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.

Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.

They know that if you need a 40GB card, you probably can't use a 24GB card at all because it'd run out of VRAM. They charge such insane prices for those higher capacity cards, because they know that they can make you pay thousands upon thousands for every tiny scrap of extra VRAM, and you will have literally no choice but to pay that price because you need the VRAM.

Demand isn't low, demand is actually so high that they don't care if lowly consumers want VRAM, they already have enterprise customers that need it so badly that they'll pay prices higher than most consumers would ever imagine.

And that's why they're so stingy with VRAM on the consumer cards. They need to be careful not to accidentally make them useful in the datacenter, so they can ensure that their enterprise customers continue to be forced to pay those extortionary prices.

You see, it's all about the money, and it always will be. Capitalism, baby!

I can produce a featureful webapp in Python-Django solo because I do not have to worry about optimally filling registers and CPU cache. As you cut deeper to the hardware limitations, you have to be significantly more cognizant of writing to the hardware constraints than solving the business problem.

Taken to the extreme, we have Electron applications consuming multi-GB of RAM, but it does expand the universe of possibilities.

Unified memory is honestly a really, really good idea.

Supposedly, if you have professional Apple hardware experts hand-optimize a machine-learning model, then Apple's most powerful M3 chips can compete against, say, a typical/unoptimized PyTorch implementation of the same model on a 4090.

Except the M-series chips can currently have up to 8 times the memory of the 4090, and they don't have to worry about shuffling things in and out of VRAM. The GPU can just work with any of the data that the CPU already has.

I'm not sure if you're asking if they're used for a compute application or if you're asking if they can be used from the same execution path as 'compute' code.

You didn't account for the "hiccups", which can vary from 5-20 years until competition catches up, longer than the life of many companies. In spherical cows worlds of economics that would be just a hiccup.

No, it's not anywhere near maxed out anymore. The chip shortage has been over for a while; now there is too much supply.

But that's not the issue. The issue is that Nvidia really wants to force these enterprise customers to pay extremely high prices. This is why they are so afraid to give any more VRAM to the consumer chips, because if they were at all suitable for VRAM-heavy workloads, every HPC company would buy a couple $2,000 consumer cards in place of each $15,000 datacenter card. That'd lose Nvidia something like 70% which they would find entirely unacceptable.