Hacker News Reader

Comment by amelius | Hacker News Reader

That's overkill for any type of graphics application though.

What you want is a DL or parallel compute card, not a graphics card.

They are far more expensive though because compute doesn't sell to the average consumer like graphics does.

A modern graphics card already is a parallel compute card. A modern graphics pipeline is mostly compute with only very specific stuff using fixed function functionality.

Yes, but that was not the point. The point is that they are sold as graphics cards, not compute cards. Therefore, expect these cards to be good at (and have the memory for) typical graphics operations.

I don't entirely agree, especially with the popularity of LLMs nowadays. While graphics are still a big portion of the pitch, compute oriented features are increasingly expected as part of the offering, especially at the higher end of the lineup. If you've bought a 3090/4090, you likely had in mind more computationally oriented tasks compared to if you've bought a 1660 or AMD card.

Are the ray tracing cores in Nvidia cards used for compute?

All that the ray tracing cores do is accelerate calculation of BVH intersections. The rest of the tracing (ie the part that gives meaning to the BVH intersection result) is done via compute, with the RT cores being able to be 'invoked' essentially arbitrarily. This is in contrast to the way graphics cards used to be, where you had certain stages for certain things, designed specifically around producing graphics, making the 'graphics' part of the name hold more weight than nowadays.

I'm not sure if you're asking if they're used for a compute application or if you're asking if they can be used from the same execution path as 'compute' code.

Seen from first principles, BVH is quite close to some kinds of hierarchical clustering. The ray intersection part can be understood in turn as a kind of normalization, or projection to the unit sphere. So depending on how much hardware is actually reserved to this task on-chip, there might be ways of making it somewhat useful in the ML context.

Just asking if those cores tend to sit unused when "standard" compute jobs are run on the cards. :)

Meh, that's like saying 640kb of RAM is more than anyone would ever need. Demand follows hardware development, which in turn accelerates demand. I'm sure game developers would easily find a way to use 128GB of VRAM if it was commonly available in their target market.

To be fair, 640kb is more than anyone really needs. It's just far, far less than we want.

Can you imagine the download sizes?

Turns out that demand for graphics memory got stuck at a point where compute is still hungry for more.

That may certainly change but it doesn't help compute much, today.

It's supply that's down - Nvidia is strictly enforcing their artificial segmentation where the jump from 24GB to 40GB of VRAM multiplies the price by around 5 (!).

They know that if you need a 40GB card, you probably can't use a 24GB card at all because it'd run out of VRAM. They charge such insane prices for those higher capacity cards, because they know that they can make you pay thousands upon thousands for every tiny scrap of extra VRAM, and you will have literally no choice but to pay that price because you need the VRAM.

Demand isn't low, demand is actually so high that they don't care if lowly consumers want VRAM, they already have enterprise customers that need it so badly that they'll pay prices higher than most consumers would ever imagine.

And that's why they're so stingy with VRAM on the consumer cards. They need to be careful not to accidentally make them useful in the datacenter, so they can ensure that their enterprise customers continue to be forced to pay those extortionary prices.

You see, it's all about the money, and it always will be. Capitalism, baby!

You need a special license to run nVidia cards in the datacenter, so this rules out consumer gaming cards already.

There are already ways to get around this. For example, renting compute from people who aren't in datacenters. Which is already a thing: https://vast.ai

This isn't an argument against giving users this much VRAM, but I'm pretty sure they'd just drop the whole game into VRAM and call it a day instead of actually optimizing.

Isn't that exactly how things improve? When you can reduce the cognitive load on bookkeeping, you can accomplish more productive work.

I can produce a featureful webapp in Python-Django solo because I do not have to worry about optimally filling registers and CPU cache. As you cut deeper to the hardware limitations, you have to be significantly more cognizant of writing to the hardware constraints than solving the business problem.

Taken to the extreme, we have Electron applications consuming multi-GB of RAM, but it does expand the universe of possibilities.

> Isn't that exactly how things improve? When you can reduce the cognitive load on bookkeeping, you can accomplish more productive work.

What are you talking about? Sure you can appear to be more productive if you are reckless and wasteful of resources, and adding more resources means there is more that you can afford to waste.

But for the past decade or so we have been seeing what happens if you teach every software developer that it's okay to waste resources: resources get wasted.

Back when you couldn't afford to waste resources, you'd see extremely skilled optimizations and extremely clever hacks to get advanced software to run on extremely slow and primitive CPUs. But ever since computing power increased by a couple orders of magnitude, what happens? The exact same software implemented by today's developers would more than compensate for the increased computing power by being a couple orders of magnitude less efficient.

Sure it expands the universe of possibilities. I do want more resources, I really do. But "reducing the cognitive load on bookkeeping" is making software get slower faster than hardware gets more powerful. Which is absolute insanity to me.

Think of all the software which does not exist if we are still writing in assembly because we are constrained by the hardware. Python, RPGMaker, Roblox, PICO-8, etc are no longer possible because they are too wasteful. Having bounty enables art to be made which is impossible if creators are bit twiddling.

Do I bemoan that my 2000s era super computer squanders resources to load 20 kb of text? Sure, but it is also amazing the things that are possible because tools are not restricted to the priesthood of assembly programmers.

> Think of all the software which does not exist if we are still writing in assembly because we are constrained by the hardware.

There is an entire spectrum between "640k ought to be enough for anybody" and "this app needs its own fully independent copy of an entire browser engine just to display a couple screens of text".

About Electron: Operating systems provide their own system-level webviews. Use those. Tauri uses those; Tauri apps are absolutely tiny and use very little RAM, because the system-level webview doesn't have to duplicate the overhead of an entire browser engine for every single app that uses it. It typically only has to duplicate the overhead of a single page.

But that is for apps that even need a webview. For apps that don't, you don't have to drop all the way down to assembly to pick something like WinForms. Even WinUI 3 isn't always terrible, at least compared to webviews.

Hell, I wonder if even Python + wxPython would be more efficient than using a webview. Honestly, none of the software you mentioned is "too wasteful", or at least those aren't what I was referring to with my original comment.

Have you heard about Microsoft Teams, and how it can take up to 10 seconds just to display the splash screen, even on a decently fast computer? I'd like to see you try to tell me that it would have literally killed them to optimize it a bit more. Just think back to, say, the IRC clients from back in the Windows 7 days where you wouldn't even have to blink before they finished opening up.

Another example is just how few simple optimizations I see in code now. Something like

   `while i < someFunction():`

Instead of

   ```
   criteria = someFunction()
   while i < criteria:
   ```

I've seen simple lines like this compound in complexity as someFunction() gets more complex or as the loop increases. There's a billion variations on the idea but I think it is because people aren't thinking about how someFunction() is getting called every iteration. I do think every programmer should have some basic optimization skills just because these types of things would be nonobvious otherwise. Because frankly, this kind of optimization is going to have no real difference in a toy program or even in many isolated test cases but it can easily add meaningful latency to a program. Definitely more egregious examples but just trying to think of something that is extremely low hanging fruit.

The real issue comes when the user is doing things more than just your program...

wouldn't any compiler optimize this anyway?

No, and a lot of python code is written these days.

I have this theory that if you remove too much friction then things get worse. Because when there's friction it makes people cognizant of limitations. But when there's not enough friction it becomes easy to sweep aside or ignore. You can be too reliant on your metrics. Idk if this actually makes sense but maybe there's a nugget somewhere in the idea.

> Isn't that exactly how things improve?

There's a joke "20 years of software progress has completely undone the 20 years of hardware progress."

Essentially the joke is about how people don't optimize software like they used to. There is some tongue in cheek though because there are more platforms and variations that we have to account for these days but there is also truth that there is some laziness. I mean I have a laundry app on my phone that takes 5s to load each time because it redraws the home screen while it is trying to connect to the network and find the machines but won't use the bluetooth or nfc chips that both my phone and the machines have.

I do think there is the case to move fast and break things, but at some point you got to slow down and fix things. There should be cycles. Don't forget that debt has interest and that interest compounds. If you have enough debt you'll be "moving fast" but being fast in molasses isn't fast. It's kinda why a lot of software have just declared bankruptcy and started over (e.g. "rewriting in rust"). We should definitely encourage people to hire a mixture of those that move fast and those that move slow. There's harmony there but I think we've forgotten that.

cough Apple Silicon cough cough

Unified memory is honestly a really, really good idea.

Supposedly, if you have professional Apple hardware experts hand-optimize a machine-learning model, then Apple's most powerful M3 chips can compete against, say, a typical/unoptimized PyTorch implementation of the same model on a 4090.

Except the M-series chips can currently have up to 8 times the memory of the 4090, and they don't have to worry about shuffling things in and out of VRAM. The GPU can just work with any of the data that the CPU already has.

I'd be hard pressed to buy a comparison to a 4090. But yeah, things would be better if they didn't "dump everything into VRAM"

They're comparing incredibly hand-optimized MLX inference on the Mac to off-the-shelf stuff on the 4090. So I'd believe it just because the 4090's being given an artificial disadvantage, and the Mac is being given meticulous hand-optimization by the foremost experts on the very platform it runs on.

In other words, the Mac is being given a huge optimization advantage compared to the 4090. And it's roughly competitive.

Even that is no mean feat.

Source: https://appleinsider.com/articles/23/12/13/apple-silicon-m3-...

Reply on news.ycombinator.com