Hacker News Reader

Comment by treme | Hacker News Reader

Crazy that video game graphics business happened to be the secret level portal to AI land

Watch this 90s commercial for 3dfx: https://www.youtube.com/watch?v=ooLO2xeyJZA

GPUs always had more compute / Gigaflops than traditional computers. GPUs in fact have more to do with 80s-era supercomputer architecture than normal CPUs.

https://www.youtube.com/watch?v=ODIqbTGNee4

I'm not sure if it's just nostalgia, but I still like the final 3dfx logo, at least in a video game context.

That was an unexpected twist.

Ever since GPUs became programable, which goes earlier than many think (see TMS34010 from 1986), there has been many attempts to use the cards for general purpose compute.

It turns out that anything related to neural networks, and similar AI approaches, is all about compute.

I'd say almost the opposite in practice.

GPUs, as they became programmable (and maybe even a little bit before that...), started to take cues from SIMD Supercomputers. So the compute methodologies were researched first, and then GPUs (ie: applications to graphics) were applied afterwards.

I've heard rumors that the first programmable GPUs were considered because GPUs already were in SIMD-style compute and running instructions in a programmable way at the hardware/firmware level. It just needed to be "revealed" to OpenGL or DirectX programmers.

TMS34010 used a C like SDK, in 1986, there were hardly any SIMD supercomputers, unless you consider Cray and Connection Machines as model.

I'm thinking closer to 90s period, where "graphics cards" developed by Voodoo / 3dfx and its Glide API were clearly along the DirectX / OpenGL inspiration path. (3d Graphics pipeline / transform & lightning / etc. etc.). I don't know the exact point when these 3d accelerators became programmable with vertex/pixel shaders, but Glide is clearly on the path and part of the history here.

--------

You're right in that 80s / Arcade chips look to be different. Its a programmable graphics chip alright, but I wouldn't call that chip SIMD, not from what I can see from Wikipedia at least.

I wasn't the one bringing SIMD into the picture. :)

In hindsight I think it makes good sense - game graphics were always aiming to represent worlds with as high fidelity as possible.

Good point, but it is interesting that the computations to render visual worlds in high fidelity is the same computation to "ingest" the data to create a model.

Reminds me of how a mic is a speaker and speaker is a mic.

I think it speaks to the generalizability of linear algebra.

nVidia claims that they went into video games because it provided a way to fund their goal of compute accelerators

I don't believe that claim one bit. That reads like revisionist history if I've ever seen it. If there's sources that back it up, fine, but until I see pretty reasonable proof I'm going to take that as CEO grandstanding while the market is hot.

Nvidia has been peddling compute for decades. They've long been selling enterprise tier cards for industrial purposes that are ill-suited to games. In many ways CUDA has been their long running attempt to break out of gaming.

I'm not surprised here. They invested and invested and invested and now everyone is playing catch up while they nearly corner the market. In industry circles we've been lamenting their dominance in this area for well over a decade.

Google and Apple are the only companies I see that have a shot at disrupting their market but they tend to end up navel gazing instead of burning bridges. And Nvidia has bridges with everyone.

It's absolutely an industrial geopolitics mastercraft scenario. We'll be studying them for a century.

Everyone has long known that the secret to unlocking the next tier of performance is solving concurrency. Look at all the advances that have made it into production in the last twenty years. But threading models only get you so far on a CPU, and programming GPUs is comparatively very difficult. Nvidia has always aimed to displace the CPU's prominence, since day one.

It's horizontal scaling 101.

They've been peddling compute since mid-2000's, however Nvidia has been around for much longer. To the revisionist history point, I'm unfamiliar seeing any of this prior to that. Unless I'm missing something, they were very much a graphics only company competing with 3dfx, Matrox, S3, ATi, etc

They were founded in 1993, but until 1998 had only put out 2 chips, the first was a quadratic surface renderer that didn't sell well, and the second was a normal triangle rasterizer that saved them from bankruptcy. Then in 2001 they released the first GPU that could run code, each subsequent generation expanding on that capability until they were able to do general purpose compute with CUDA.

Nvidia was selling cards with hardware acceleration since 1997. The first one I own is from 1998, and its advantage wasn't that it had superior compute, but that it also included the non-3d circuitry of the rest of the video card. So instead of having a 2d card and a Voodoo, which were daisy chained, you could use that a lone Riva TNT that will do both. So they started being less about compute than the competition.

Between the first nvidia cards and the first release of CUDA, you have a decade.

> So instead of having a 2d card and a Voodoo, which were daisy chained, you could use that a lone Riva TNT that will do both

Or get the slightly underpowered Banshee.

Decades is still long after the first GeForce. Before CUDA we had third parties using graphics primitives to run general purposes computing.

So I'm not buying it.

You can find slides from IIRC one of their first GPUs talking about how they can be made into very capable compute accelerators. They've supported CUDA on most of their GPUs since 2007 while the competition is still struggling to put out a truly equivalent solution. Even before that they had Cg.

I think it's pretty obvious that they saw the writing on the wall about the value of GPUs for massively parallel computing and actively pushed to be in the right spot for when everything took off.

And probably they'd been at least thinking about that since the early 2000s to have it ready by then. But the claim is that they were thinking about it since their beginning in 1993, and that's the part that just seems hard to believe.

That's the claim Jensen has made, and while there isn't any particular evidence besides his word to go off for the founding goal, it's worth considering that between 1993 and 1998 they only put out 2 chips. The first one was quadratic primitive based (which, sounds more focused towards a future of compute applications, due to the ease of representing equations in such a form), and the second one was a last ditch effort triangle primitive chip to avoid bankruptcy after DirectX settled on triangle primitives. So I don't really think those years count for much in terms of determining what their goal was.

Their release cadence only really picked up in 1998, and just 2-3 years later they had the first 'programmable' GPUs with early shaders.

As such, saying that NVIDIA wasn't always aiming for GPU compute feels kinda like saying that SpaceX wasn't always aiming for feasibly reusable rocketry because it took them ~15 years to build up to their current version of it.

Also a funny 'existential' point to remember. You have a clean idea (doing math / quad-splines) but the market sinks it.. doesn't mean it won't be good later. It's a game of long and patient grind.

It's not hard to believe.

The idea that future of computing is related to parallel numerical computing and linear algebra (HPC, graphics ,scientific computing, data science, machine learning) predates current deep learning boom.

Already in 2003 and GPGPU (General-purpose computing on graphics processing units) was a thing. OpenVidia came out in 2003.

They tried to do physics acceleration at one point. I think they were trying to find a niche for their DSP-style compute.

> They tried to do physics acceleration at one point

Well, there is PhysX which they bought in 2008 and still have; when they bought PhysX they shifted the acceleration implementation from relying on dedicated physics accelerator hardware to GPGPU.

There’s APEX which was built on top of PhysX but later discontinued.

There’s FleX…

“At one point” might be a slight misstatement.

I'm 50/50 on this. I don't think they ever envisioned (npi) that gaming would grow that much and would allow them to pivot back into GPU based HPC. That said they might always had a dream hinting at this.

What is so hard to believe? Technology evolves rapidly. I can't imagine that anyone investing big money in compute technology wouldn't have expected that.

Because that happened in a time when there were many players in the 90s each making their own GPUs for gaming purposes specifically. Compute was not even on the picture until things like CUDA and OpenCL came out.

Back in the late 90s, there was a project at SGI, called Bali, to make all their pipelines work in IEEE 32-bit floating point (they were using Intel i860 chips) so that they could do HW rendering of scenes written in Pixar's Renderman language.

Sony copied that idea for the 1st Playstation, and then folks like NVidia & 3DLabs quickly followed suit, the idea being they would enable that functionality for games like Final Fantasy.

In the early 2000s, the HPC folks realized that you could use a GPU for physics & engineering codes, and here were are 20 yrs later.

You do understand nVidia dates back to fixed pipeline accelerators for essentially some vertices and textures, right?

Of course, obviously ever since inception of the company they had the noble goal of uplifting human race. It was just the set of unfortunate circumstances that forced them to make trivial utility devices just to survive long enough

They did a lot on medical scan visualization pretty early on too.

"When we were selling shovels to coal miners, we always secretly knew there was gold in those hills."

Yeah, sure.

Highly doubt this was the initial idea for nvidia given they were graphics-only for a very long time. CUDA definitely felt like more of a value-add for the first 5-6 years than a concerted effort to build accelerators and to fund that with graphics demand. First "tesla" line of GPUs - which had very little compute-only focus - were in 2007.

>First "tesla" line of GPUs - which had very little compute-only focus - were in 2007.

CUDA was first released in ~2006 but started at least in 2004, largely building on top or the momentum of Cg which was released in 2002 and being worked on since 2000. There was at the time in the late 90s about general purpose parallel programming and how it could possible be done on GPUs or doing it on something like PS2 Cell.

I dont know if Nvidia was really started with CUDA in mind in 1993. But Nvidia was into CUDA like GPU usage WAY ahead of anyone else in the field.

Wasn't there a time when people were hacking shaders to do compute tasks and read the results back instead of displaying to the frame buffer?

Yes, multiple such attempts, including complete languages. Very thin hardware support as different GPUs varied very much on what could be done.

GeForce 8 introduced CUDA because graphics pipeline similarly evolved into position where instead of forcing the use of dedicated cores for specific tasks (vector and fragment shaders) due to simple economies of chip making, the Shader Model 4 introducing yet another stage justified making unified shader cores... which could be just as easily used for other tasks.

Cell was PS3 (2006)

And tell me what is Nvidia trying to fund with the AI business?

Then why does my graphics card have 5 video outputs?

Nah, Intel said Nvidia just got "lucky"

So says every loser when they are bested by their own ignorance.

Does a series of matrix multiplications have a soul?

Matrix multiplication is just an effective representation for the dense graph structures. The same graphs could be implemented in other ways (adjacency list, edge list), perform the same overall logic, without doing vector/matrix/tensor math. It seems like the magic is in the overall idea of neurons and networks of them following increasingly interesting architectures and training mechanisms.

But because these graphs are mostly dense and involve numerical operations, matrices/tensors are a great implementation.

Does a series of electrical impulses in a brain?

Obviously no, because signals take time to move from A to B, while humans live in the now.

Define "soul".

does a SQL table provide a stochastic representation of conceptual symbolics?

oh, sorry, I thought we were just asking questions

Does a bunch of atoms arranged into proteins have a soul?

Reply on news.ycombinator.com