Hacker News Reader

Comment by jmward01 | Hacker News Reader

12GB max is a non-starter for ML work now. Why not come out with a reasonably priced 24gb card even if it isn't the fastest and target it at the ML dev world? Am I missing something here?

View on HN

I think a lot of replies to this post are missing that Intel's last graphics card wasn't received well by gamers due to poor drivers. The GT 730 from 2014 has more users than all Arc cards combined according to the latest Steam survey.[0] It's entirely possible that making a 24gb local inference card would do better since they can contribute patches for inference libraries directly like they did for llama.cpp, as opposed to a gaming card where the support surface is much larger. I wish Intel well in any case and hope their drivers (or driver emulators) improve enough to be considered broadly usable.

[0] https://store.steampowered.com/hwsurvey/videocard/ - 0.19% share

> Am I missing something here?

Video games

It's insane how out of touch people can be here, lol

I have been trying to hold my slurs in reading this thread.

These ML AI Macbook people are legit insane.

Desktops and gaming is ugly and complex to them (because lego is hard and macbook look nice unga bunga), yet it is a mass market Intel wants to move in on.

People here complain because Intel is not making a cheap GPU to "make AI" on when that's a market of maybe 1000 people.

This Intel card is perfect for an esports gaming machine running CS2, Valorant, Rocket Leauge and casual or older games like The Sims, GoG games etc. Market of 1 million + right there, CS2 alone is 1mil people playing everyday. Not people grinding leetcode on their macs. Every real developer has a desktop, epyc cpu, giga ram and a nice GPU for downtime and run a real OS like Linux or even Windows (yes majority of devs run Windows)

Intel GPUs don't sell well to gamers. They've been on the market for years now.

>market of maybe 1000 people

The market of people interested in local ai inference is in the millions. If it's cheap enough the data center market is at least 10 million.

Yes, Intel cards have sucked. But they are trying again!

Yes, everyone knows the best way of making a product not suck is to give up and never try again.

> Intel GPUs don't sell well to gamers. They've been on the market for years now.

Intel has only had discrete GPUs on the market for 2 years. I guess that is a plural number of years, but only barely.

Most devs use windows (https://www.statista.com/statistics/869211/worldwide-softwar...). Reddit llocallama alone has 250k users. Clearly the market is bigger than 1000 people. Why are gamers and Linux people always so aggressive diminutive of other people’s interests?

>Why are gamers and Linux people always so aggressive diminutive of other people’s interests?

Both groups have a high autism %

We love to be "technically correct" and we often are. So we get frustrated when people claim things that are wrong.

How big is NVIDIA now? You don't think breaking into that market is a good strategy? And, yes, I understand that this is targeted at gamers and not ML. That was the point of the comment I made. Maybe if they did target ML they would make money and open a path to the massive server market out there.

A video card that beats the 4060 for under $250 is very much going to be a problem for AMD and is going to eat the "low end" market if it is reasonably stable.

ML is about hit another winter. Maybe intel is ahead of industry.

Or we can keep asking high computers questions about programming.

> ML is about hit another winter.

I agree ML is about to hit (or has likely already hit) some serious constraints compared to breathless predictions of two years ago. I don't think there's anything equivalent to the AI winter on the horizon, though—LLMs even operated by people who have no clue how the underlying mechanism functions are still far more empowered than anything like the primitives of the 80s enabled.

Yeah… I want to think of it like mining, where you’ve found an ore vein. You have to switch from prospecting to mining. There’s a lot of work to be done by integrating our LLMs and other tools with other systems, and I think the cost/benefit of making models bigger, Bigger, BIGGER is reaching a plateau.

I think there'll be a "financial" winter - or another way a bubble burst - the investment right now is simply unsustainable, how are these products going to be monetized?

Nvidia had a revenue of $27billion in 2023 - that's about $160 per person per year [0] for every working age person in the USA. And it's predicted to more than double in 2024. If you reduce that to office workers (you know, the people who might actually get some benefit, as no AI is going to milk a cow or serve you starbucks) that's more like $1450/year. Or again more than double that for 2024.

How much value add is the current set of AI products going to give us? It's still mostly promise too.

Sure, like most bubbles there'll probably still be some winners, but there's no way the current market as a whole is sustainable.

The only way the "maximal AI" dream income is actually going to happen is if they functionally replace a significant proportion of the working population completely. And that probably would have large enough impacts to society that things like "Dollars In A Bank" or similar may not be so important.

[0] Using the stat of "169.8 million people worked at some point in 2022" https://www.bls.gov/news.release/pdf/work.pdf

[1] 18.5 million office workers according to https://www.bls.gov/news.release/ocwage.nr0.htm

Well, "AI" is milking cows. Not LLM's though. Our milking robot uses image recognition to find the cow's teats to put the milking cup on.

Yeah, but automated milking robots like that have been in the market for more than a decade now IIRC?

Seems like a lot of CV solutions have seen fairly steady but small incremental advances over the past 10-15 years, quite unrelated to the current AI hype.

Improving capabilities of AI isn't at odds with expecting an "AI Winter" - just the current drive is more hype than sustainable, provable progress.

We've been through multiple AI Winters, as a new technique is developed, it does increase the capabilities. Just not as much as the hype suggested.

To say there won't be a bust implies this boom will last forever, into whatever singularity that implies.

I think the more accurate denominator would be the world population. People are seeing benefits to LLMs even outside of the office.

How do LLMs make money though?

> I think the more accurate denominator would be the world population. People are seeing benefits to LLMs even outside of the office.

For example ?

(besides deep fakes)

I use one in the kitchen, because it's easier than the ads and prose on most recipe websites, and it can adapt to whatever ingredients I actually have rather than being a fixed list.

Used them in the garden while weeding and in the garden store while planning what to plant, in both cases to identify plants by image and tell me about them — though I'd say image capable AI are no longer mere "large language models".

Used ChatGPT while shopping to help me locate products in the store I was in, when I couldn't find just by wandering the isles, by uploading a photo of the aisle I happened to be in at the point I gave up.

While I'd agree monetisation seems to be a challenge in the long term (analogy: spreadsheets are used everywhere, but are so easy to make they're not themselves a revenue stream, only as part of a bigger package)…

> Nvidia had a revenue of $27billion in 2023 - that's about $160 per person per year [0] for every working age person in the USA

As a non-American, I'd like to point out we also earn money.

> as no AI is going to milk a cow or serve you starbucks

Cows have been getting the robots for a while now, here's a recent article: https://modernfarmer.com/2023/05/for-years-farmers-milked-co...

Robots serve coffee as well as the office parts of the coffee business: https://www.techopedia.com/ai-coffee-makers-robot-baristas-a...

Some of the malls around here have food courts where robots bring out the meals. I assume they're no more sophisticated than robot vacuum cleaners, but they get the job done.

Transformer models seem to be generally pretty good at high-level robot control, though IIRC a different architecture is needed down at the level of actuators and stepper motors.

Sure, robotics help many jobs, and some level of the current deep learning boom seems to have crossover in improving that - but how many of them are running LLMs that affect Nvidia's bottom line right now? There's some interesting research in that area, but it's certainly not the primary driving force. And then is the control system the limiting factor for many systems - it's probably relatively easy to get a machine today that makes a Starbucks coffee "as good as" a decently trained human. But the market doesn't seem to want that.

And I know restricting it to the US is a simplification, but so is restricting it to Nvidia, it's just to give a ballpark back-of-the-envelope "does this even make sense?" level calculation. And that's what I'm failing to see.

Machines that will make espresso, automatically, that I personally like better than what Starbucks serves are widely available. No AI needed, and they aren't even "robotic". They can use ordinary coffee beans, and you can get them for home use or for commercial use. You can also go to a mall and get a robot to make you coffee.

Nonetheless, Starbucks does not use these machines, and I don't see any reason that AI, on its current trajectory, will change that calculation any time soon.

I love how the fact that we might not want AI/robots everywhere in our lives isn't even discussed.

They could serve us a plate of shit and we'd debate if pepper or salt is better to complement it

It's pretty often discussed, it's just hard to put everything into a single comment (or thread).

I mean, Yudkowsky has basically spent the last decade screaming into the void about how AI will with high probability literally kill everyone, and even people like me who think that danger is much less likely still look at the industrial revolution and how slow we were to react to the harms of climate change and think "speed-running another one of these may be unwise, we should probably be careful".

What we had in the 80s was barely able to perform spell-check, free downloadable LLMs today are mind-blowing even in comparison to GPT-2.

I think the only good thing that came out of the 80s was the 90s. I’d leave that decade alone so we can forget about it.

> even operated by people who have no clue how the underlying mechanism functions are still far more empowered than anything like the primitives of the 80s enabled.

I'm still not convinced about that. All the """studies""" show 30-60% boost in productivity but clearly this doesn't translate to anything meaningful in real life because no industry laid off 30-60% of their workforce and no industry progressed anywhere close to 30% since chat gpt was released.

It's been released a whole 24 months ago, remember the talks about freeing us from work and curing cancer... Even investments funds which are the biggest suckers for anything profitable are more and more doubtful

> All the """studies""" show 30-60% boost in productivity

The services that provide serious productivity boosts aren't being heavily used or marketed yet. They:

1. Attempt to do narrow tasks with high proficiency. 2. Replace specific job titles. 3. Are high-value enough to be slower to engage in layoffs.

What I'm suggesting is not a fungible solution, but it is one that will be highly profitable and productive.

Haven't people been saying that for the last decade? I mean, eventually they will be right, maybe "about" means next year, or maybe a decade later? They just have to stop making huge improvements for a few years and the investment will dry up.

I really wasn't interested in computer hardware anymore (they are fast enough!) until I discovered the world of running LLMs and other AI locally. Now I actually care about computer hardware again. It is weird, I wouldn't have even opened this HN thread a year ago.

What makes local AI interesting to you vs larger remote models like ChatGPT and Claude?

Not OP but for me a big thing is privacy, I can feed it personal documents and expect those to not leak.

It has zero cost, hardware is already there. I'm not captive to some remote company.

I can fiddle and integrate with other home sensors / automation as I want.

Curious as I’m of the same mind - what’s your local AI setup? I’m looking to implement a local system that would ideally accommodate voice chat. I know the answer depends on my use case - mostly searching and analysis of personal documents - but would love to hear how you’ve implemented.

llama.ccp and time seems to be the general answer to this question.

If you are just starting up, you can try out 'open-webui' as inspiration. After that you can just use llama.cpp to build out your own things.

Hardware side, I just have a beefy server that acts as a router (mellanox card to provider fiber optic and local fiber network), firewall, wifi access point, zigbee coordinator, host to various services, camera video feed ingestion and processing, and so on...

define brefy server

Lack of ideological capture of the public models.

Control and freedom. You can use unharmonious models and hacks to existing models, also latency, you can actually use AI for a lot more applications when it is running locally.

Selling cheap products that are worse than the competition is a valid strategy during downturns as businesses look to cut costs

The survivors of the AI winter are not the dinosaurs but the small mammals that can profit by dramatically reducing the cost of AI inference in a minimum Capex environment.

The ML dev world isn’t a consumer mass market like PC gaming is.

Launching a new SKU for $500-1000 with 48gb of RAM seems like a profitable idea. The GPU isn't top-of-the-line, but the RAM would be unmatched for running a lot of models locally.

You can’t just throw in more RAM without having the rest of the GPU architected for it. So there’s an R&D cost involved for such a design, and there may even be trade-offs on performance for the mass-market lower-tier models. I’m doubtful that the LLM enthusiast/tinkerer market is large enough for that to be obviously profitable.

That would depend on how they designed the memory controllers. GDDR6 only supporting 1-2gb modules at present (I believe GDDR6W supports 4gb modules). If they were using 12 1gb modules, then increasing to 24gb shouldn't be a very large change.

Honestly, Apple seems to be on the right track here. DDR5 is slower than GDDR6, but you can scale the amount of RAM far higher simply by swapping out the density.

It's a 192 bit interface, so 6 16gbit chips.

Of course you can just add more RAM. Double the capacity of every chip and you get twice the RAM without ever asking an engineer.

People did it with the RTX3070. https://www.tomshardware.com/news/3070-16gb-mod

Can you find me a 32Gbit GDDR6 chip?

give me 48gb with reasonable power consumption so I can dev locally and I will buy it in a heartbeat. Anyone that is fine-tuning would want a setup like that to test things before pushing to real GPUs. And in reality if you can fine-tune on a card like that in two days instead of a few hours it would totally be worth it.

I would love too, but you can't just add the chips, you need the the bus too.

The bigger point here is to ask why they aren't designing that in from the start. Same with AMD. RAM has been stalled and is critical. Start focusing on allowing a lot more of it, even at the cost of performance, and you have a real product. I have a 12GB 3060 as my dev box and the big limiter for it is RAM, not cuda cores. If it had 48GB but the same number of cores then I would be very happy with it, especially if it was power efficient.

Because designing a low end GPU with a very wide memory interface isn't useful for gaming, and that is where the vast majority of non-datacenter discrete GPU sales are right now.

> give me 48gb with reasonable power consumption so I can dev locally and I will buy it in a heartbeat

https://a.co/d/1LMNatf

It's not technically possible to just slap on more RAM. GDDR6 is point-to-point with option for clamshell, and the largest chips in mass production are 16Gbit/32 bit. So, for a 192bit card, the best you can get is 192/32×16Gbit×2 = 24GB.

To have more memory, you have to design a new die with a wider interface. The design+test+masks on leading edge silicon is tens of millions of NRE, and has to be paid well over a year before product launch. No-one is going to do that for a low-priced product with an unknown market.

The savior of home inference is probably going to be AMD's Strix Halo. It's a laptop APU built to be a fairly low end gaming chip, but it has a 256-bit LPDDR5X interface. There are larger LPDDR5X packages available (thanks to the smartphone market), and Strix Halo should be eventually available with 128GB of unified ram, performance probably somewhere around a 4060.

Interesting, will have to try it out when it is released.

Okay, Nvidia.

You're right: nobody's doing ML these days. /s

Look at the value of a single website with well over a million users where they publish and run open-weights models on the regular: back in August of 2023, Huggingface's estimated value was $4.5 billion.

These are $200 low end cards, the B5X0 cards. Presumably they have B7X0 and perhaps even B9X0 cards in the pipeline as well.

There has been no hint or evidence (beyond hope) Intel will add a 900 class this generation.

B770 was rumoured to match the 16 GB of the A770 (and to be the top end offering for Battlemage) but it is said to not have even been taped out yet with rumour it may end up having been cancelled completely.

I.e. don't hold your breath for anything consumer from Intel this generation better for AI than tha A770 you could have bought 2 years ago. Even if something slightly better is coming at all there is no hint it will be soon.

> These are $200 low end cards

Hm, i wouldn't consider 200$ low end.

It is when the high end is 1000$ or more.

There isn't a cheaper card that's worth buying over using the iGPU you already have so yeah, that's the low end.

NVIDIA and AMD don't even make new GPU silicon this low-end. Their smallest current-gen GPUs all debuted at higher price points, though the Radeon RX 7600 is now available at the same price that the B580 is launching at.

I was watching a few of the preliminary commentaries on the Battlemage cards, e.g. Linus Tech Tips and so on, and they said the same thing "focus on the low/mid-end". My conclusion is that what they are talking about is "low end, brand new, current gen. graphics card". It's a very special type of low end.

The Intel cards are getting more interesting for me as I'm questioning my continued use of macOS. Intels focus on Linux support makes their options really interesting, though I don't see a need for something as powerful as these new cards.

Although there are some less powerful cards (the Intel A310/A380s are available for ~$100 and would be fine as basic display adapters w/ good video engines) if you go lower end, modern iGPUs are generally good enough for anything below the $200 level, hence why enthusiast sites (focused on DIY) consider that "low end" these days.

The Radeon 780M on Ryzen APUs can power 1080P gaming, and output to 3-4 4K/8K displays (the latest Intel Xe iGPUs are about on par, but generally pricier). A Ryzen 5700G goes for about $150 (or a 5600G for closer to $100), or you can get entire Ryzen 7840HS minipcs w/ 32GB RAM for $400-500).

> 12GB max is a non-starter for ML work now.

Can you even do ML work with a GPU not compatible with CUDA? (genuine question)

A quick search showed me the equivalence to CUDA in the Intel world is oneAPI, but in practice, are the major Python libraries used for ML compatible with oneAPI? (Was also gonna ask if oneAPI can run inside Docker but apparently it does [1])

[1] https://hub.docker.com/r/intel/oneapi

There is ROCm and Vulkan compute.

Vulkan is especially appealing because you don't need any special GPGPU drivers and it runs on any card which supports Vulkan.

https://github.com/intel/intel-extension-for-pytorch

these are the entry level cards, i imagine the coming higher end variants will have the option of much more ram.

I was wondering the same thing. Seems crazy to keep pumping out 12gb cards in 2025.

I still don't understand why graphics cards haven't evolved to include sodimm slots so that the vram can be upgraded by the end user. At this point memory requirements vary so much from gamer to scientist so it would make more sense to offer compute packages with user-supplied memory.

tl;dr GPU's need to transition from being add-in cards to being a sibling motherboard. A sisterboard? Not a daughter board.

GDDR does not exist in sodimm form factor.

Intel and AMD internal GPUs can use normal computer RAM. But they are slower for that reason and many others.

One of the reasons GPUs can have multiples of CPU bandwidth is they avoid the difficulties of pluggable dimms - direct soldered can have much higher frequencies at lower power.

It's one of the reasons why ARM Macbooks get great performance/watt, memory being even "closer" than mainboard soldered RAM so getting more of those benefits, though naturally less flexibility.

Even DDR5 has this problem. Go look at what soldered DDR5 can do frequency wise compared to DIMMs. It's one of the problems the new CAMM form factor aims to help solve, making it tractable to push the memory frequency up beyond what DIMMs can get yout currently.

I have always wondered: would it be possible to put memory on the back side of the motherboard to get I closer to the CPU? And if it is, would it solve anything else than ram clearance for CPU coolers?

that's kinda what camm2 is all about. The new interface methods provide a much better link and improve a bunch of stuff but take up more board space.

This makes sense. Do we need to begin exploring different socket configurations? IE, using something that more closely resembles a CPU socket versus a traditional RAM slot.

> Am I missing something here?

This is a graphics card.

Sir, this is a Wendy's

Who cares?

Yes exactly!!

This is not an ML card... this is a gaming card... Why are you people like this?

Reply on news.ycombinator.com