Nvidia might do for desktop AI what it did for desktop gaming

Even 200B for 3k USD is really good, 128 GB of memory for that price is surprising!

Right now using a two RTX 2080 setup for a pilot project, where it runs ollama and qwen2.5-coder (14B quantized version), a more serious step up from there on a budget would realistically be an RX 7900 XTX (24 GB, though I've had issues with setting up ROCm) for 1k EUR or RTX A5000 (24 GB) for 2.5k EUR in the current market.

Honestly, I don't even need the best performance for what I'm trying to do, even an Arc A770 (16 GB) for 350 EUR would be enough to iterate, except that it's not actually supported by ollama and lots of stuff out there: https://github.com/ollama/ollama/blob/main/docs/gpu.md (I know ollama isn't the only solution, but it sucks when the tools you like aren't available)

nvidia could just release variations of their "gaming" cards with more RAM. There is absolutely nothing stopping them from releasing 64GB or 128GB 5080s and 5090s.

But they don't because that would cannibalize the extremely overpriced enterprise offerings. The #1 reason people are forced into the tens of thousands of dollar cards is memory needs.

So considering that, ask what niche this device really fills: Is it a new "supercomputer" for the home? Not really, given that it is silicon and memory bandwidth restricted so much that their $600 GPU can beat it soundly on every metric (not surprising when you look at the power and airflow/cooling needs of "real" GPUs) but in scenarios requiring large memory. But while this can hoist those larger models, it is going to be far removed from state of the art.

It's neat, but the market for this is being grossly overstated on a lot of these hype advertorials. The large models you'll run on it will be quantized the point of absurdity, not to mention that for 99.9%+ of users, anything short of state of the art is basically useless.

It's a neat eGPU of sorts for a Mac or something (they really hype the fact that you use CUDA for this). Still really trying to figure out what value it possibly brings outside of trying to lure a bunch of enthusiasts to blow money on this so they can fiddle with Llama for a week and then realize it's a waste of time.

Is there any info about how many tokens a second you would get with a 400b model? Without that it's like claiming a graphics card can output at 8k, but neglecting to state the frames per second. Suspect, in other words.

Outputs at 8k resolution! (+)

...at 5 fps (+)

What special sauce do they have to run 400B models fast exactly?

A local NAS makes sense for some use cases. Privacy, local control, and network bandwidth adds up to having a NAS for some use cases.

I see similar coming for AI. Tagging local photos, reading/summarizing private documents, helping you code (without uploaded it to 3rd parties), using uncensored LLMs, maybe even playing NPCs once games support API for LLMs.

It's going to take awhile, but I wouldn't be surprised if NASs start supporting AI accelerators to provide local endpoints for AI.

I'm not too familiar with the AI space but I wonder if this is an effort from NVIDIA to combine their AI and Gaming markets. Did this come from a conversational question stating, 'How do we sell discrete cards through our existing manufacturing partnerships to both gaming enthusiasts and AI enthusiasts?'. I do wonder how comfortable they are pivoting back to being a consumer hardware company if AI becomes a more competitive space, or if the 'hype' subsides. Pure speculation and I'm probably off the mark.

A video editor wants the tool sitting on their desk. Not pay-per-gen SaaS where the results are garbage nine out of ten times.

The comfy ecosystem is rife with people that want local tools.

I see the DIGITs box as mainly for inference, not training. It allows me to load a fairly large model (e.g. 70B llama or 12B flux) and run it locally at decent speeds.

privacy for machine learning to be locally usable besides a chat interface in a meaningful way I would have to give it access to pretty much all my digital data

sending that over the network feels very idk icky because it's not just photos or emails

"I'll never buy a SBC from Nvidia unless all the SW support is up-streamed to Linux kernel," top comment from prev discussion https://news.ycombinator.com/item?id=42623030

Not everyone has low latency high bandwidth reliable internet, and almost no one has it all the time. In addition, the privacy and security benefits from running things locally is a requirement for some.

The people I know that use local AI instead of remote AI like the privacy, response time, and not being charged per-query.

we do now but Stadia failed to have a meaningful impact for a reason

obviously companies will try to migrate everything to the SaaS model because it is better for revenue and bug fixes but not necessarily the best user experience

don't want games to stop if I dont't have wifi and i don't want ML stuff to require internet or feel comfortable sending my entire digital data to a third party or have models be updated because of some compliance policy

but it's still in the infancy companies will do everything to make it over the network one or two nice tech people will try to make it local/hybrid hopefully they succeed

The Shield TV came out in 2015, was last refreshed in 2019 (which you can still buy today), and the entire line is still getting updates. That's longer than e.g. the PS Vita.

It’s just been announced, shipping in July last I heard, will be a few months before you can pre order

An ad masquerading as “insight” is nothing new in this space. Author is eating up the NVDA marketing and joining the cult.