Sorry I should have qualified it as Nvidia Pascal architecture.
1,006 karma · joined October 25, 2022
Nvidia might have given up support but it doesn't mean vllm have to (llama.cpp didn't).
Nvidia don't share CUDA, don't open source their drivers, don't support capable but older hardware (forget Pascal etc), and generally charge a premium over competitors for hardware.
This acquisition will likely solidify this general stance, reduce free compute allowance for a subscription, cap downloads, push advertising, generally favour models that are Nvidia prescribed or have been sponsored, and potentially ban models and datasets that are deemed risky legally (abliterated etc).
I see no other reason why Nvidia would want this kind of vertical.
I'm still testing but oh wow. My new driver now works with my ultra wide 21:9 ratio at 2048x864, it also manages 2048x1152.
The driver works well, and now has full DRM and DKMS support. It also runs on modern Linux after the manufacturer decided only to go up to kernel 5.x, windows support obviously still fine.
It found many faults in the original source, like somebody didn't read the HDMI specs / didn't have any idea what they were doing.
The new driver is fully spec timings and sequence compliant, doesn't hang on shutdown anymore, and ignores EDID for the purpose of allowing more screen modes.
It also has double buffering, and shadow buffering, and a custom magic square dither mode for 16bit colour and it absolutely flies vs the 32bit mode. The dither I invented was derived from one I created years ago for some retro hardware, but it's so good it's (imho) indistinguishable from general jpeg artifacting and quite difficult to find/see. I've had to ask codex a few times to check the GPU isn't in 32bit colour.
The GPU still has an annoying bug and won't work over KVM consistently without losing sync in VESA modes, but I'm not convinced its the GPU hardware doing this, it works perfectly well directly connected.
I'm due to put a GitHub repo up for this as soon as it's battle tested, and obviously ensuring it uses EDID by default, rather than ignores it.
I'm hoping somebody can fix the KVM issue, or audits the source to confirm there's nothing that can be done, but that's the best thing about open source :)
People are afraid of change, that's natural. When the technology beds in, becomes mature, and is accessible by those threatened by it we'll have acceptance.
Until then we have "AI slop" which is the equivalent of "it'll never be as good as handmade" because the worst AI examples are being compared to the best human examples.
When we work with, and improve the standard of the technology, we'll ultimately produce a better society, and have better outputs, but that stage is still to come.
Aside from the diehard fans, it's the behaviour between the companies that stopped me buying their products, and I'm probably not the only one.
I applaud it's finally wrapped up but in a rapidly dwindling fan base they were eroding the value of their own brands - brands that should have been complementary.
I hope now they can recover that iconic status, but I feel it's simply going to go the way Atari is iconic now - mostly trendy t-shirts and retro sales.
Then I fitted my own custom 40x40x30 fan shroud for 2x6000 rpm fans at one end.
The other 2 I did something similar for but a single 2 card block and a stock fan shroud online.
I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.
I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.
The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.
Strategy patterns and code-centralised feature management ftw :)
My takeaway for other people like me from this is "computer is correct" because the proof shows that we can't define arccosh using a single proof across the entire complex plane (specifically imaginary, including infinity).
The representation of this means we have both complex functions that are defined as having coverage of infinity, and arccosh, that a proof exists in only one direction at a time during evaluation.
This distinction is a quirk in mathematics but means that the equation won't be simplified because although it looks like it can, the underlying proof is "one sided" (-ve or +ve) which means the variables are fundamentally not the same at evaluation time unless 2 approaches to the range definition are combined.
The QED is that this distinction won't be shown in the result's representation, leading to the confusion that it should have been simplified.