And yet this linked page is filled with AI slop tells, annoyingly, so it makes it hard to separate the wheat from the chaff in terms of useful information.
13,369 karma · joined May 31, 2012
Email me at josh (at) jgirvin (dot) com
https://jgirvin.com for my blog
@girvo on Twitter/Threads
This one!
I'd recommend pointing your agent at it (after installing sparkrun), and asking it to research the absolute latest in TP=1 Flash-Next - mine grabbed particular vLLM nightlies and mods to improve performance, and it was well worth it.
It’s an NVFP4 quant, but it fits, and is surprisingly capable.
But at least I can run Chinese models locally, and strip a lot of that censorship/refusal.
It was pretty easy to setup… but I cheated, and use Dokploy which handled most of it AFAICT
(At least til I get around to setting up my new usb c yubikeys!)
Still works great though!
It’s good enough that I’m considering a second spark, or selling this and buying an M5 Ultra with 256GB for it
That’s the point though.
This has happened multiple times, and it’s their algorithm that they are choosing to run.
But it is a good example of people talking past one another and not communicating well, I would think.
Nvidia is also one of the most closed hardware developers around. Two things can be true.
StepFun for example, will happily answer it when running Step 3.7 Flash locally
I do set up my initial runs and likes like quantisation-aware-distillation on my Spark-like to test it out and get it working, so it has value! But its not "worth" it other than its fun hardware to tinker with, IMO.
Exactly; its a development box for fiddling with GPU hardware with a large amount of video-addressable memory. It's not an inference box, really, though it's neat that I can at all!