I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet.
A bit like you can self host your apps at home on a Umbrel box.
I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?
A bit like you can self host your apps at home on a Umbrel box.
I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?
Smartphones aside, little Ryzen 6000 boxes would be OK.
Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.
You could maybe run something on the 4GB Jetson Nano?
But very slow: https://www.reddit.com/r/LocalLLaMA/comments/12c7w15/the_poi...
A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC, just barely) do llama 70B for far less money and a similar TDP.
I struggle to see any sizable market for the tinybox, but I wish them good luck.
The Vision is a whole different universe… with their cash and position they could (and may) take 10 cracks at it.
It doesn't take much testing to come up with the ideal power limit for your given workload(s).
[0] - https://www.pugetsystems.com/labs/hpc/nvidia-gpu-power-limit...
They have their "special place" for certain applications but the software is a mess (old driver and CUDA versions, Jetpack is still based on Ubuntu 20.04), the ARM cores are (very) weak relative to ARM flagships, and the performance/price ratio makes no sense unless you really need the form factor and energy efficiency. Oh yeah and the SD card storage is typically frustratingly slow and often unreliable.
A $500 Jetson Nano devkit with 8GB of shared RAM has roughly 10% of the performance of even ancient cards like the GTX 1070 (8GB VRAM alone) that you can throw in a random used x86_64 tower or whatever for $300 all-in. For the extra couple of hundred dollars difference you can get a more recent GPU with higher compute capability, extra storage, more system RAM, whatever. Significantly higher power usage and larger form factor but with power optimization, scaling, etc this approach makes very little difference in practice for occasional inference workloads for "typical home use". I have this configuration idling (with models loaded) at 20 watts, scaling up to around 100-150w for however long given inference loads take to execute, and then scaling back down.