Ask HN: Cheapest box to run a foundational model?
I was thinking of a Mac studio. If you were to build out a server that can run llama 3 or any of the other open source models, what would you recommend?
Cheap is a 4060 16GB, which can run the lower parameter models but there are obviously limits to speed, which models, and size of context.
2 x 24GB 7900xtx/3090/4090 on a motherboard that can split the PCIe 4/5 x16 bandwidth provided by the CPU across a second x16 slot (so 2 x8) is likely the most GPU memory that can be useful on consumer hardware. The bandwidth splitting is sometimes reserved to the top end gaming branded boards so "cheap" is stretching it.
Beyond two cards you either need to fork out for A6000/RTX6000 48GB cards or need an Epyc, Threadripper or Xeon platform that's more generous on the PCIe lanes and 16x slots.