I'm building a pre-built AI setup to test various LLMs and have successfully integrated it into my business. I want to share it with everyone here. This already integrates Qwen2.5-Coder 32B, Llama 3.1 70B 405B, FLUX.1 [dev] and Hermes-3-Llama-3.1-8B. It's highly customizable and can be used as a cloud AI, similar to Runpod when it's not in use.
Tech specs:
- AMD Ryzen Threadripper PRO 5955 WX
- 2 x RTX 4090
- 128GB RAM
- SSD NVMe 1TB