Hacker News Reader

Comment by gdsdfe | Hacker News Reader

I've been thinking about hosting my own LLM to see if I can hyper customized it to me basically, kinda like an AI companion. My main issue is building the hardware, there so much fluff in that space, it's hard to know what to get and what works well together

View on HN

- Intel Core i9-11900KF 3.5 GHz 8-Core Processor

- Corsair H150i PRO 47.3 CFM Liquid CPU Cooler

- MSI MPG Z590 GAMING EDGE WIFI ATX LGA1200 Motherboard

- G.Skill Ripjaws V 64 GB (4 x 16 GB) DDR4-3600 CL18 Memory

- Samsung 970 Evo Plus 1 TB M.2-2280 PCIe 3.0 X4 NVME Solid State Drive

- MSI GeForce RTX 3090 TI SUPRIM X 24G GeForce RTX 3090 Ti 24 GB Video Card

- Corsair Carbide Series 275R ATX Mid Tower Case

- Corsair RM1000x (2021) 1000 W 80+ Gold Certified Fully Modular ATX Power Supply

- Microsoft Windows 11 Pro

On this setup I've been able to run every model 13B and below with 0 issue. Even been able to fine-tune Llama 2 13B using my own data (emails, SMS, FB messages, WhatsApp, etc.) with pretty fun results!

What was your stack for doing the fine tuning with your own data? I've been looking around at various different approaches and I think I have a general idea of how to approach it (generating embeddings, putting them in a vector DB, somehow linking that to a useable UI, etc.). Would definitely be keen to understand what your solution looks like though!

I am interested in this as well. Pretty please!!!

> On this setup I've been able to run every model 13B and below with 0 issue. Even been able to fine-tune Llama 2 13B using my own data (emails, SMS, FB messages, WhatsApp, etc.) with pretty fun results!

Is it useful? Would you let it draft responses at you? Curious about the fun results :)

Reply on news.ycombinator.com