HNHacker News
TopNewBestAskShowJobs

ithkai92

3 karma · joined April 30, 2024

submissionscomments
ithkai92··on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
Thanks for the headstart, I saw hf also has PQ2_0 and able to finetune the command and in a MBA M4 24GB averages around 10t/s with the command.

./llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PQ2_0.gguf \ --port 8331 -ngl 99 -fa on -c 65536 --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

ithkai92··on Discovery of a new OpenAI agent message board
Do we know which website? Were the Agents GDPR compliant ;-)?