silentbob7

Here's my model.ini for 4090 (using the chat template from froggeric):

version = 1

[*] ; Global defaults shared across all presets

threads = 16

[Qwen3.8-27B]

; Model files

model = ..\Qwen3.8-27B-FP8\Qwen3.8-27B-UD-Q4_K_XL.gguf

mmproj = ..\Qwen3.8-27B-FP8\mmproj-BF16.gguf

; Context & compute

ctx-size = 114688

parallel = 1

n-gpu-layers = -1

batch-size = 1024

ubatch-size = 512

flash-attn = on

cache-type-k = q8_0

cache-type-v = q8_0

; Speculative decoding (MTP draft)

spec-default = true

spec-type = draft-mtp

spec-draft-n-max = 5

cache-type-k-draft = q4_0

cache-type-v-draft = q4_0

; Multimodal

no-mmproj-offload = true

; Context behavior

no-context-shift = true

; Chat template & reasoning

jinja = true

reasoning = on

chat-template-kwargs = {"preserve_thinking":"true","reasoning_effort":"medium"}

; Sampling

temp = 1.0

top-n-sigma = 1

top-k = 0

top-p = 1.0

presence-penalty = 0.0

repeat-penalty = 1.0

fit = off

; Auto-load this model when the server starts

load-on-startup = true

I'm especially interested in details on process-in-memory, combined with that large sizes this could be very powerfull in some usecases.

It depends on your usecase and the size of your ram is not the only driver. I think the primary performance drivers (without sacrificing precision) right now are QAT, MTP/DFlash, MoE and Hybrid approaches to avoid full attention in parts by replacing with linear / sparse attention. So Gemma4 26B A4B QAT+MTP (from unsloth) would be a good pick atm. I'd love to see some smaller models with all that features.

Also worth checking out https://github.com/OHF-Voice/linux-voice-assistant

Tested with german and this kinda works, but confidence suffers.

I also wonder how small a LLM trained on catching only subject (e.g. living room) and action (light on) from text input could be compared to needle - the json wrapping could be done afterwards using templates.

The wyoming protocoll seems to be the path for home assistant audio, so you need STT (wyoming-faster-whisper), TTS (wyoming-piper for wide language support) API endpoints and some Ollama or OpenAI API endpoint available for your home assistant server.

I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).