I guess “A” by itself would be a seperate token but my point is the model might be trying to say something that begins with that letter rather than actually answering.
You’d need to use an approach that links the output to a closed set.
You’d need to use an approach that links the output to a closed set.
Yesterday, just to prove to a friend that Jev isn't that "revolutionary" I extracted some image classification code that Claude had written for my private image organizer that used Qwen3-VL, and stopped the output at a single token, then used the probabilities. Input processing on my GPU was somewhere around 1000ms per image, so not too fast, but each question used the prompt cache, so followups were 100ms-ish.
That was my baseline of an untrained, non-optimized single pass classification. It would connect to my llama-server, and use the logprobs for the choices.
After that, I had Claude remove llama-server from the solution, and write it directly to the transformers, then I kept prompting it to profile and find more speed. Eventually my "Decision Engine" running locally on a trained 1B model got to just under 85% accuracy across my 500 validation prompts (images and text) not used or derived from the training set, and an 8MB image, with 10 questions with 5 choices per question, got down to just under 500ms. Pure text prompts and questions are below 100ms for 300tokens + 10 questions + 5 choices per question (average).
It did better on text than images, just because my training set included 90% text. I'll do more training and validation for images when I get home, but for now, I'm more than happy that I can get a local "decision engine" running in 4GB of VRAM and responding in under 30ms for most use cases I've had.