Would having a locally-hosted model offset any of these costs?
This is a problem for coding as smarter really has an impact there, but there are so so so many tasks that an 8b model that runs on a $200 gpu can handle nicely. Scrape this page and dump json? Yeah that’s gonna be fine.
This is my conclusion based on a week or so of using ollama + qwen3.5:3b self hosted on a ~10 year old dell optiplex with only the built-in gpu. You don’t need state of the art to do simple tasks.
Only gonna be fine on a trusted page, an 8b model can be prompt injected incredibly trivially compared to larger ones.
Obviously you should also take precautions, like never instructing it to invoke the browser tool on untrusted sites, avoiding feeding it untrusted inputs where possible in other places, giving it dedicated and locked-down credentials where possible....
But yeah, at this point it's inherent to LLMs that we cannot do something like SQL prepared statements where "tainted" strings are isolated. There is no perfect solution, but using the best model we can is at least a good precaution to stack on top of all our other half-measures.
Claude 4.6 is at least a bit resilient to prompt injection, but local models are much worse at that, so using a local model massively increases your chance of getting pwned via a prompt injection, in my estimation.
You're kinda forced to use one of the better proprietary models imo, unless you've constrained your claw usage down to a small trusted subset of inputs.