Hey everyone! Author here. Just saw a lot of traffic from HN. I'm happy to answer questions. :-)
Do you run LLMs locally? I'd be keen to have an overview of your stack.
In the article, I mostly mean working with LLMs inside the applications I'm building, as opposed to as a tool as part of development. But I do both.
Right now, I'm trying out Zed, which supports multiple LLMs natively. Just today, I tried Zed + Ollama + Qwen 2.5 Coder 32B running locally, and it worked! Blows my mind that I can have GPT-4o-level assistance running on my laptop. :D
How do you gauge "GPT-4o-level assistance"? Have you tried side-by-side comparisons?
I'm doing what you're talking about 100%. I've found that established and popular FOSS web software that has normal people users and their support forums, plural, for their normal users (as any popular software has) - this type of "established and non-sexy probably used for office work software" is fully known and understood by LLMs. It's very simple to put a support AI Agent inside the interface to that software, and because the LLM has both the user support forums and the source code itself in the training, that AI Agent can do things against the software's API and data structures by request - it knows them!
Yeah, "boring software" is no longer boring when it can be integrated with characters that know that software as good as the lead developer who wrote it and have the knowledge of a dozen PhDs too.
I'm glad the approach works for you as well! :D It's fascinating to watch a statistical document completion model be able to do so much.
But if that's what you're productive in, I'd say, absolutely!