Comment by kyxsc | Hacker News Reader

Comment by kyxsc | Hacker News Reader

Parent Full thread

Sol is way too eager to hone in on small details and ends up with massive over-engineering. Fable does it too - to be fair - but noticeably less.

After extensively using both on Max 20x plans, I've concluded that Fable is better for problem solving and coding, whereas Sol 5.6 Ultra shines in debugging specific issues: tackle a problem with Fable then leverage Sol to clean up, double check, or fix specific issues.

Fable (imo) had the edge on the $200 plan, but after this 50% reduction I'd say Codex is better value by far and there's no contest.

---

Using Fable as the orchestrator and delegating tasks to Sol 5.6 Ultra via the codex plugin in Claude Code yielded good results, but still there was a lot more over-engineering (thus time and tokens spent) than Fable by itself would've done.

Both models suffer from doing-too-much. But both models are fundamentally really smart and knowledgeable. I think it's really close and pricing cuts really spice things up for us consumers! Sol is a clear winner in the value department and the $100 plan is enticing!

---

*Claude Code usage is reducing by 33% in 2 days, Wednesday August 19... cmon anthropic: clau.de/cc-50-promo

Yeah, I very much agree on this. I think Sol and Fable code quality is on par. Maybe Fable is just a tiny bit better, but Sol compensates with its ability to work through things, while Fable, in my experience, generally tends to avoid solving problems that require many LOC.

However, I think these are very different models in terms of orchestration. Long-horizon tasks are way more predictable with Fable. It just doesn't lose track of details. Thus I ended up building a small wrapper around Pi (where I run Sol) so that CC can delegate via background tasks, automatically wait for completion, and do what was one of the most effective parts - steer Sol toward simplicity, getting Sol out of code-review infinite loops (Pi calls for Codex review to ship better, but generally gets stuck on P2 and results in vastly overengineered work).

One of the worst experiments was enforcing coverage at 100%. Only Sol, with an enormous amount of code and significant pushback (on architecture decisions) to Fable, was able to reach it. It made me think this is somehow related to overengineering in general, so that instructions on acceptance criteria in claude.md plus proper DX (e.g., Lefthook) actually led to okay results. It mostly helped that responsibilities were clearly split: Fable designs architecture, Sol handles coding and debugging.

I get a lot of mileage using Fable to spec, then Sol to review the spec, then Fable to plan, then Sol to review the plan, then Fable to implement, then Sol to review the implementation. It's a lot of steps, but a great boost in quality of output.

with what levels of effort? I only see you mention sol-5.6 on ultra, which I don't find is worth it at all. Sol-5.6 on medium is amazing.

https://winstonrc.github.io/ai-coding-agents-leaderboard/

Sol 5.6 Ultra doesn't seem worth it because its long-horizon task scaling is not that good.

Remove the one thing that differentiates these 2 SOTA models (long-horizon task scaling) and you're left with assessing the raw intelligence of both models.

Both models are really fucking smart. We should be intentional when discussing effort levels when it comes so SOTA models because currently, effort levels are the essential lever to evaluate task scaling.

I've never seen that leaderboard link but I think my sentiments reflect exactly the findings: Sol 5.6 is smart, Fable is smart, but Sol is more value for the end user (even more so when you lower effort levels because that doesn't degrade the model's raw intelligence/knowledge). Not to mention Codex resets, that's just the cherry on top!

But smart != capable and this is evident once you start assessing both models on long-horizon tasks with higher effort levels. While Fable is (imo) at least a little bit better, both are still very good. If you want to test the raw intelligence of said models, you should lower the effort level.. if you want to test the model's capabilities fully, you should increase the effort level.

Economics aside, Fable is the better model (imo), but there's no need for a binary stance here. Both models are very good yet there is a clear winner on the value front.

Interesting the use of Max and Ultra. I don’t doubt the complexity, but would someone use Max or Ultra on Typescript or Go, for example?

Is it more about just avoiding any mistakes? Seems like that would be costly when medium or high would work fine?

*The "Max" I referred to was the plan tier, not the effort level btw

For small tasks, you can just use something like low or medium effort and it can usually avoid mistakes; after all, the model will test the code anyways and can do some baseline level of iterating.

In regards to cost, we need to acknowledge how generous OpenAI was in the last couple months with Codex usage credits (no weekly limits) and usage resets. It afforded me many a dive with Codex! Yes it uses more tokens, but sometimes it's worth it -- just depends on what you're working on.

Finally, Ultra(code) isn't that bad when it comes to cached tokens. I think folks overstate the general token usage of ultra effort on both providers.

---

Both models are great at green-fielding a project when given detailed specs.

Both models overthink too liberally (imo) during these larger multi-shots. Sol overthinks more than Fable.

Both models are really smart and perform great for general knowledge and regular coding tasks.

Sol w/ Effort -> Low

It's great, don't get me wrong, but so is Fable. I'm just comparing the long-horizon task performance between the two at the same or similar effort levels.

Given the 50% discount on Sol and how smart it is, yeah it's unprecedented value. If you only want to use low effort, there's a clear winner here on value and it's not even close!

Reply on news.ycombinator.com