codelion

Is it available for public yet or just selected companies?

How do you do cross entropy analysis for Claude without logits?

Another reason to avoid it at all costs if you needed it - https://x.com/thdxr/status/2079623963995467869?s=20

Finally good seeing someone else also getting frustrated with OpenCode and posting the issues.

Just use OpenEvolve for such problems.

I use a 2010 Macbook Air with Linux XFCE desktop and it works well for browsing and simple office work.

Using qwen3.6 27b locally with Claude code, it works well for simple coding tasks

You can consider using Frame for the SAST part - https://github.com/lambdasec/frame

How does it compare to some of the newer mlx inference engines like optiq that support turboquantization - https://mlx-optiq.pages.dev/

the key issue is the interpretation of "consent" when continued use is the only option. aree users truly consenting, or are they simply left with no alternative?

When mammals hunt other mammals strange things can happen.

Whom to believe? Devin or Claude? - https://www.anthropic.com/engineering/multi-agent-research-s...

It is by design. OpenAI is not going to reveal any architectural innovation they have made in their own commercial models.

You can run in two modes, by default you run in the inference mode without learning. So, the changes you made will be used. If you switch to learning mode then the strategies are updated/refined and merged based on a config that you can control.

# How often to perform maintenance operations (merge, prune)

MAINTENANCE_INTERVAL = 40

# Strategy selection thresholds

STRATEGY_CREATION_THRESHOLD = 0.7 # Higher threshold to avoid creating similar strategies

STRATEGY_MERGING_THRESHOLD = 0.6 # Lower threshold to merge more similar strategies

MIN_SUCCESS_RATE_FOR_INFERENCE = 0.4 # Minimum success rate for a strategy to be used during inference

The configs are all defined here - https://github.com/codelion/optillm/blob/main/optillm/plugin...

We do not allow the strategies to keep growing there is a refinement phase where we refine and merge existing strategies. The experiments were run with this config - https://github.com/codelion/optillm/blob/main/optillm/plugin... which allows a maximum of 10 strategies of each type.

Re-reading the problem apparently works well - https://arxiv.org/abs/2309.06275

Here the system seems to have discovered this strategy by itself. The prompts are generic because during learning there is a part to refine and combine them. I haven’t experimented yet by adding all prompts to every query, given the large context it will be interesting to see.

We have some examples in the plugin README: https://github.com/codelion/optillm/tree/main/optillm/plugin...

E.g. This was the strategy discovered by optiLLM for solving word problems:

*Refined Strategy for Solving Word Problems:*

1. *Understand:*\n * Read the problem carefully (multiple times).\n * Identify the question (what are you trying to find?).\n * List all given information (facts, numbers, units).\n * Clarify ambiguous terms/units.

2. *Organize Information & Identify Unknowns:*\n * Choose an organization method: (e.g., table, diagram, list, drawing).\n * Clearly identify the unknowns (what you need to solve for).

3. *Plan and Translate:*\n * Define all variables with units (e.g., `p = number of pennies`, `c = number of compartments`).\n * Identify relationships between knowns and unknowns.\n * Convert units if necessary.\n * Write equations or expressions, including units, that relate the knowns and unknowns.\n * Ensure units are consistent throughout the equations.\n * Outline the solution steps.

4. *Solve:*\n * Show work step-by-step.\n * Track units throughout calculations.\n * Calculate accurately.\n * Solve for the unknowns.\

5. *Evaluate and Verify:*\n * Check if the answer is reasonable.\n * Verify the answer.

6. *Summarize:*\n * State the answer with units

Full list of strategies discovered is available here -https://github.com/codelion/optillm/blob/main/optillm/plugin...

Optillm works with llama.cpp but this approach is implemented as a decoding strategy in PyTorch so at the moment you will need to use the local inference server in optillm to use it.

Thanks for checking this out! A few additional details that didn't fit in the main post:

The system maintains two separate limits: a storage limit (max 10 strategies per problem type in the database) and an inference limit (max 3 strategies applied per query). This keeps the database manageable while ensuring the system prompt doesn't get too long.

One interesting finding was that strategies only get used for inference once they have at least 5 attempts and a 40% success rate. This prevents the system from applying unproven strategies to new problems.

The approach works particularly well with reasoning models like DeepSeek-R1 and QwQ - the learned strategies seem to guide their thinking process effectively.

I'm especially curious about:

1. How this might work with different model families

2. Whether the community sees value in sharing strategy databases between users

3. Ideas for extending beyond text-based reasoning to multimodal problems

The plugin integrates with our broader optillm project which has other inference optimization techniques. You can combine SPL with methods like mixture-of-agents or MCTS using the "&" operator.

Next I'm thinking about meta-learning - having the system learn how to create better strategies more efficiently. Also exploring collaborative strategy sharing.

Would love to hear thoughts on the approach or if anyone has ideas for other problem domains where this might be useful!

If we try to benchmark GPQA-Diamond with DeepSeek-R1 in the suggested configuration of 0.6 temp and 32k max_tokens and say if every instance takes the maximum tokens it will require 6.4 M tokens. Which without batching on a single H100 at 80 tok/s will take 23 hrs to run. To run with 32k context length on a single H100 a 1.5B model will require ~15-20 GB VRAM so you cannot run 10s or 100s of queries in parallel.

MMLU-PRO is 12,000 instances. To avoid this we set a 600 seconds timeout for each instance to run.

This is an interesting idea, I hadn't thought of it. It is worth experimenting I am not aware of anyone else trying it yet.

Hey, yes the reported results do not restrict any time limit or token limit for the benchmarks. We run our baseline with the same config 0.6 temp and max_token 32k but we set a timeout after 600 secs. Otherwise it would take forever to benchmark with the resources we had. I have a note in the actual paper on that in the implementation details section.

We use an adaptive classifier to learn how many tokens the model takes to respond correctly on a known dataset. I used the https://huggingface.co/adaptive-classifier/llm-router for experiments it is based on distilbert.

The short answer is in general yes it helps improve the accuracy, there is a whole line of work on self consistency and critique that supports it. Many of those approaches are already implemented in optillm.

Yes, the goal here is to avoid overthinking and be as efficient as possible in terms of the minimal tokens required to solve a query. Often, queries that require too many tokens are unlikely to lead to correct answers anyways otherwise they would show up when we are learning the classifier.

Query complexity in this context is based on how many tokens it took for the model to respond to a query correctly based on a ground truth dataset like GSM8k. The adaptive classifier learns over this dataset and then we use it at inference for classification.

This sounds like an interesting idea, can you elaborate more may be with a concrete example. I am wondering if this can be implemented easily as a plugin in optillm.

Yes, we started with the idea of trying to replicate similar control on thinking processes for open reasoning models. They also announced the Deep Think approach at IO which goes even further and combines parallel CoTs at inference.

The motivation for AutoThink came from watching how current reasoning models waste computation - they spend the same amount of "thinking time" on "what's 2+2?" as they do on complex mathematical proofs. This seemed obviously inefficient.

The breakthrough was combining two techniques I'd been working on separately: adaptive classification (which can learn new categories without retraining) and an open source implementation of Pivotal Token Search from Microsoft's Phi-4 paper. When I put them together with dynamic token budgeting, the performance gains were much better than expected.

What surprised me most was that the technique actually uses fewer tokens on average while improving performance. The adaptive allocation means simple queries finish faster, offsetting the extra computation on complex ones.

A few technical notes:

- The steering vectors are small (typically <1MB per pattern) and add minimal memory overhead

- Classification adds about 10ms latency, which is negligible

- Target layer selection matters - I found middle layers (15-20) work best for most models

I'd love feedback on:

- Have you tried similar adaptive approaches with your models?

- What other reasoning patterns would be useful to steer toward?

- Ideas for automatically detecting the optimal target layer?

Thanks for checking it out! Happy to answer any questions about the implementation or results.

Checked my openrouter stats, it took ~3M tokens but that involved quite a few runs of various experiments.