sparacha

Hi HN — we’re the team behind Arch-Router [1], A 1.5B preference-aligned LLM router that guides model selection by matching queries to user-defined domains (e.g., travel) or action types (e.g., image editing). Offering a practical mechanism to encode preferences and subjective evaluation criteria in routing decisions.

Today we’re extending that approach to Claude Code via Arch Gateway[2], bringing multi-LLM access into a single CLI agent with two main benefits:

1. Model Access: Use Claude Code alongside Grok, Mistral, Gemini, DeepSeek, GPT or local models via Ollama.

2. Preference-aware Routing: Assign different models to specific coding tasks, such as – Code generation – Code reviews and comprehension – Architecture and system design – Debugging

Why not route based on public benchmarks? Most routers lean on performance metrics — public benchmarks like MMLU or MT-Bench, or raw latency/cost curves. The problem: they miss domain-specific quality, subjective evaluation criteria, and the nuance of what a “good” response actually means for a particular user. They can be opaque, hard to debug, and disconnected from real developer needs.

[1] Arch-Router: https://huggingface.co/katanemo/Arch-Router-1.5B

[2] Arch Gateway: https://github.com/katanemo/archgw

Hey! I built this. AMA. The model router is built into the proxy layer here: https://github.com/katanemo/archgw

But you can also use tokens to implement routing decisions in a proxy. You can make RBAC natively available to all agents outside code. The incremental feature work in code vs an out of process server is the trade off. One gets you going super fast the other offers a design choice that (I think) scales a lot better

There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve

That’s an example of what the edge component could do. Did you give the preference-based automatic routing a try?

RouteLLM is essentially a benchmark-driven approach. Their framework chooses between a weak and a strong model and helps developers optimize for a metric called APGR (Average Performance Gap Recovered) — a measure of how much of the stronger model’s performance can be recovered when routing some queries to the weaker, cheaper model. However, their routing models are trained to maximize performance on public benchmarks like MMLU, BBH, or MT-Bench. These benchmarks may not capture subjective, domain-specific quality signals that surface in practice.

Arch-Router takes a different approach. Instead of focusing benchmark scores, we lets developers define routing policies in plain language based on their preferences — like “contract analysis → GPT-4o” or “lightweight brainstorming → Gemini Flash.” Our 1.5B model learns to map prompts (along with conversational context) to these policies, enabling routing decisions that align with real-world expectations, not abstract leaderboards. Also our approach doesn't require router model retraining when new LLMs are swapped in or when preferences change.

Hope this helps.

Arch is developer friendly, but designed for enterprise-grade customers in mind. The core contributors of Envoy redesigned the proxy substrate to handle prompts - offering something that is battle tested in terms of resiliency, speed, and deployments. Second, OpenRouter offers choice of models, but dynamically routing to LLMs based on user-defined usage policies is uniquely available in Arch. Hope that helps

Overall performance degrades from 93.17 -> 92.99 with a quantized version

Can you share more about your evaluation setup? I would love to see the specific usage pattern as we have tested our model against smaller LLMs and foundational models and our results show things differently. Of course, routing policies should follow best practices here: https://docs.archgw.com/guides/llm_router.html

Nonetheless, super curious to learn more and see what we may be able to improve. This is technically not a classifier model - its a usage prediction model (feels like a classifier, but not quite in terms of intended usage)

yes - we have already published a quantized version here: https://huggingface.co/katanemo/Arch-Router-1.5B.gguf. The performance difference with a quant version is negligible. I'll run another analysis and update the thread shortly

Hi HN! I am one of the co-authors of the paper. If there are any questions about our approach, I would love to answer them.

good one!

build MCP servers that get plugged into different apps like Claude desktop. But what if you want to go from FastAPI functions and build your own agentic app - added bonus have common tool calls be blazing fast.

Just updated https://github.com/katanemo/archgw (the AI-native proxy server for agents) that can directly plug into your MCP tools and FastAPI functions so that you can ship an exceptionally high-quality agentic app. The proxy is designed to handle multi-turn, progressively ask users clarifying questions as required by input parameters of your functions, and accurately extract information from prompts to trigger downstream function calls - added bonus get built-in W3C tracing for all inbound and outbound request, gaudrails, etc.

Early days for the project. But would love contributors and if you like what you see please don't forget to the project too.

Opik is an evaluation tool first. Arch is a proxy server built on top of Envoy so it borrows from a very robust observability source. They both are complimentary in many ways

Speed. And separately, instruction fine-tuning an LLM for a specialized task like function calling or guardrails == better performance. Even Anthropic and other model providers suggest you separate tasks for LLMs to improve overall user experience

We happen to take those tasks that are non-business or domain specific related and trained our models to offer SOTA performance for 1/10th the cost and 10x the speed. For e.g. Arch-Function can process 5k/tokens per sec

Woud love feedback. See if it is useful, or what adaptations would make it useful.

Leaderboards are getting harder and harder as a decision tool. What does it mean to be better 0.7% or 1.6%. How does that help me? Is higher always better? What are the trade offs? Evals continue be the hardest most important parts of LLMs and tools that use them

Why We Built ArchGW?

Traditional application architectures separate routing, security, and observability from business logic - so that developers can move faster without the tax of reinventing the wheel. LLM applications should be no different. ArchGW applies these patterns to prompts, providing a structured approach to building LLM applications.

How It Works

ArchGW runs as a separate process alongside application servers. It intercepts prompts before they reach the backend and applies transformations based on predefined rules and models:

    Preprocessing: Normalizes and analyzes prompt structure.
    Security Filtering: Rejects jailbreak attempts and unsafe inputs.
    Intent Mapping: Determines if a request maps to an API function.
    Function Invocation: Extracts arguments and calls backend APIs.
    LLM Routing: Chooses the right LLM provider based on latency/cost constraints.
    Tracing & Metrics: Adds W3C Trace Context headers, tracks errors, token usage, and request latency.

Why a Dedicated Proxy?

Traditional application architectures separate routing, security, and observability from business logic—LLM applications should be no different. ArchGW applies these patterns to prompt processing, providing a structured approach to LLM integration.

OP - congrats on this launch! I think there are a lot of useful bits here, especially the communication abstractions for MCP and having clear recipes to implement patterns mentioned by Anthropic team is a big plus.

If you are building a state machine then you should use tools that enable you to do that (temporal, langgraph, etc) by orchestrating multple LLM calls - in that case archgw offers intelligent routing to your COT agent and enables you to transparently add rich observability and metrics for all calls made to LLMs in the COT/state machine scenario.

Arch-Function our fast, open source LLM does most of the heavy lifting on extracting parameter values from a user prompt, gathering more information from the user, determining the right set of functions to call downstream. Its designed for smarter RAG scenarios and agentic workflows (like buying an insurance claim through prompts). While you can use the model yourself, archgw offers a framework on usage to detect hallucinations and re-prompt the LLM if token logprobs.

The same model is currently being updated to handle complex multi-turn intent and parameter extraction scenarios, so that the dreaded follow-up, clarifying RAG use case can be effortlessly handled by developers without having to resort to complex LLM pre-processing. In essence, if the user's follow-up question is "remove X", their RAG endpoints get structured information about the prompt and refined parameters against which developers simply have to retrieve the right chunks for summarization.

This "felt" like a cool project - but the title of the post was "salesy". Plus the project lacks technical detail. As a courtesy to you because you are a first time poster, try a more technically oriented post. Why is it technically different - the audience will then figure out if they want to use it or not. HN is for generally a technically astute audience.

Great to hear that - walk us a bit more how you are planning on using the filter? The filter today does communicate with a model_server service for the intelligence piece. Were you thinking of using the filter for outbound LLM traffic management?

Yea - that’s gonna be an issue. It was a hotly debated name but we figured the SEO will adapt with GenAI and the way we know SEO today might not be problematic. Let’s see - tricky for sure

It's early days, so while there might be some overlap, I am sure there is a lot that we can do together to build complimentary products.

Based on the press release, its kinda hard to tell exactly how different/alike we will be, but Arch will always be "designed-first" for prompts and LLM application workloads without exposing all Envoy related features. And Envoy is "designed-first" for micro-services application workloads. So there will be some overlap but our design principles will deviate over time I feel. But we are very open to collaborating with the community here...

We agree Envoy is legendary - and per se doesn't lend anything to the AI space. That's essentially what we are doing here, building on top of Envoy to add capabilities specifically for AI and prompts. For instance, we use Envoy's filtering capabilities to handle and process prompts - this was get to keep all the robustness and security features for TCP/HTTP from Envoy and solve the critical but undifferentiated tasks related to prompts like safety, observability, routing, function calling, etc.

Two things

1/ Arch builds on Envoy so that we don't re:invent all the HTTP(s)/TCP level capabilities needed in a modern gateway for applications. So in that sense, we agree with you that it should "extend" something vs. rewriting the whole stack. There are several security and robustness guarantees that we borrow from Envoy as a result of this. To be more specific, a lot of Arch's core implementation today is an Envoy filter written in Rust.

1/ Arch's core design point is around the handling and processing of prompts, which we believe are nuanced and opaque user request that require secure handling, intelligent routing, robust observability, and integration with backend (API) systems for personalization – all outside business logic. This requires the use of models and LLMs that are fast, cheap and capable to help developers stay focused on application features. For example, Arch uses (fast) purpose-built LLMs for jailbreak detection, converts prompts into API semantics for personalization, and (eventually) automatically routing to the best outbound LLM based on the complexity of a prompt to improve the cost/speed of an app.

We believe #2 will continue to be different and evolve further away from traditional API/HTTP routing that it will require constant invention and work to make the lives of developers easy.

Hope this helps!

That's a fair point - technically it protects the application from malicious attempts to subvert the desired LLM experience. The more specific language (and I think we could do better here) would be that Arch ensures users remain within the bounds of an intended LLM experience. That at least was the intention behind "ensure safe user interactions"...

Hey HN - my name is Salman and I am Adil’s Co-Founder. Would love to hear and get feedback. Here is a link to our public roadmap, please lets us know if there are things you’d like for us to work on first

https://github.com/orgs/katanemo/projects/1