Hacker News Reader

Comment by simonw | Hacker News Reader

It took me quite a while to come round to OpenRouter. Originally I didn't understand why anyone would put a proxy between them and an LLM, but it actually adds some quite significant value:

1. By far the lowest friction way to support and try out all the models.

2. They offer billing caps! Most model providers still don't do this [EDIT: maybe they do, see reply comment], but if you're going to run anything in public it's very useful to have hard limits so it doesn't cost you $1m overnight because someone started abusing it.

3. Their rankings are one of the more interesting signals for which models are popular, despite their flaws (most OpenAI and Anthropic users don't go via OpenRouter, it's currently not possible to tell the difference between many users switching v.s. one "whale" changing their preferred model)

Given how API costs are becoming meaningful for a lot of companies now, having a provider like OpenRouter to help measure your spend and easily experiment with and switch providers feels like a valuable service.

View on HN

Another neat thing is, they publish hourly caching states for ALL model/provider combinations. I did some research on it to come up with a provider tiers list and found a bunch of open-source 3rd party hosts are simply trash tier https://dirac.run/posts/cache-hit-rates-agents

Thank you so much for this! I've been working on exactly this problem this week (which OpenRouter providers have the highest cache rate on average) because cache cost is sometimes half your cost: I'd much rather use a provider with more input caching with a more expensive/better LLM. Your results and lists seem more comprehensive than what I've done so far. Very helpful!

Agents push the full conversation history into context every turn

Why?

Maybe this is a dumb question, but why wouldn't an agent "keep the conversation going", like I do when interacting with an LLM through a web page? (I understand how it's impractical for long-running tasks where the agent has to wait days for the next input, but assume that's not the majority of use cases)

The "web page" does the same you just don't see it.

I’m not sure I understand your question. Every interaction you have with a model in a web page does the same thing in the backend. It feeds the whole conversation history, perhaps with a bit of processing, into the model so it can process the next generation. Filling the context window is how these models retain coherence.

LLMs are stateless, to predict next tokens they need the history. When you write your own agents you will be very selective and might trim context and heavily segment different tasks, but generic ones don't do that (at best they spawn subjects to handle smaller tasks)

That said, the KV cache is very much not stateless, so internally inference APIs will be highly incentivized to route requests to instances with as much a shared prefix cached as possible.

Thanks. If I ran it local, presumably I could keep the state cached forever. Can you "reserve" resources from a frontier provider to guarantee your state stays "hot"? (Analogous to reserving a whole VM instead of a slice)

For Anthropic, 5 minute caching costs 1.25x base input price and 1 hour costs 2x base input price. https://platform.claude.com/docs/en/about-claude/pricing#pro...

For OpenAI, it seems like you can't prolong the caching duration for money. Duration is longer during off-peak hours for in-memory caching and up to 24 hours for extended prompt caching. https://developers.openai.com/api/docs/guides/prompt-caching

For DeepSeek, caching duration of at least 12 hours (and likely longer) have been observed. Cache writes are free. https://zhuanlan.zhihu.com/p/2035737726952194774

BTW, the openai responses api has a store parameter and a thread id input. Makes it possible to send a thread id and append a new message, ask for completion. So it feels like keeping the conversation going.

Technically it does retrieve the entire history and reevaulate it since the LLM is stateless. Just more ergonomic for the developer.

And prompt caching helps cut the costs down when a conversation drags on.

Wow, this is refreshing DX compared to iterating all messages like we did back in '24.

I would disagree. Having all the messages locally and sending them with the request means you can switch inference providers or even models mid-conversation. It also means that the provider doesn't store the entire context, which often contains massive parts of proprietary codebases, secrets and PII and instead the agent harness manages all that. While a simple `continue thread` API field might seem more convenient, the cost is still determined by the input token count and cache rate, so it just abstracts this crucial implementation detail away.

I would recommend tracking this data over time. I work on Cloudflare's KV cache for Kimi K2.6, and while there are periods where our cache rate is low, we are frequently in the 80-90% range. OpenRouter shows us at 87.3% at the time of this post. We observe cache rates change quite a bit from hour to hour.

True for Kimi, but the results I published are average across the models (CF has over 10 models on openrouter). Your current Kimi K2.6 is over 80% but Gemma 4 26B A4B is 0%. https://openrouter.ai/google/gemma-4-26b-a4b-it

This is also the reason providers like Anthropic scored lower because while Opus 4.7 is close to 90%, Opus 4.5 is 45%

My point was not about our ranking specifically, but the methodology of taking a point-in-time sample.

Good points. The easy experimentation factor is helpful for development, though I would gently encourage everyone to migrate to the 1st party APIs for pricing at scale.

OpenRouter is also a good place to find free LLM access with a catch: You should expect that any inputs and outputs are going into someone's training database. Clearly anyone who can pay should be using paid models with privacy protections, but the free models have been great for learning and experimenting. Especially for younger people learning API programming and LLMs who may not have access to a credit card or funds.

It’s interesting all the focus on opt-out from training. Sometimes I worry there is an intentional focus on that so people don’t think about the other ways the company might be profiting off our data. Like I pay for Anthropic and they don’t train on that but are they selling my “anonymized” usage data in some other way?

From what I recall, these companies don't offer any option to opt out of your session transcript data being used (and sold!) for "regular" adtech targeting purposes.

Anthropic explicitly state that they don't do this, even if you use the free plan and even if you don't opt-out of letting them use your data for training:

"We do not sell users’ data to third parties."

https://www.anthropic.com/news/updates-to-our-consumer-terms

That answers for the "sold" part but not for the "used" part.

I.e. nothing about this statement prevents Anthropic from running ads within Claude, as long as they run the ad-placement auctions themselves, and so aren't leaking any of the data they're using to decide which placements are relevant to which users+sessions. (This is the same thing Google does for SERP ad auctions.)

But actually, and perhaps more interestingly, nothing about this statement prevents Anthropic from building a Google AdSense competitor either. Other sites (or mobile apps, etc) could plop in an Anthropic ad iframe; and it'd be Anthropic's knowledge of your interactions with Claude that would drive what ads would show up in that iframe. The embedding site doesn't know what ads the users are seeing, so that's still not "selling users' data to third parties", per se.

> You should expect that any inputs and outputs are going into someone's training database.

True enough, in theory; but what exactly are you imagining would be a useful-enough signal in the OpenRouter request+response stream, that any company would want their data as training material?

Even a single OpenRouter-API-key-identified subscriber's traffic, may consist of an mixture of traffic from multiple different sessions, under potentially multiple different end-users. (Where, if the subscriber is doing security correctly, then their OpenRouter key lives on a gateway rather than in a frontend app; and so the only IP address / UA / etc OpenRouter sees is that of the gateway itself.)

And the traffic stream may also invoke multiple models, and provide multiple different system prompts for those models; which, while marked in the traffic (i.e. conveyed as part of each request), makes the resulting data much less useful in aggregate, than if it were all training data for one model with one system prompt.

Plus, there are no RLHF signals in OpenRouter data. Even if OpenRouter wanted to build a general model-neutral framework for collecting RLHF-type data, it can't force subscriber apps to do the UI-level stuff necessary to collect it (i.e. the things ChatGPT/Claude do, with "thumbs-down" buttons, A/B tested responses, etc.) Analysis would have to rely on pure transcript-level user sentiment extraction.

I've wondered this too - exactly how are our inputs and outputs useful as training data? So I asked Gemini. Apparently using negative sentiment in user or llm responses can serve as RLHF, and the human prompts can also serve as useful data for what problems the llms need to be able to solve. There's also that smaller models can train on and improve from data from larger models but that's less relevant when not switching models in context.

You get a 1% discount if you give OpenRouter your traces so at least they think there's some (a lot) of value.

I had no idea what traces are in this context. While looking, I found this post from @OpenRouter:

https://x.com/OpenRouter/status/2041193329270878707

  > Privacy:
  > 
  > Private I/O logging and the 1% data sharing discount are separate settings. You control each independently.
  > 
  > Input & Output Logging stores prompts and completions for your use only and makes them visible in your logs. OpenRouter does not access this data. You can configure it in your observability settings.
  > 
  > As always, more in the docs!

https://openrouter.ai/docs/guides/features/input-output-logg...

https://openrouter.ai/docs/guides/features/broadcast/overvie...

The logging thing is like langfuse

How about protection of intellectual property? Doesn’t have to be patented to be valuable.

> what exactly are you imagining would be a useful-enough signal in the OpenRouter request+response stream, that any company would want their data as training material?

Isn't this a treasure trove for any model distillation effort?

> Plus, there are no RLHF signals in OpenRouter data. Even if OpenRouter wanted to build a general model-neutral framework for collecting RLHF-type data, it can't force subscriber apps to do the UI-level stuff necessary to collect it (i.e. the things ChatGPT/Claude do, with "thumbs-down" buttons, A/B tested responses, etc.)

The majority of RLHF data doesn't need this. The majority is software development and/or tool calling where the agent gets a signal back as to if it succeeded (eg compilation errors, test errors). It's true that end-of-trajectory signals (eg, did this task do what you wanted) are even more useful but even partial signals are great for RL training.

I recommend nvidia nim for completely free dev access for young people.

It's free, but not unlimited. Besides rate limits, new sign-ups get 1000 credits (requests), and once those are gone, they're gone for good. Only business accounts might get a couple of free refills.

Is there a way to check/track your available credits?

It is unlimited under the free NVIDIA Developer Program. You're talking about a different sort of acct I think. The dev program acct is 40 rpm unlimited for personal use.

> Clearly anyone who can pay should be using paid models with privacy protections

Clearly, anyone who needs privacy should be using models with privacy protections. Some people build open source and the models will get the code anyway.

> You should expect that any inputs and outputs are going into someone's training database.

OpenRouter explicitly lets you filter by zero-data-retention providers: https://openrouter.ai/models?zdr=true

The main friction reduction, for me at least, is the consolidated billing that avoids extra bureaucracy in corporate environments. The API-translation/abstraction tends to cause more problems than it solves.

I’d prefer something that consolidates billing, but still lets me use providers' APIs directly (or via some "raw HTTP" proxy). There are plenty of unified API gateways, but I haven’t seen one that is just billing/auth in front of the native provider APIs.

At the moment for DeepSeek V4 it messes up caching and that's a key pricing feature for V4.

https://news.ycombinator.com/item?id=48319827

Did you know that if you put some money into your OpenAI account it expires after a year? I was very annoyed when that happened, no refund no warning it’s just gone as if it was a promo credit.

Openrouter is very nice since it puts a barrier between you and those suppliers that were supposed to be like utilities. I got the feeling that if OpenAI was left alone they would be nice as a telco.

The way how you manage the caps in OpenRouter is how every metered API provider should do it: keys have limits, and you can change the limits, and you set the limits to refill periodically, and you can create as many keys as you want.

Out of interest, why OpenRouter over a free option like Cloudflare’s AI gateway or another paid option like Vercel’s — any specific benefit to OpenRouter you’ve found, or just first you used that’s good enough?

I'll be honest, I hadn't clocked that Cloudflare and Vercel were offering equivalent products.

Looks like Vercel even have their own leaderboard: https://vercel.com/ai-gateway/leaderboards/models

Surprising that they have Opus 4.8 and 4.6 listed on the leaderboard but not Opus 4.7.

Huh, Claude Opus 4.8 is number 4 in number of tokens at 10%, not even in the top 10 in terms of requests, yet is #1 in costs at a whopping 43%!

I didn't know about these options either. I am using Cline: Cloudflare isn't an option but Vercel is. My spending is pretty low overall now that I'm using local models much more but good to know that there are cheaper alternatives to try or at least suggest to others.

Other features I've just noticed: - configurable prompt injection protection using OWASP regex (https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_In...) - configurable PIM protection for outbound prompts - input/output logging - "JSON healing" to auto-correct minor hallucinations

Lots of other stuff too. The business model seems pretty simple and the value-add features don't look particularly expensive or difficult to copy.

Separation, I don't want to have my domains blocked the day AI bill go Brrr

Make two accounts...

> a free option like Cloudflare’s AI gateway

Free? They take the same 5% fee as OpenRouter does.

https://developers.cloudflare.com/ai-gateway/features/unifie...

I love their product and use them myself. But where's the value proposition for investors? Unless they get purchased by one of the large cloud providers, they will get pushed out of the market sooner or later.

What's the value proposition for the typical AWS startup to go with openrouter, if Amazon offers similar rates with direct integration into all their other offerings?

The only reason OpenRouter can exist at the moment is because we are in the wild-west phase of this technology, and lots of people and companies are exploring. In 5 years they will have to have transformed their business fundamentally, or go the way of the dinosaurs.

They never claimed it was technically hard. Brand recognition is their forte. They found out there is a need, developped a product around it.

Yah I don’t think they have a long term play without a pivot

functionally they operate as a marketplace for cloud providers. I feel like there is value there, especially as API costs rise and companies explore cost-saving/efficiency. IMO, this is a particularly attractive value prop in the SMB space, where it is common to interoperate between multiple SaaS/software stacks.

AWS does not provide nearly as many different models as OpenRouter. Perhaps they have an incentive to not do that, move slower as a big company or more legal risks to consider. If AI model outputs becomes commoditized then having one place where you can switch effortlessly from one to the next based on price might just justify OpenRouter. It could become a commodity marketplace/exchange.

If you believe there will be lots of LLM providers in the future, then OpenRouter could be a DoorDash play.

Established restaurants didn't need DoorDash because they were already on everyone's speed dial. But new or small restaurants couldn't afford to advertise or maintain a team of delivery people. DoorDash created a two-sided marketplace that made it a lot easier for new entrants to bootstrap. Today even the established restaurants have to pay them their tithe because hungry people have learned to start with the DoorDash app. A bit of a prisoner's dilemma.

If OpenRouter plays its cards right and gets very lucky, a large number of people will configure their hungry LLM clients to start with OpenRouter, and then LLM providers will have to join the marketplace or else miss out on all those customers.

not sure that works as well when they don't own their API though; how much software is openrouter-only in a way that's not 5min of deepseek to patch the source for, or 15min of opus to patch the binary instead

Everyone (except Anthropic) seems to be settling on the same API, so nobody "owns it" anymore. I expect there to be practically no software that's OpenRouter-only.

https://openresponses.org/

I agree that technical lock-in wouldn't cause the consolidation. Instead, if it happened, it would be because of the network effects of the two-sided platform.

People could email cat photos and resumes. But Facebook and LinkedIn are where everyone already is, so that's what they use instead.

DoorDash is viable only because the restaurant business (minus national chains) is extremely balkanized. Restauranteurs have very little power.

Billing caps are underrated! I don't understand why they aren't present everywhere. As an indie dev there are some services I'm really hesitant on trying by fear of getting an enormous bill for a mistake, this is even more true with vibe coding IMO.

I’m just not sure they have a moat or a long term play? I put $20 in and tried a few models. Then I went right to the model provider to put in $1000 and avoid the middleman tax. Now imagine a big corp spending millions on AI. That’s a lot of middleman tax.

I tend to agree, but there's also a lot of tax to build and maintain the different provider abstractions that OpenRouter eliminates.

Everything has a cost of some sort. It's just who you're going to pay and what the currency is.

There are enough services that don't want the model provider to know who they are.

The top model / prices are changing all the time though. Lately I've been auditioning 4-5 models before a big ingest and I wouldn't be able to do that easily without OR.

And what do you do when Fireworks is down? If you stuck with Openrouter, when Fireworks is down it would auto route you to Friendli.

What if Fireworks stops offering your preferred model?

Honestly I am 98% on Claude, and when claude is down I suffer through GPT.

The value of openrouter isn't as a middleman for users of claude, gemini or chatgpt, it's for those looking to find a model that fills the use case at a lower price than the top 3.

Except the latency is significant and not suitable for clients with advanced agent features. The experience between using a frontier model via first party API and the best open weight models via OpenRouter is night and day. Can't get any real work done with it.

Good point. When I use it, the inference doesn't seem very fast compared to the big providers, esp Time to First (non-reasoning)Token.

They also do a good job working over the little differences between APIs. Tool calling sometimes breaks on major providers, and OR will patch it before the provider does. Libraries like LiteLLM do this too, but OR is faster.

It's not just comparing all the models, it's also comparing all the providers and configurations of those models.

If you're doing any kind of production AI work you'll end up with outages caused by calling a single provider, OpenRouter seamlessly switching between providers is a godsend for uptime.

But even more than that there's meaningful cost+speed differences.

Here's Sonnet 4.6 being served direct, via Amazon and via Google

https://la9q13gg8w.evvl.io/

(spoiler: Google was both fastest and cheapest)

Both OpenAI and Anthropic have billing caps… who doesn’t?

Google Vertex

https://aistudio.google.com/spend ? Monthly spend cap

I spent two hours the other day trying to figure out how to manage spend on gcp, i gave up and used openrouter and cloudflare.

> Long-running tasks like batch mode completions and agent sessions may incur overages beyond your project spend cap.

> Billing data processing times can be delayed in AI Studio, up to around 10 minutes. You may experience overages beyond your project cap if billing data hasn't processed before more charges are accrued.

https://ai.google.dev/gemini-api/docs/billing#project-spend-...

That's a soft cap, not a hard cap

AI studio added it recently, Vertex not.

Huh, so they do.

Anthropic: https://support.claude.com/en/articles/8977456-how-do-i-pay-... - you can pre-pay and get a hard cutoff.

OpenAI: https://community.openai.com/t/how-to-set-billing-limits-and... - last time I looked OpenAI had a soft but not hard limit, I guess they fixed that last year.

I remember bugging them both about this last year, I need to update my mental model!

Based on experience, Google Cloud. No idea if that translates to Gemini usage billing.

Gemini added prepaid billing and spending caps a few weeks ago: https://twitter.com/OfficialLoganK/status/204451626215244231...

I cancelled my whole GCP account a month ago because I was too afraid of getting charged hundreds of thousands overnight like all peoples on Reddit

I tried Alibaba Cloud. They have no caps. This was the reason to cancel my account there.

Deepseek has a prepaid model. (Pretty impressive, what fits into 10 Dollar)

Literally every credit card I own allows me to make a virtual card that is either single use or has a cap.

Like which one? Most I know don’t have this feature

My business card is a Cap one spark business 2%.. get 2% cash on everything which is nice.

Does not matter, you still owe what you used for a service

There is a scheme to send gifts with a compromised anthropic key even if the limit is reached.

Microsoft

Who doesn't have hard billing caps for inference? Microsoft, Google and AWS my friend. And you know who uses Microsoft, Google and AWS? Almost all big corporations do use them instead of direct OAI or Anthropic API because all their contracts and infra are built around the big cloud providers.

And what is their business model?

Credit is prepaid at a 5% surcharge.

middle man? Model providers/hyperscalers -> OpenRouter -> consumers?

coffee farmers -> middle man -> you

Commission on API calls extracted from you when you charge your account.

> By far the lowest friction way to support and try out all the models

Check out Kagi Ultimate.

Would you recommend Kagi Ultimate over OpenRouter? I'm already a customer of Kagi and would rather give them my money, but only if I'm not really compromising.

> Would you recommend Kagi Ultimate over OpenRouter?

For personal use, yes. The all-in pricing model encourages experimentation. And the privacy pitch seems tighter.

The privacy part is definitely important. Appreciate you!

Maybe someday the VM I run agents in will have a dedicated GPU so that I can stop using APIs altogether. One can dream...

How do you use Kagi Ultimate for programming though?

The difference between GLM 5.1, ChatGPT 5.4 and Opus don't really matter to me when I'm asking/talking, but programming is a different beast that needs a harness.

Unfortunately the model companies will simply reinject the friction by mandating BYOK (Bring Your Own Key -- i.e. the end user must onboard with each model company individually).

OpenAI and Anthropic have already done this.

Mandated BYOK will sink OpenRouter.

There is also the ability to fallback is one of the clouds degrades in performance.

Though you pay 5% fees? Not worth it for me with the volume of tokens used.

OpenRouter is merely only a proxy. They also host some open-weight models

I don't think they do. They proxy to a bunch of open-weight model hosts, but I've not seen that they host them themselves.

They don't list themselves on https://openrouter.ai/providers

OpenRouter has stealth models that are indicated as "by OpenRouter" but indicate an external provider.

https://openrouter.ai/openrouter/owl-alpha

What kind of compensation are they giving you?

None. It's sometimes possible for people to say something positive about a project without being paid!

(If they were paying me they got a bad deal, since I called out the flaws in their leaderboard approach half way through my post.)

The biggest benefit is that it creates competition among models. If more people use open weight models or models from other providers, it’ll be harder to ban them. Which is what OpenAI and Anthropic will try to accomplish. OpenAI by lobbying the Trump administration for favorable treatment (see Brockman’s MAGA PAC donations), Anthropic by using religious leaders and nonprofits to push “safety” justifications for difficult regulations.

Reply on news.ycombinator.com