Hacker News Reader

Comment by miki123211 | Hacker News Reader

I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible.

For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.

View on HN

Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests:

> Fireworks scored 46% on TAU, a 30 point gap

Another surprise was DigitalOcean being bottom of barrel too.

Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.

DigitalOcean (at least through OpenRouter) is pretty reliably bad in my experience. Fireworks can be great, but it depends on the model and on the day.

DigitalOcean is bad not just through OR... Direct accesss through DO gave me such a miniscule context and maximum tokens, it was pretty much unusable. So, accessing it through OpenRouter gave bigger context length, but model behaves like it was seriously damaged - tool calling was producing paths that had / replaced with some other symbols, files were not found etc., and this is just for things that errored out, I have no idea how bad reasoning was. I had to blacklist DO completely on OpenRouter.

It's not always deceptive. Bugs in inference can be very subtle (well, bugs everywhere can be subtle).

The providers should be benchmarking their offerings daily

Reply on news.ycombinator.com