Hacker News Reader

Comment by monster_truck | Hacker News Reader

Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty significant gains without introducing any new problems.

I'm happy

View on HN

50% cache hit is really low - in a standard agentic loop you should expect like 99%+ cache hit percentage (which should also lower that $12.50 to like a couple of $ for the same amount of tokens).

If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking tokens or compact tool call results to reduce context size but it's a trap - you want to never change history because of how cheap cache is, even more so with deepseek because their cache hit pricing is so low compared to most other models.

In my experience, that's the OpenRouter tax. Even a session that does everything right to remain sticky ends up getting moved between providers on a few requests, which bills you the full context as input every time the switch happens.

I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).

This has not been my experience. Generally I do pin to 1 provider, or 1 provider with a couple fallbacks (especially with deepseek - most providers are 10x the cached token price compared to deepseek themselves), but even when I don't I still usually see 99%+ cache hit percentage. Specifically using pi with various ad-hoc customisations (that I was careful not to break prompt caching with).

then what is the point of using operouter for this model? Just use the deepseek API and save the 5% fee on top of the better caching rate.

Because they don’t want to sign up for 10 different providers and subscriptions/etc, especially if some models are just going to receive light, or rare usage?

Why not pin to specific openrouter provider and disable fallback?

There is only one provider for this model, so shouldn't be running into that.

Seems like pro 0813 is exclusively served by Deepseek themselves at the moment so I wouldn't say that's the case?

It is a pain from openRouter if you don't define your providers correctly, but for DeepSeek, surely not- the weights aren't released yet and there's only one provider, DeepSeek.

With Deepseek as the provider, there's no issue of course, but that means you don't filter providers for data retention, and you could also choose direct API use with them at that point.

Fallback are still very useful and won't poison much your cache hits too much if the provider is down anyway.

That’s a very heavy tax tbh.

You can set it up to always use the official provider.

Thank you for the advice but it looks like it's an issue with Deepseek's dashboard, I think they are calculating the value incorrectly.

This prompted me to dig back through the logs, it seems to be closer to 96% at its absolute worst and 99.3 at best.

I only sell to openrouter, would much rather support Deepseek directly.

Can you tell me about your engine?

It's ~a traffic simulator with the fidelity of a rally sim like DiRT Rally or Asseto. Full engine/drivetrain, grip modelling with tire deformation, suspension, aero, collisions, etc. I just really like Rally Racing and Car Accidents

The absurd goal was to be able to simulate all of the active traffic in NYC, so 3-500,000 cars, without using any of the macro flow corner cutting that you see commercially or academically.

Initially thought that one machine was not going to be enough to do this in realtime so I spun off a very big fork and built out a webtransport stack to split the effort over a local network. It was promising until I also wanted to cover the highway in thousands of giant beach balls [1] [2].

In the process of leveraging codegen to shrink the data that needed to be relayed by >10000x (packing and dynamically updating sparse continous arrays of floats), threw in a fuckton of LOD work (both spatial & temporal), statistical aggregation, and a lot of differential equation bullshit to derive LUTs. It's at the point where a base model M2 mini can handle much more than that by itself. All of the networking effort paid large dividends in cross thread coordination and lock-free data passing. Went from struggling to fit each of the sim kernels for just 32 cars @ 60hz (~11ms) to 0.03, 0.0003ms p99s with the corresponding jumps in car count (north of 32k/thread). Multiplayer works well enough, it falls apart where it should (~128 people or LLMs driving around in the same square quarter mile, and a great deal more NPCs, latency permitting)

The remaining work is making it look and sound cool as fuck. Building out a physically-based audio synthesis engine that simulates the pulses of exhaust gas starting from the cylinder count/size/firing order, intake and exhaust count and size, header and exhaust configuration, and another one for the tire sounds, and another one for the collisions, and then rendering those out to wavetables so it scales and frees up the cycles for occlusion. And then getting it to visually render and control performantly in a browser tab, lots of instancing and shader work.

There's also some bullshit cooking that uses the motion estimation built into GPU video compression engines for ... other purposes at stupid low latency. I figured that's what Waymo had to be doing so I let it rip

It's fucking nuts, I'm having so much fun :) It will be done when it's finished

[1]: https://i.imgur.com/BDQSuLv.png [2]: https://i.imgur.com/CpreOWT.png

Can you explain how you used 12 billion tokens to do useful work?

(Not the original commenter.)

You can rack up quite a lot of tokens if you ask it to try out a lot of things, eg for performance investigations and trying out optimisation ideas.

is there a standard pattern for this? Like spawn an agent for each technique to try?

I regularly do a “go to DynaTrace, look at how this service gets used in production then use a profiler and see if there’s any low hanging optimisations we could make” on stuff. LLMs are really good at doing everything that was fun about software development.

Ha, I have a whole hobby work-stream going on about finding low-hanging fruit in various open source projects to turn into valuable contributions.

Two premier sources: (1) look at good contributions someone already tried to make, but that got stuck in review or were otherwise abandoned. (2) look at user reported bugs and see if we can find a user reported bugs, and see if we can reproduce and fix cheaply.

Both are explicitly scoped as best-effort affairs: move on, if you can't quickly make progress.

Most of the work I have to do as a human is review and navigating the submission process: tokens are cheap these days, so you really need to make sure the contribution is actually worth someone's time to review.

maybe this? https://github.com/karpathy/autoresearch

Not sure. I usually tell the agent to spawn subagents at will. (And they are doing that on their own anyway.)

Writing custom WMMA/MFMA kernels for an exact integer matrix multiplication library that uses RNS & CRT + Int8 GEMM to get ~90% of the theoretical i64 TOPS output from a 7900XTX (3.9 TOPS vs the 0.5 or so you get with naive hip-direct usage)

I'm not sure if you think that's a lot, but that was barely even 8 hours. I've had 200B+ months lol

Reply on news.ycombinator.com