I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
[0] https://news.ycombinator.com/item?id=49244312
@dang, the creator of this idea is having his comments killed off for some reason.
The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists.
I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds.
I need to discard the requests that overlap the model changeover for a truer result.
The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this.
which is a decent excuse for a 24h telemetry update.
Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat.
That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves.
After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!