100ms

Rereading this, I have no idea where 3.9 MiB/sec came from, that 200 requests/sec would be closer to 8 MiB/sec

Not quite $5, but a $6.71 Hetzner VPS

    # ioping -R /dev/sda

    --- /dev/sda (block device 38.1 GiB) ioping statistics ---
    22.7 k requests completed in 2.96 s, 88.8 MiB read, 7.68 k iops, 30.0 MiB/s
    generated 22.7 k requests in 3.00 s, 88.8 MiB, 7.58 k iops, 29.6 MiB/s
    min/avg/max/mdev = 72.2 us / 130.2 us / 2.53 ms / 75.6 us

NVME read latency is around 100usec, a SQLite3 database in the low terabytes needs somewhere between 3-5 random IOs per point lookup, so you're talking worst case for an already meaningful amount of data about 0.5ms per cold lookup. Say your app is complex and makes 10 of these per request, 5 ms. That leaves you serving 200 requests/sec before ever needing any kind of cache.

That's 17 million hits per day in about 3.9 MiB/sec sustained disk IO, before factoring in the parallelism that almost any bargain bucket NVME drive already offers (allowing you to at least 4x these numbers). But already you're talking about quadrupling the infrastructure spend before serving a single request, which is the entire point of the article.

Happy eyeballs, that makes a lot more sense thanks. Someone's "magic eyeballs" here apparently isn't reading his own writing :)

Aha that makes more sense, I thought it was specifically to do with job scheduling from the description. You can do something similar at home as a poor man's CDN by racing requests to regionally replicated S3 buckets. Also magic eyeballs (ipv4/v6 race done in browsers and I think also for Quic/HTTP selection) works pretty much the same way

> Google's strange job optimisation technique (for jobs running on hard disk storage)

Can you give more context on this? Opus couldn't figure out a reference for it

I dislike how Zen (and many similar cases, not picking on Zen here) report being not for profit or transparent, while the auto-recharge mechanism guarantees they are sitting on a float of at least $5 per account, and presumably an average of at least $10. That's something like 50 cents of interest income per year per account. It's not nothing and it's hardly egregious fraud, but I feel if they will do this when it's obvious what they're doing, what other corners might they cut

Honesty as a marketing strategy is really undervalued in cases like this

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output

Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, but simply exist for delineation, so there can be no way to express the token in the user input unless the engine specifically inserts it

Why is it half dead? They only jettisoned the ancient support library, the compiler itself AFAIK remains best in class and has commits on GitHub as of 15 hours ago

I don't understand enough about the US system of government. Are there any hopes of seeing Trump unseated before his term is up? If not for the astonishing damage he's doing to the western world, then only for the sheer fatigue from having every media outlet saturated by him on a daily basis.