Has anyone actually managed to find that this is the case? Just because your app is stalling on db queries doesn't mean it's "IO bound". Even the most trivial service that's just routing messages between two other services is likely to be radically faster when implemented in Rust vs JS.
OK, "IO bound" may not be the best description in this case. Because you're bound not by your ethernet card or operating system drivers, but literally by the work that's happening on a separate, database server's CPU. I should work on my terminology, thanks for the correction.
In my experience, it is usually waiting for DB. Regardless of whether your web app is well or badly written, it would get, at my estimate, would get no more than 5-10% request time improvement even if you made the app CPU infinitely fast, as 90-95% of the time is taken waiting for that database query.
But if you spend the same developer time rewriting all your ORM-generated n + 1 select statements to a single SQL query, you would still spend the same proportion of time waiting for the database, but your overall latency would get 10 times less. I've had some situations where I could get 10-20 time performance improvements by not touching application code at all, just tweaking the database indexes.
So, rewriting it all in a new language where you can, at most, 10% improvement, while you probably have areas where you can get 200-500% improvements seems unwise.
> Almost every web app.
No? My point is exactly that almost every web app is not io bound. They're just... slow. They write bad queries, they make bad usage of resources and block on requests, etc.
> Even WhatsApp scale runs on a “slow” dynamic language.
But that has nothing to do with being IO bound. Whatsapp, via Erlang, works by having tons of connections on a single box. And you can watch plenty of talks by them about how they've had to work with BEAM to get Erlang to actually be IO bound.
"IO bound" is a meme. Most people just mean "my queries are slow". Unless you're saturating your NIC, you're not io bound with regards to throughput, and unless your queries execute in ~<100ns you probably aren't IO bound on latency either.
However the important point is that they aren't CPU bound either. Most are "user acquisition bound" (like tens of requests per second peak). Some are more like "developer time bound" (cutting the server farm in half is not worth the cost in dev time).
Either way, using a slow development framework (including language, DB, etc.) that the developers like is a reasonable choice for what they're trying to do. (I personally wish most software was faster, but they have no reason to care what I think.)
Maybe the terminology around this could be developed some more.
4G latency 50ms
AWS inter AZ latency 1ms
Thread context switch 2us
The response times from things like search APIs even at FANG companies with vast resources and highly skilled engineers are often in the hundreds of milliseconds.