The Mythical IO-Bound Rails App
byroot.github.io
byroot.github.io
But then you need some authentication. And you need authorization, and more deeply than just "yea" or "nay" but doing non-trivial work to determine on some complex resource tree what is and is not permitted. And then you're taking a couple of queries and putting it together with some API call results in scripting code with some non-trivial logic. And then it grows beyond your dev prototype of 10 values to 100,000. And you use a few convenient features of your language without realizing the multiplicative slowdown they impose on your code. You add some automated plugins for turning DB values into these other things without realizing how much that takes. And you add some metadata-driven interpreter action based on what the user is asking for. And one feature at a time, you just keep stacking more and more on.
And it can really sneak up on you, but in a mature scripting language codebase it's really easy for it to not just not be blocked on database IO, but for IO to ultimately be such a tiny portion of the runtime that even if the IO was completely free that you'd still have a slow page.
Computers are very fast, and very good at what they do, and there's a lot of power to play with in a lot of cases... but it isn't always quite as much as programmers can think it is.
However to be able to do that, C with Classes abstractions needed to perform just as well as writing raw C.
What its related to is the narrative you shared in your second paragraph, which I think you were writing mostly for color, perhaps seeming to you as the only or necessary story of how software gets made.
"And then", "And then", "And then"... "One feature at a time, you just keep stacking more and more on."
There's no foresight. There's no planning. There's no architecture. There's no readiness. There's no preparation. Tomorrow's needs are always beyond the horizon and today's are to be approached strictly on their own terms.
If you've taught software development to anyone, you're used to seeing this behavior. Your student doesn't know what they don't know (because you haven't taught them yet), and they're overwhelmed by it, so they just scramble together whatever they can to make it past the next most immediate hurdle.
Much (not all) of the industry has basically enshrined that as the way for whole teams and organizations to work now, valorizing it as an expression of agility, nimbleness, and efficiency, and humility.
But it inevitably results in code and modules and programs and services and systems where nothing fully snaps together and operational efficiency is lost at every interface, and implementations for new features/challenges need to get awkwardly jammed in as new code rather than elegantly fleshed out as prefigured opportunities.
So you're right that most modern projects eventually just become swamped by these inefficiencies, with no ripe ways to resolve them. But it's not because of Rails vs Go or something, it's because pretty much everyone now aspires to build cathedrals without committing to any plans for them ahead of time.
What they gain for that is some comfort against FOMO, because they'll never face yesterday's plan conflicting with tomorrow's opportunity (since there is no plan). But they lose a lot too, as you call out very well here.
And whatever truth there is in that, it’s all out of balance and lacks pragmatism. Sometimes the pragmatic choice isn’t to keep slapping shit together, it’s to step back and see what the big picture is. Even if you don’t have PMF yet, you need to design for something on the off chance it succeeds or you need to pivot.
And it’s not even a pure tech issue, it can be as simple as thinking how a team of engineers could contribute, how you could onboard someone new and have them up and running quickly, what happens if you fire someone who kept everything in their head…
The best environment I've ever worked in was, ironically enough, fully invested in Scrum, but it wasn't what's typical in the industry. Notably, we had no bug tracker[0], and for the most part, everyone was expected to work on one thing together[1]. We also spent an entire quarter out of the year doing nothing but planning, roleplaying, and actually working in the business problem domain. Once we got the plan together, the expectation was to proceed with it, with the steps executed in the order we agreed to, until we had to re-plan[2].
With the rituals built in for measuring and re-assessing whether our plan was the right one through, e.g., sprint retrospectives, we were generally able to work tomorrow's opportunity into the plan that we had. With the understanding that successfully delivering everything we'd promised at the end of the sprint was a coin toss, if we were succeeding a lot, it gave us the budget to blow a sprint or two on chasing FOMO and documenting what we learned.
0: How did we address bugs without a bug tracker? We had a support team that could pull our andon cord for us whenever they couldn't come up with a satisfactory workaround (based on how agonizing it was for everyone involved) to behavior that was causing someone a problem. Their workarounds got added to the product documentation, and we got a product backlog item, usually put at the top of the backlog so it'd be addressed in the next sprint, to make sure that the workaround was, e.g., tested enough such that it wouldn't break in subsequent revisions of the software. Bad enough bugs killed the sprint and sent us to re-plan. We tracked the product backlog with Excel.
1: Think pairing but scaled up. It's kinda cheesy at first, but with everyone working together like this, you really do get a lot done in a day, and mentoring comes for free.
2: As it went: Re-planning is re-work, and re-work is waste.
If you're going to be concerned about the sort of things you mention, then as an engineer, you need to turn those concerns into significant negative value placed on the option of dynamic scripting languages.
Which is not to say that static languages are immune... indeed, I marvel at times at the effort and ingenuity put into bringing these issues at great cost into the static languages. Log4Shell, for instance, fundamentally stems from Java, with great effort, importing a very dynamic-scripting-language style reflection feature from the dynamic world, and then getting bitten hard by it. That's not a performance issue in this case, just a vivid example of that sort of thing. You can with enough effort layer enough frameworks and indirection in any language to make anything slow.
At one point I shaved almost 20% off TTFB just by cutting the setup overhead of all that telemetry, tracing, logging, feature toggle, etc code that had become idiomatic and set up first thing every time you instantiated an object. By my estimate I cut the overhead by less than 2/3 (couple big dumb mistakes, and less than 1/2 of what was left after), so that’s still over 1/8th of the overall page weight. All before a single service call. But at least I think I pulled the slope of the perf regression line down below the combined trend of faster VMs and faster EC2 instances.
Bookkeeping and other crosscutting concerns add up, particularly when every new feature gets it. It’s that old joke about how you can put so many gauges on an engine that you know its precise speed at all times, but that speed is 0.00.
Most databases make it trivial to see how long your queries are taking. Use that to help you identify the performance bottlenecks! If a request/response cycle takes 200 ms from the entrypoint of your rails server, and the database query reported (from the db) took 1 ms, that's a whole lot different than if the db query took 150 ms.
On scaling in general, in my experience inefficient queries are the vast majority of performance problems for most apps. It's usually fairly easy to identify these as different endpoint will perform very differently so you can get an idea of the baseline and identify aberrations. If that's happening then check your queries. If the whole app (baseline) is slow, TFA has phenonmenal guidance.
Yes! And add in that you are holding a database connection for longer than needed, putting scaling pressure on your DB, which is the harder thing to scale.
I thought this was low CPU but it was not low CPU enough, in the end I switched to gunicorn (running in WSL) with IIS fronting and serving the images (woulda been nginix in a pure-Linux environment) I think the tiny amount of CPU to serve the images was still getting blocked by requests that used a little more CPU. With gunicorn I'm not afraid to add tasks that crunch on the CPU a little harder too.
[1] built a lot of mini-apps that worked great before
As sysadmin I've seen many cases when not letting backend to serve static files drops latency and load significantly. In the worst case, X-accel-redirect is still better than serving through most of the frameworks.
RAC isn't sharding. The Oracle JDBC drivers do support automatic sharding across independent databases or clusters too, but RAC is full horizontal scaling of full SQL including joins.
It's also worth noting that if your Ruby app is CPU constrained you could take a look at TruffleRuby. It runs on the JVM, has a full JIT compiler more powerful than YJIT, can deal with native modules, and doesn't have a global interpreter lock (threads do really run in parallel).
Although it often gets lost inside app frameworks, you can use the Oracle drivers to dispatch every query at once in one request, or even just execute a stored procedure that has all the queries for a page registered inside the DB then work through the results from each query client-side. So you can really optimize the latency heavily if you can structure your app in such a way that it can exploit such things. JDBC supports this for instance but Active Record doesn't.
Too many folks lose on great technology because of hating the man, 70's hippie style.
And yes, I have used Postgres and MySQL and such.
It is similar to all the programming languages that have never moved beyond raw command line and basic editor tooling, only better nowadays, because of Microsoft (that they hate) making VSCode (driven by Eric Gamma of Eclipse fame), and LSP a common thing for all workloads.
I mean even mentioning MySQL is better than Postgres in some areas is not even welcomed on most of the tech internet including on HN. MySQL reached 9.0 after so many years and its news doesn't even reached HN's front page.
>Too many folks lose on great technology because of hating the man, 70's hippie style......... It is similar to all the programming languages
And It is not just PL. It is pretty much all across tech spectrum, and even beyond tech. I am a tech enthusiasts, not an PL, Editor, DB enthusiasts. We should be able to admit Oracle and SQL Server are better now but we strive to make similar if not better DB that is open source.
At least that's my understanding. I've not tuned an app that runs on such a cluster.
Moving to a fully concurrent runtime (rust, golang, JVM, beam, etc) suddenly makes so much of that no longer a concern.
I read their comment as needing an external database pool to scale, not that they don't need a pool.
When running JVM or the other runtimes, the database pool can be part of the application itself because it uses a different threading model.
To handle 25k qps with Rust, you can run 10 boxes. Maybe less.
Deployments are complicated (even when we wrap them in simple abstractions). A "rare" delay that impacts 1% of boxes during a deploy will become common on 100 boxes but still somewhat rare on only 10.
Running a single rust process per box allows a connection pooler to be in-process. It's more efficient, it supports postgres prepared statements, it's one less operational headache, etc.
It's interesting how easy it is to default to surface-level fixes instead of addressing the root cause. Have you thought about adding a case study or some benchmarks for common workloads? It could really help teams put these ideas into practice.
I haven't taken the time to fully analyze the data, but surface level show lobster.rs is indeed ~40% IO, as suspected: https://gist.github.com/byroot/e1fcfd1f8e172f5d9c5bfecefebdb...
Taking your figure of 300MiB per process, and assuming you might want to run 1.5 process per core to account for IO-wait, that's 450MiB per core (ignoring copy-on-write).
If you look at various hosting offering, most offer something like 4GiB of RAM per core (e.g. EC2 "general purpose" instances).
As for the lack of concurrency support, Active Record has asynchronous queries since a few versions, and is now even async/fiber compatible. So if you truly have an app that would benefit from more concurrency, you can do it.
And they don't know both such places are actually faster than the trivial computation because their code runs way slower than the ones at the origin and destination.
Hence why afterwards I never been a fan of any language for deployment production without AOT or JIT tooling on the reference implementation.
There are many assumptions here especially not considering anything about the database itself. The Rails issue also considers benchmarks which when benchmarking Rails would configure the database to not be the bottleneck. That's not true of real systems. The advice to use async queries for an IO-bound app could backfire if the reason the queries are slow is because the database is overloaded--adding concurrent queries only increases its thrashing and latency.
The best thing to do is consider the whole system. Don't throw the everything that's not the Rails app as IO. Is it actually doing network IO, or is it CPU, memory, or IO bound in the database? Maybe its not even a lack of CPU proper on the database but that it's being wasted on write contention/locks. Only then will you be able to choose the right course. Another way to go is blindly try different configurations and use what works well without full understanding, which is fine until you have an outage you can't explain then scale everything just-in-case.
The ways I've seen this play out:
- small apps (few users) just need good schemas and queries
- medium apps scale(*) webapp instances that solves client CPU & memory and vertically scale single database writer instance
- large apps scale database via sharding or federation/microservices
- other apps may not be able to shard a single tenant so also scale via federation/microservices
(*) Even if webapps are the CPU/memory bottleneck, overscaling them can make the problem of too many connections to the database for long held transactions.
It could be that the original files are used by other processes, and that they for some reason don't want to create two separate files. Maybe an issue with office politics (works on a different team), or an attempt at saving disk space
As it runs on windows, I might move it to linux and add sed or something in the chain, losing the php completely. But for now, php it is.
Instead of guessing if the DB is performing badly by looking at Rails, why not go directly to the DB?
First, I/O bound isn't necessarily a very well defined term, but generally speaking, it implies that you can't substantially speedup such system by speeding up code execution. The fact that YJIT did yield substantial performance improvement in the real world suggest many Rails apps aren't in fact strictly I/O bound.
Now about the 15-30% vs 2-3x, what I mean by that is the benchmark where YJIT yield this much are mostly micro-benchmarks, on more complex and heterogenous code, the gains are much smaller, hence we can assume the "Ruby" part of these applications wasn't improved by this much.
So for YJIT to be able to yield `15-30%` latency gains, this latency must have been in large part composed of Ruby execution. One last thing to note is that YJIT can only speedup pure Ruby code, in a Rails application, a large part of the CPU time isn't in pure Ruby code, but in various methods implemented in C that YJIT can't speedup.
Ultimately the lobste.rs maintainer kindly offered to run an experiment in production, so we should soon see if my assumptions hold true or if I was way off, at least for that particular app: https://github.com/lobsters/lobsters/pull/1442
Edit:
> Instead of guessing if the DB is performing badly by looking at Rails
That isn't really the topic though. The question is more about how much Ruby's GVL is actually released, and its implications.
Suppose that the CPU-intensive part of the program was actually speed up by 3x. This means that previously it took 1.5x as much time as the one that was saved, so thoughly on the order of 22-45% of the total time of your program. In practice the speedup was probably lower than 3x, which means the 22-45% figure would be even higher.
These are really really high number for a system that was supposed to spend most of the time (>90%) blocked in IO (which was not speed up by YJIT).
For comparison, if the CPU-intensive part actually took 10% of the time, then 10% would be maximum speedup you could get (which is way lower than 15-30%) and a 3x speedup would only result in a ~6% speedup.
In a real production setup, that timing would be reported into Datadog / NewRelic or whatever APM service, but collected in a similar way.
Other sanity check: nvme drives are ~10000x faster than hard disks were. Are you doing 10000x more useful disk IO than people were doing with hard disks?
Just as an example, here's the results of pgbench -S of a single client, with pgbench running on different servers (a single pkey lookup):
pgbench on database server: 36233 QPS
pgbench on a different system (local 10gbit network): 2860 QPS
It's rare to actually have that low latency to the database in the real world IME.
If the same pkey lookup is executed utilizing pipelining, I get ~145k QPS from both locally and remotely.
Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? So latency should be sub millisecond, and irrelevant for a single request. So we assume we're talking about many requests, but then why not combine/batch your queries (also an architectural choice)? Now latency (and other things like database latches and fsyncs. Very important.) is amortized and again irrelevant. You should be able to hit close to the pure pg bulk query case (and e.g. a rust application server can do this in practice for me with a 4 core machine and an old SATA SSD with ~50k read IOPS).
Point is, the optimal case for building a web page is probably to do pg queries that return the xml of the page (letting you do subselects to combine everything into one query). So see how fast that is, and if your application can't match that, the database is not your bottleneck.
Because that'll be too much of a scalability limitation. Rails etc are rather CPU heavy. In cloud environments it's also typically much more feasible to scale the stateless parts up and down than the database.
> Even if you have it on another machine, it's on the same switch, right (also an architectural choice)?
Yep, was on the same switch in my example.
> So latency should be sub millisecond, and irrelevant for a single request.
In my testcase it was well below a millisecond (2.8k QPS on a single non-pipelined connection would not be possible, it implies a RTT <= 0.35ms), but I don't at all agree that that makes it irrelevant for a single request.
In small apps it's easy to keep track of the number of queries going to the DB per http request and their cost, but with bigger apps what I often find is ActiveRecord, services, policies, all calling the database way too much and too often.
Monitoring which individual queries are slow is easy, but it doesn't help if operations are overcomplicated. Even if you chase all obvious N+1s, you might still have "sets of queries that could have been joins" but are in different classes, so it's difficult to notice.