Usually it's the inefficiencies of architectural decisions (your algos, layouts, etc.) you put into that rails app.
In small apps it's easy to keep track of the number of queries going to the DB per http request and their cost, but with bigger apps what I often find is ActiveRecord, services, policies, all calling the database way too much and too often.
Monitoring which individual queries are slow is easy, but it doesn't help if operations are overcomplicated. Even if you chase all obvious N+1s, you might still have "sets of queries that could have been joins" but are in different classes, so it's difficult to notice.
Other sanity check: nvme drives are ~10000x faster than hard disks were. Are you doing 10000x more useful disk IO than people were doing with hard disks?
Just as an example, here's the results of pgbench -S of a single client, with pgbench running on different servers (a single pkey lookup):
pgbench on database server: 36233 QPS
pgbench on a different system (local 10gbit network): 2860 QPS
It's rare to actually have that low latency to the database in the real world IME.
If the same pkey lookup is executed utilizing pipelining, I get ~145k QPS from both locally and remotely.
Even if you have it on another machine, it's on the same switch, right (also an architectural choice)? So latency should be sub millisecond, and irrelevant for a single request. So we assume we're talking about many requests, but then why not combine/batch your queries (also an architectural choice)? Now latency (and other things like database latches and fsyncs. Very important.) is amortized and again irrelevant. You should be able to hit close to the pure pg bulk query case (and e.g. a rust application server can do this in practice for me with a 4 core machine and an old SATA SSD with ~50k read IOPS).
Point is, the optimal case for building a web page is probably to do pg queries that return the xml of the page (letting you do subselects to combine everything into one query). So see how fast that is, and if your application can't match that, the database is not your bottleneck.
Because that'll be too much of a scalability limitation. Rails etc are rather CPU heavy. In cloud environments it's also typically much more feasible to scale the stateless parts up and down than the database.
> Even if you have it on another machine, it's on the same switch, right (also an architectural choice)?
Yep, was on the same switch in my example.
> So latency should be sub millisecond, and irrelevant for a single request.
In my testcase it was well below a millisecond (2.8k QPS on a single non-pipelined connection would not be possible, it implies a RTT <= 0.35ms), but I don't at all agree that that makes it irrelevant for a single request.