in my experience the general sentiment is that the ORM is a leaky abstraction because it's common to make performance decisions based on the ORM's peculiarities.
up to a certain size, most ORMs will behave just fine! the bigger problem is when your mental model of how the ORM should be behaving for a certain data access pattern is thwarted by the actual implementation. This usually only happens when you notice a performance bottleneck around a semi-exotic query that you know to be trivial in sql or whatever.
it's also possible to put up performance abstractions between you and that ORM bottleneck, but often the experience and tradeoff sours most people. i don't blame them, but i agree that for most day-to-day cases, orms are pretty handy.