Yes, but the Django ORM helped them launch faster. There is nothing wrong using it to start with and then writing your own SQL queries when you need to scale or even changing your data model to accommodate scaling.
... as in: check the output of SQL queries constructed by Django ORM and tweaked it in your Django Model layer such that the resulted SQL query will be tuned for performance?
Or at the very least, shouldn't Django ORM at least provide a way to retrieve data with "filtering" as optional parameters? (get only a few columns, but not all..)
If you're reading "just tables" without, or with simple relations and conditions, it's ok (because, as you pointed out, that's basically SELECT with)
The problem is when for example you have inheritance, or moderately complex relations.
Then you'll see Django spamming your DB with requests and the DB may be fast but the operation will be slow because of the sheer number of requests.
So yeah, in a sense you can optimize for it, or use SqlAlchemy (which is good, but it has a different philosophy than the Django ORM in the way it's used)
My experience is heavily in Java (JPA2 and Hibernate) and less of Rails (simple/moderate apps). Typically an ORM provides a way to handle custom queries but still help maps the result to the Java object model (which helps a lot vs writing getter/setter :)).
I don't know how people down south in Silicon Valley design their model (especially young scrappy startup) but I have a slightest doubt that they go that far (advanced/complex). It's not like startup start with "analyzing requirements" and move on to "modelling session" :) => they just go ahead and write code.
So in theory, the data model shouldn't be complex enough to warrant edge-case.
Use the tools to launch faster, then optimize.
Though I've had to work around some of Rails defaults for my use cases, there is no doubt that without it we'd not have been able to launch and iterate so quickly.