It is the other way around. The ORM sits on top of core, the sql abstraction.
This separation is what I like about it, if I know how to do something in SQL I can use the tools sqla provides and it maps very close to 1:1. Or I can use a more ORM driven approach similar to what Django does. I can adapt it over time as the project evolves.
But yes to write the queries in Python and not raw strings, you need to have either a Table or ORM Model defined, which you could generate from the database [1]. But I admit, except for some scripts, I have never used it where Python wasn't the source of truth, so don't know what it all involves.
I don't know about very old versions of SQLAlchemy, but at least in the last 4.5 years since I'm using it, I have been much happier with it for medium sized projects over any other similar tool in Python I have tried because of this flexibility
[1] Edit: Or use reflection to load schema information from the database
>>> messages = Table('messages', meta, autoload=True, autoload_with=engine)
>>> [c.name for c in messages.columns]
['message_id', 'message_name', 'date']