For some things ORM's get in the way, but avoiding them by your own data layer sounds like a pretty big time-suck.
Being able to write a single line of SQLAlchemy to build relationships into your models seems way simpler than implementing relationships in a custom data layer.