ORMs are basically trying to recreate SQL in their own special non-portable immature bug-ridden manner. Thus, even the schema has to be defined in the ORM and that is where the problem comes in.
With that in mind, most people don't want the availability of methods and classes in their application to depend on external things like the database schema. That makes error reporting and debugging tough. So, you dump the DB schema to some format your ORM understands, and then consider that authoritative.
Many of us consider the database a tool for storing and querying data, rather than a way of life. In our case, the way the ORM works is perfect. The database is abstracted away, and we can write our software in our language of choice.
In our case we have a config file because A) it makes it easier to abstract the schema across different databases and B) we have extra metadata in there above and beyond just the schema information.
Wrapping a relational database with an object layer is really a simple, straightforward problem. It surprising that so many smart people feel compelled to spend so much time and effort engineering overly complex solutions to it.
The problem with pretty much any kind of framework, ORM or not, is that eventually it becomes bloated by being flexible enough to meet the requirements of thousands of different developers, meaning that any given one of them only ever uses a small percentage of the overall functional footprint. Keeping things simple yet powerful is definitely the hardest part about it all, especially given that different users have vastly different opinions about what's simple or what power they need.
But the amount of polish that you need to turn an in-house thing into an open source project is just immense. And worse, by the time you have it flexible enough to handle every possible use case, you've bloated it out to the point where it's no fun to use anymore. Pick pretty much any off the shelf ORM to see the end result of that path.