I haven't used Postgraphile, but it sounds like you are just exposing your database directly out through a GraphQL wrapper. You'd be experiencing the same pain if you tried this approach with any API technology (gRPC, REST, SOAP, etc).
You should still have code for authorization and business logic, and your API should probably be exposing a focused subset of your data model unless your clients really need to be able to traverse the entire database.