If you have any complexity, programming against a good abstraction (Gradle is not good good, but decent) beats finding the magical incantation of configuration to get the tool to do what you want.
Of course, sometimes the limitations are good: preventing you from doing "the wrong thing", or encouraging cacheability, etc. But as with any abstraction layer - getting a model that fits across so many disparate use cases can be very challenging.
You can also download custom CLI tools and invoke them as part of a well-defined build lifecycle.
At the very least, I would move to Gradle which does have proper knowledge of your build graph. But Mill is also a good choice and fills the same niche, with the added benefit that imperative-looking ordinary scala code will simply become a parallelizable, cacheable build graph.