(And because its "config" files are arbitrary turing-complete code, its IDE integration is never going to be as good as Maven's)
And as in all things if your code gets too complicated you need to refactor, extract logic into methods/classes, etc.
I do have a problem with Gradle which is that it is almost entirely magical unless you are a pretty advanced Groovy programmer to understand how it is doing what it is doing. I have never felt more disoriented than when trying to learn how to customise a simple aspect of my build and having people post snippets that work but seem completely disconnected from anything else in the build process.
And maybe this is making a virtue of necessity, but I find the overhead of creating a plugin stops people from putting random "different compiler arguments on a Wednesday" conditionals in every build, which is all too common if you give them immediate access to a turing-complete language.
Your criticism applies equally to non-build code. Don't approve PRs for bad/inscrutable code.
In gradle you can build plugins just like maven. But a simple 'if then' doesn't require all the heavy lifting of a plugin.
But code is where logic goes. People expect logic there. It's the same objection as to logic/conditionals/looping in web templates, or routing config files, or persistence object mappers. If it's logic, it belongs in code!