Not having to deal with the monstrosity that is ESP-IDF is a huge benefit.
I don't want a framework that takes over an entire project and mandates the use of a given build system, configuration system, source code structure, etc. This tends to break apart when you want to have more complex build steps; eg. matrix builds, subsequent build actions on artifacts, integration with codegen like string interning, multiple target platforms which are likely not an ESP32 (like a simulation target running on the host), integration with linters/checkers, integration with test frameworks, etc.
And the technology choices ESP-IDF made are also... controversial (CMake and Kconfig! Plus a whole bunch of Python glue to actually make it work together).
Just give me libraries that I can build/link against and let me bring my own build system - or pick a build system that is actually extensible and will scale to more complex scenarios.
They even provide you with a fully offline SDK. Extract it and open a terminal and build things. No dependencies, no docker, no nothing.
Your criticism regarding their usage of cmake and the kconfig language hold a bit more water. I've never had any issue with the kconfig system (at least not since they've reimplemented the menu system in python instead of whatever it was) but I have hit CMake limitations a few times.