Its biggest Achilles heel IMO is the need to always compile everything from scratch with no possibility of caching. Also the fact that incremental rebuilds after changing config options isn't guaranteed to work unless you manually know to know which packages to force nuke and force to recompile (this is intentional design choice for implementation simplicity and well documented). Also having two different boards that inherit from common shared config isn't really supported, everyone just rolls their own custom config splicer at that point.
Yocto fixes those issues, but IMO the most other things are much less ergonomic. While buildroot is extremely simple, yocto is total opposite to that, an architecture astronaut's dream. There is custom language to learn, ton of functionality, config and documentation split in tiny files in different repositories. Of which everyone using slightly different subset to accomplish almost the same thing.