My experience with buildroot is that it's really slow to compile because it doesn't compile packages in parallel (so you'll only get the parallelism of an individual package's build system, with sequential stuff inbetween), and you end up recompiling from source a whole lot because it doesn't do dependency tracking between packages so if you change a library, you either have to manually remember to recompile the whole chain of dependents, or do a clean build. Yocto, on the other hand, compiles packages in parallel and tracks which packages need to be recompiled due to a changed recipe or config file.