Having just found Yocto yesterday and built my first custom image that I made by extending the layer in the OP, I am not yet "married to" Yocto.
Yocto initially appeared to be exactly what I was looking for. When someone else raised concerns about Yocto ITT I somewhat brushed them off, mainly because they were talking about managing the layers on a headless system. I used Yocto from the command-line on my workstation, so a) I am not on a headless system and b) even if I was doing this on a headless server, everything I have done so far would be done exactly the same way.
But, since then, multiple other people have raised concerns about Yocto. I also talked to one of the mods on /r/raspberry_pi and they said something similar. I looked through some past discussions about Yocto on /r/raspberry_pi using the search feature on Reddit, and found a couple of other similar experiences.
After looking through what has been said, I find that these concerns resonate with me rather deeply. They are all pointing to a single sentiment that seems to be rooted in similar real-world experiences; Yocto makes things complicated in a way that with time will make managing things needlessly complex and which will result in a lot of wasted time.
I was still a bit inclined to think though that perhaps these experiences were had long enough ago that Yocto has since improved, but after a little bit of further thought I realized that fundamentally Yocto is engineered in the fashion that you and others are pointing to as being a fundamental flaw of the system.
I am going to try Buildroot instead. Same guy that wrote the post I shared in the OP also made a post about using Buildroot. https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Buildro...