Building a platform to deploy stateless services are the easy part. This might even work nicely at a smaller scale. It is gets super complicated when you want to mount volumes, re-use those volumes. This is like a nice weekend project, contrarian in nature, but it is just not a replacement for kubernetes. This is some bash script orchestration on top of podman.
Bulk of complexity in opensource distro of kubernetes comes from the fact that everything is interface driven and has to work in an generic way. For example, you don't really have to implement cri-o like thing if you are opinionated platform and only supports docker. Similarly, you don't have to build CSI, if you only support ceph or aws' storage. I love the contrarian thought, but matter of the fact is modern orchestration platforms that want to be a general purpose platform is going to be big.
It is a really nice exercise however build something like this when you want to learn about networking, containers, orchestration, storage etc. and how to tied everything together.