Part of our challenge (though not all of it) is that apps need to be defined twice: A single app needs configs, secrets, usually multiple "daemon" processes and background jobs, and then you of course have things like databases. We have the Kubernetes manifests for production/staging, and then the end-to-end tests have to define everything all over again. A single end-to-end test may end up using a dozen apps, so this is a lot of duplication.
Last I checked, CircleCI did not support running a single job across multiple machines? Some of our end-to-end tests run some very big things that require multiple machines.