Your CI has to be fully codified, stateless and possible to redeploy with a single command. That's the only way it can remain sustainable. No persistent hidden state, no manual configs (even as an option!) and automatically rebuilt on every release as the new version is deployed.
As a really big bonus, that also makes your CI testable.
In the previous job, we built such a thing: https://smarketshq.com/building-a-reproducible-ci-system-for...