I’ve been looking for a way to do integration tests for our pipelines. So far haven’t found anything I could use.
Basically it fires up elasticsearch using docker-compose and then the integration tests run against that. You could use a similar strategy to test different feature flag combinations.
For some of our private projects, we use kts to generate the github action yaml files using this: https://github.com/krzema12/github-workflows-kt
Well worth checking out if you have more complex workflows. Yaml is just horrible in terms of copy paste reuse. Also nice to get some compile time safety and auto complete with our action files.
The tooling around pipelines is awful. A single typo in some variable’s name in a later stage can take minutes to catch. The feedback cycles are very long (cloud machines are much slower than local ones) and IDE tooling is bare-bones.
Just give me one large Python file with some library to manage common actions (building up the job DAG, accessing pull requests, easy shell access, …). We’d have refactoring, Turing completeness, type safety and so much more. A core downside would be managing the complexity of DevOps scripting going berserk. Personally I’d prefer that trade off.
The kotlin scripting support for github actions that I mentioned addresses things like typos, refactoring, and IDE tooling. Try it, it's pretty nice and easy to use. We actually have an integrity check as part of our build that runs the kts script to verify the yaml file stored in the repository is consistent with what the script generates.
I'd love some feedback on what else I could add to this project to make life easier for people.
I'm continuously amazed that none of the major CI providers offer standalone tooling to run and debug your CI pipelines locally. Seems like it'd be a killer feature for anyone working with complex pipelines.