Hacker News Reader

Comment by kyrofa | Hacker News Reader

I would respectfully suggest that the author is misusing CI. If you have trouble running your tests locally, you have a problem. If you have trouble deploying from your local code, you have a problem. All of those capabilities should exist as simple scripts in your project already. Once you have that done, the CI yaml is a simple glue layer that defines an order of operations, e.g.:

1. Run static tests

2. If those pass, run unit/integration tests

3. If those pass, deploy

If you find yourself screaming about YAML, you're leaning too heavily on it and need to refactor your project's scripts.

Maybe a good question to ask would be "if I had to switch to another CI system today, how hard would it be?" If the answer is "hard", perhaps you're leaning too heavily on it and need to refactor your project's scripts.

View on HN

90% of the tricky parts of CI are secret management. As long as you write scripts to pick up credentials in a sane way (.awsprofile or similar) you should be able to configure your CI to provide the credentials just as well as you can locally - but in practice, the various different ways that things like artifact repositories, integration test databases, and cloud deployment tools want to manage auth is the cause of most of the complexity in getting your build/test/deploy pipeline working on the runner.

I would say the secret management part typically isn't the most insane or annoying part.

For me it's external system state management. Like making sure the integration test db is cleaned up correctly.

This is all fine and good if you are the principal developer of the project. However, the author makes it clear that he is migrating other people's CI pipelines. He is a DevOps engineer working across several teams.

This is why he makes the important point that discipline is not enough. The reason is most teams simply don't care. I find one in five teams where everyone on the team cares about the build (when I'm lucky!), most teams have one person who cares, and some teams have no one that cares.

When I am tasked with the proper care and feeding of the pipelines of others, I want tools that can work and help me out even when the developers who created the software are Holding it Wrong.

With these requirements in mind -- managing and migrating the many different CI piplines across an organization -- it would be a major breakthrough to have a tool that 1) transpiles to the workflows of all the CI tools and 2) allows for local testing. So many orgs have different teams using different CI stacks, and the local testing problem is always a struggle. I would use a tool like that into the ground.

So I would qualify your original statement: The author isn't misusing CI. Rather, the author is attempting to survive in a world where others are misusing it, and where the author is tasked with managing all the CI pipelines.

Testing locally, I agree with.

Deployments, I think it'd be fair to consider the requirements. At work, our softare can be tested locally but deployments are all registered against a central authority, and after a point of composing enough access requirements, only then does a role (cicd in this case) have enough policy allowance to perform a deployment.

The entire transaction is auditable. And I think that with a deployment, that's how it should be; allowing that trust down to a local environment strikes me that too much permission is accured with a single entity.

I guess that we could better define what a deployment is; to some nonprod environments I'd agree, but I'd still probably insist on the heavy machinery up at the test, perf, qa, areas, and then getting into staging and prod, there'd be no wiggle room.

Fair critique, totally depends on what kind of software we're talking about and where the deployment is happening. In general though, how screwed are you if your CI environment goes down? Can you not deploy anything? That would be scary.

My point, however, was mostly that the logic necessary to deploy should live as part of your codebase, not written out in YAML. The privs necessary to deploy are a separate discussion.

This sounds fine and well, but it's not how Github Actions work (or at least, not the encouraged workflow). Let's have a look at the snippet from one of the projects I work on:

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v2
  - name: Build and push mwdb-core image
    uses: docker/build-push-action@v4
    with:
      file: ./deploy/docker/Dockerfile
      tags: |
        certpl/mwdb:${{ github.sha }}
        certpl/mwdb:master
      cache-from: |
        type=registry,ref=certpl/mwdb:buildcache
      outputs: type=docker,dest=./mwdb-image
  - name: Upload mwdb-core image
    uses: actions/upload-artifact@v3
    with:
      name: mwdb-image
      path: mwdb-image

Good luck running this locally. There's no script code to speak of, just references to external "actions" and parameters (for example, https://github.com/docker/setup-buildx-action).

Some CI platforms are just a simple glue layer (Gitlab CI - which I prefer - is one of them), but in most cases Github CI is not. Maybe it adds to the author frustration?

Building it that way is a choice. It's not mandatory.

You can use gitlab CI with special-purpose docker images for all your steps and magic parameters driving everything too (Gitlab AutoDevops works that way).

But if you just run your steps in shell scripts in vanilla docker images containing your build-time dependencies, you should be able to produce something that works the same in any CI pipeline, or locally.

The most annoying thing for me is that a lot of CI engines make docker-in-docker complicated. I love using compose to set up integration test environments, but doing that in CI is often a fight.

> Building it that way is a choice. It's not mandatory.

This ^ . In GitHub Actions, I personally try to use pre-baked actions as little as possible, for exactly the reasons I outlined.

I prefer GitLab CI, but you can make a mess of that just as easily. In general, if you approach CI as I suggested, you end up with something maintainable regardless of the CI engine in use.

Yeah, Github Actions has its cool parts, but I still prefer Gitlab CI since its just 'dumber', and doing custom things is simpler to achieve.

As another nod to GitLab's CI, it has sane autocomplete in the editor since the schema is mostly static. I'm not aware of any GitHub yaml tooling that downloads the descriptors(?) for any actions in use and with then allow autocomplete of its uses: or with: blocks

> I would respectfully suggest that the author is misusing CI. If you have trouble running your tests locally, you have a problem.

I would respectfully suggest that you misread the author. The issue isn't running tests locally, it's running the CI config locally.

I experienced the same problem with gitlab CI years ago, where, basically, you can lint the file and not much more. Past that, you need to run it through your CI and debug if you get slightly different results compared to running a script locally.

Yeah, it's a horrible experience all around. CircleCI solved this problem like a decade ago, enabling SSH builds so you can troubleshoot straight up in the build itself, and once you've figured it out, just copy-paste the steps to your Makefile/CI config.

I don't understand how one could build a CI service so long time after CircleCI launched, and still not have that very same feature (or something similar).

The authors speak about how hard it is nowadays to provide a CI extension to the major platform including GitHub, Azure DevOps and Gitlab. (and others)

Wanting to run locally a developed extension is totally legit as some can be really tricky and depends on the behavior of [runner & OS].

Reply on news.ycombinator.com