Hacker News Reader

Comment by tobinfekkes | Hacker News Reader

This is the joy of HN, for me, at least. I'm genuinely fascinated to read that both GitHub Actions and DevOps are (apparently) so universally hated. I've been using both for many years, with barely a hiccup, and I actually really enjoy and value what they do. It would never have dawned on me, outside this thread, to think that so many people dislike it. Nice to see a different perspective!

Are the Actions a little cumbersome to set up and test? Sure. Is it a little annoying to have to make somewhat-useless commits just to re-trigger an Action to see if it works? Absolutely. But once it works, I just set it and forget it. I've barely touched my workflows in ~4 years, outside of the Node version updates.

Otherwise, I'm very pleased with both. My needs must just be simple enough to not run into these more complicated issues, I guess?

View on HN

It really depends on what you do?

GitHub CI is designed in a way which tends to work well for

- languages with no or very very cheap "compilation" steps (i.e. basically only scripting languages)

- relatively well contained project (e.g. one JS library, no mono repo stuff)

- no complex needs for integration tests

- no need for compliance enforcement stuff, especially not if it has to actually be securely enforced instead of just making it easier to comply then not to comply

- all developers having roughly the same permissions (ignore that some admin has more)

- fast CI

but the moment you step away from this it just falls more and more and more apart and I every company which doesn't fit the constraints above I have seen so far has non stop issues with GitHub Actions.

But the worst part, which maybe is where a lot of hatred comes from, is that it's there for cheap maybe even free (if you anyway pay for GitHub) and it doesn't need an additional contract, billing, etc. Not an additional vetting of 3rd party companies. Doesn't need managing your own CI service etc. So while it does cause issues non stop it also seems initially still "cheaper" solution for the company. And then when your company realizes it's not and has to setup their own GitHub runner etc. it probably isn't. But that is if you properly account dev time spend on "fixing CI issues" and even then there is the sunk cost fallacy because you already spend so much time to make github actions work and you would have to port everything over etc. Also, realistically speaking, a lot of other CI solutions are only marginally better.

> languages with no or very very cheap "compilation" steps (i.e. basically only scripting languages)

This is not true at all. It's fine with Haskell, just cache the dependencies to speed up the build...

except that

- GitHub Action cache and build artifact handling is a complete shit show (slow upload, slow download and a lot of practical subtle annoyances, finished off with sub-par integration in existing build systems)

- GitHub runners are comparatively small, so e.g. larger linker steps can already lead to pretty bad performance penalties

and sure like I said, if you project is small it doesn't matter

I see the slow cache problem as universal. On a single-machine gitlab runner instance the upload to _local_ cache seems to take ages, double digit number of seconds for 100 MB archive.

GitLab drives me insane with this. My runner is on a 5 year old alienware desktop with an nvme ssd and all the trimmings but loading an uncompressed cache from the same local disk takes ten goddamn minutes. No matter what, all of my jobs take five to ten minutes just to start up.

The caching is atrocious and seems to not correctly restore our cache randomly in parallel jobs in the same run. It’s the worst CI that I’ve ever used.

> no need for compliance enforcement stuff

I find github actions works very well for compliance. The ability to create attestations makes it easy to enforce policies about artifact provenance and integrity and was much easier to get working properly compared to my experience attempting to get jenkins to produce attestations.

https://docs.github.com/en/actions/security-for-github-actio...

What was your issue with it?

They also work very well to leak all your secrets and infect people who download your software from pypi :D

> Also, realistically speaking, a lot of other CI solutions are only marginally better.

This is the key point. Every CI system falls apart when you get too far from the happy path that you lay out above. I don't know if there's an answer besides giving up on CI all together.

> But the worst part, which maybe is where a lot of hatred comes from, is that it's there for cheap maybe even free (if you anyway pay for GitHub) and it doesn't need an additional contract, billing, etc.

Or even if you pay $$$ for big runners you can roll it onto your Azure bill rather than having to justify another SAAS service.

I use GH actions. You should treat it like all build systems: let them do what they are good at and nothing else. The rest should be shell scripts or separate docker containers. If it gets complicated, dumb it down to "run this script". Scripts are a lot easier to write and debug than thousands of lines of yaml doing god knows what.

The problem isn't github actions but people overloading their build and CI system with all sorts of custom crap. You'd have a hard time doing the same thing twenty years ago with Ant and Hudson (Jenkin's before the fork after Oracle inherited that from Sun). And for the same reason. These systems simply aren't very good as a bash replacement.

If you don't know what Ant is. That was a popular build system for Java before people moved the problem to Maven and then to Gradle (without solving it). I've dealt with Maven files that were trying to do all sorts of complicated things via plugins that would have amounted to two or three lines of bash. Gradle isn't any better. Ant at least used to have simple primitives for "doing" things. But you had to spell it out in XML form.

The point of all this, is that build & CI systems should mainly do simple things like building software. They shouldn't have a lot of conditional logic, custom side effects, and wonky things that may or may not happen depending on the alignment of the moon and stars. Debugging that stuff when it fails to work really sucks.

What helps with Yaml is using Yaml generators. I've used a Kotlin one for a while. Basically, you get auto complete, syntactical sanity, type checking and if it compiles it runs. Also makes it a lot easier to discover new parameters, plugin version updates, etc.

> I use GH actions. You should treat it like all build systems: let them do what they are good at and nothing else. The rest should be shell scripts or separate docker containers.

That's supposedly CICD 101. I don't understand why people in this thread seem to be missing this basic fact and instead they vent about irrelevant things like YAML.

You set your pipeline. You provide your own scripts. If a GitHub Action saves you time, you adopt it instead of reinventing the wheel. That's it.

This whole discussion reads like the bike fall meme.

People hates YAML because doing so makes them look cool and trendy. Just like Python-hating. Even if their 'hate' is misdirected.

I'm an experienced SaltStack user. If I found something I need is too complex to be described in YAML, I'll just write a custom module and/or state. Use YAML just to inform Salt what should happen, and shove the logic in the Python files.

People really should become generalists if they handle the plumbing.

If the sole purpose of GitHub Actions is to run a few shell scripts in order, why does it have expression evaluation, conditions, and dozens of stock actions other than `run`?

> If the sole purpose of GitHub Actions is to run a few shell scripts in order, why does it have expression evaluation, conditions, and dozens of stock actions other than `run`?

For you to make that comment, I'm not sure you ever went through any basic intro to GitHub Actions tutorial.

GitHub Actions has 'run'.

https://docs.github.com/en/actions/writing-workflows/workflo...

Now that we established that, GitHub Actions also supports custom actions, which is a way to create, share, and reuse high-level actions. Instead of copy-pasting stuff around, you do the equivalent of importing a third party module.

https://docs.github.com/en/actions/sharing-automations/creat...

Onboarding a custom GitHub Action does not prevent you from using steps.run.

I don't even know where to start regarding your comment on expression evaluation and conditions. Have you used a CICD system before?

The problem with half the comments in this thread railing against CICD in general, YAML, etc is that they clearly do not have a faintest idea about what they are doing, and are instead complaining about ther own inability.

I thin one big benefit of being able to do this is getting more visibility into the pipeline from the GitHub Actions UI/API (including status checks in PR builds). It also helps with reusability. If something is packaged as a GitHub Action, it's much easier to reuse than a shell script written for some other project. GitHub Actions is far from perfect, but I think once you get used to it, it works pretty well.

We see quite a lot of organisations inside because of the business we have, and, while usually this is not our task, when I hear these stories and see people struggle with devops stuff in reality, the first thing we push for is to do anything to dumb it down and remove all the dependencies on 3rd party providers so we are back to having everything run again like, in this case, the hello world of github actions. It is literally always the case that the people who complain have this (very HN, so funny you say that) thing of absolutely grossly overarchitecting and writing things that are just there because they read it on HN/some subreddits/discord. We sometimes walk into struggling teams where we check the commits / setup only to find out they did things like switch package manager/bundler/etc 5x in the past year (this is definitely a HN thing where a new packagemanager for js pops up every 14 minutes). Another terrible thing looking at 10+ year codebases, we see js, ts, py, go, rust and when we ask wtf, they tell us something something performance. Of course the language was never the bottleneck of these (people here would be pretty scared to see how bad database setups are even for multi million$ projects in departmental or even enterprise wide; the DBA's in the basement know but they are not consulted for various reasons), mostly LoB, apps. And the same happens with devops. We only work for large companies, almost never startups, and these issues are usually departmental (because big bad Java/Oracle IT in the basement doesn't allow anything so they have budgets to do their own), but still, it's scary how much money is being burnt on these lame new things that won't survive anyway.

Sounds like you have the same pain points as everyone else; you're just more willing to ignore them.

I am with the author - we can do better than the status quo!

It probably depends on your org size and how specialised you are. Right now I dislike GitHub Actions and think that Gitlab CI is way better, but I also don't give it to much thought because it's a once in a blue moon task for me to mess with them. But I would absolutely hate to be a "100% DevOps guy" for a huge organisation that wants me to specialise in this stuff all the time. I think that by the end of week 1 I'd go mad.

I don't mind it per se; to me the problem is then that some devs don't bother with basic debugging steps of CI failures - if anything works locally and fails in CI, their first step is to message me - so instead of being "100% DevOps" I spend a pile of time debugging other devs' local environments.

My favorite is when they post an absolutely massive error message, most of which is utterly unrelated, but also the answer to their problem is contained within it.

I guess it's possible. But I also don't really have anything to ignore....? I genuinely never have an issue; it builds code, every time.

I commit code, push it, wait 45 seconds, it syncs to AWS, then all my sites periodically ping the S3 bucket for any changes, and download any new items. It's one of the most reliable pieces of my entire stack. It's comically consistent, compared to anything I try building for a mobile app or pushing to a mobile app store.

I look forward to opening my IDE to push code to the Actions for my web app, and I dread the build pipeline for a mobile app.

> I genuinely never have an issue; it builds code, every time.

Well yeah because nobody is saying it isn't reliable. It's the setup stage that is painful. Once you've done it you can just leave it mostly.

I guess if your CI is very simple and always the same you are exposed to these issues less.

> I dread the build pipeline for a mobile app.

I would recommend looking at Fastlane[0] if you haven't already.

[0] https://github.com/fastlane/fastlane

The pain points sound pretty trivial though.

You notice a deprecation warning in the logs, or an email from GitHub and you make a 1 line commit to bump the node version. Easy.

Sure you can make typos that you don’t spot until you’ve pushed and the action doesn’t run, but I quickly learned to stop being lazy and actually think about what I’m writing, and get someone else to do an actual review (not just scroll down and up and give it a LGTM).

My experience is same as the commenter above, it’s relatively set and forget. A few minutes setup work for hours and hours of benefit over years of builds.

The non-solution solution, to simply downplay the issues instead of fixing them. You can solve almost anything this way, but also isn't it nice when things around you aren't universally slightly broken?

I guess I'd disagree that this is "slightly broken". That's just how it works. I don't think there's some universally perfect solution that magically just works all the time and never needs intervention or updating.

> That's just how it works.

It's how it works now. It doesn't have to forever. We can imagine a future in which it works in a better way. One that isn't so annoying.

> I don't think there's some universally perfect solution that magically just works all the time and never needs intervention or updating.

Again you seem to be confused as to what the issue is. Maintenance is not painful. Initial development is.

Unless I'm misunderstanding, you can use workflow_dispatch to avoid having to make useless commits to trigger actions.

I have a small gripe that I think exemplifies a bigger problem. actions/upload-artifact strips executable permissions from binaries (1). The fact they fucked this up in the first place, and six years later haven't fixed it, gives me zero confidence in the team managing their platform. And when I'm picking a CI/CD service, I want reliability and correctness. GH has neither.

When it takes all of a day to self host your own task runner on a laptop in your office and have better uptime, lower cost, better performance, and more correct implementations, you have to ask why anyone chooses GHA. I guess the hello-world is convincing enough for some people.

(1) https://github.com/actions/upload-artifact/issues/38

You must have simple, straightforward flow touched only by a handful of folks max.

The world is full of kafkaesque nightmares of Dev-ops pipeline "designed" and maintained by committees of people.

It's horrible.

That said, for some personal stuff I have Google Cloud Build that has a very VERY simple flow. Fire, forget and It's been good.

You might like 'git commit --allow-empty' to make your somewhat-useless commits.

But honestly, doesn't github now have a button you can press to retrigger actions without a commit?

GitHub Actions are least hassle, when you don't care about how much compute time you are burning through. Either because you are using the free-for-open-source repositories version, or because your company doesn't care about the cost.

If you care about the compute time you are burning, then you can configure them enough to help with that, but it quickly becomes a major hassle.

GitHub actions is nice. People are just not accustomed to being punched in the face. The stuff I work on regularly makes GitHub actions seem like a Hello World app.

I thought the same until having to do slightly more complicated and "off the beaten path" workflows. I'm still amazed at how easy they make building CI jobs now, but I also get frustrated at how it's not a "local first" workflow that you then push to their service.

Yes, your needs are simple. I've also been using GitHub actions for all my needs since Travis shut down and haven't run into any problems.

I wouldn't want to maintain GitHub actions for a large project involving 50 people and 5 languages...

Software engineer thrives on iteration speed. Things have to change, if your pipeline is difficult to change it will cost you.

Reply on news.ycombinator.com