Automating a Software Company with GitHub Actions | Hacker News Reader

Automating a Software Company with GitHub Actions

posthog.com

247 pointsmariusandra70 comments

I just want to be able to write all my workflow code as typescript (including the config - no YAML, for the love of god, no more YAML!) and run it locally with a debugger attached.

It's cost me hundreds to thousands of dollars to implement nontrivial workflows because of how the YAML is parsed (for example, empty strings when using a secret that has been renamed or removed) and the lack of introspection or debuggability when something goes wrong.

It's gotten to the point where new any new workflows I write are thin wrappers around a single script and I don't import any actions besides actions/checkout (even that has been bug prone, historically).

All that said, it's not like other platforms are better. But they certainly are cheaper and don't have dumb breakages when you need cross platform builds (has upload-artifact been fixed for executables on MacOS yet?)

Yea, making it so statements are composing via YAML was a very poor choice. One case I had was needing to change my actions so that steps D, E, and F could be retried together. What is the github YAML specification for decrementing a 'tries_remaining' variable, then going back to a previous step? I couldn't find it, so ended up just having to rewrite the action so that most of the steps are now in one bash script.

Not being able to execute it locally has also wasted a lot of time, and people needing to make 50+ changes to the master branch until they get it right.

This might be an unpopular opinion but bundling CI/CD logic with the application code has been a huge blunder IME. If you're adding CI/CD for the first time to a brownfield project you're looking at dozens of minor YAML tweaks to master directly (or a bunch of rubber-stamped PRs).

Why can't I have a separate interface where I just say "build this Github project, and put the content on this on-prem server/kube cluster/VM/whatever."

If I'm doing something complex with GitHub Actions I like to work in a branch, iterate with a bunch of messy tweak commits and then squash-merge into main once I've got it working exactly right.

Another trick that works well is putting GitHub Actions in an entirely separate repository. There's nothing to stop actions in one repo from checking out code from another - I use that trick quite frequently.

You do have to jump through a few extra hoops to set it up so that code in your actions repo starts running automatically on commits to your main repo, but you can do that with a small action in the main repo that triggers a build in the actions repo.

Interesting approach. Do you mind sharing link with repos?

I don't have any at the moment where a commit to one triggers a build in another, but I have a bunch that run on a schedule, pull code from other repos and do stuff with it.

Best example of that is here: https://github.com/simonw/covid-19-datasette/blob/main/.gith...

These are both great tips, thank you!

Who CICD's the CICD? It's turtles all the way up.

Testing a pipeline that depends on a merge to a branch, or a specific tag, is troublesome. Easier to just iterate in the mainline until you're ready.

Having dealt with a fair bit of CI on a decent number of services (AppVeyor, Travis, CircleCI and GitHub Actions), I’ve come to the exact same conclusion.

CI is a script, and the YAML configs for those various services configure the machine type, OS and toolchain. Everything else is contained within the script. Sometimes even toolchain setup is handled by the script.

Not following this model has wasted so much time when migrating services or trying to tweak what CI does.

With a script you can run it locally to ensure it performs the steps desired, leaving the CI “setup” to minimal environment/toolchain debugging.

Do you have a picture of what typescript code for workflow would look like? Are you thinking of something that calls APIs directly or runs and spits out some kind of JSON instructions in place of yaml?

It would call APIs directly. For example, I have a lot of release automation that integrates with AWS. Rather than requiring AWS specific actions, I could import the AWS sdk via node and make the calls myself.

A good exercise is to look at a simple but nontrivial build/test/release automation:

- on pull requests build & test, on tags build/test and release, and once a day release a nightly build.

- cache dependencies globally and build artifacts on each branch for incremental builds

- once daily, clear out artifacts on deleted branches or merged PRs

A script to do this would be around 100 lines and be readable/maintainable (if build/test or release change, it is shared by the different cases!). The script can have a main entrypoint that will dispatch to the runners as needed.

I know it sounds a lot like jenkins, but node has a much better ecosystem than groovy.

Yep. I am currently needing to try and set up code coverage reporting in GitHub (can't use Codecov as the company does not want to pay for anything) and it would be so much more trivial if we could just write regular code rather than futzing around with Bash and yaml.

You can write custom actions in JavaScript, using the toolkit: https://docs.github.com/en/actions/creating-actions/creating...

However, the workflow referencing the custom action would still be the Actions YAML syntax.

(disclaimer: I work at GH, but not on Actions)

Disclaimer? This is the good part of HN, being told of fun bits like this.

Thank you.

It's still polite to disclose as a habit.

Hi Matt! Sabiha from Codecov here - curious how many seats you're looking to add as Codecov is free for up to 5 users. From there we are currently priced at $10/seat. We've ensured that our pricing is viable for all kinds of businesses and that it outweighs the costs of building this in house :). If it makes sense, feel free to reach out to sabiha@codecov.io if I can help.

Did anybody experiment with cuelang as source for GitHub actions yaml?

When you edit an action on GitHub, the web editor has support for workflow syntax and will determine if your action is viable before committing. This is even easier if you use the new codespace shortcut (press .) on GitHub.

You might also want to take a look at act which allows you to run github actions locally, this is typically how I do actions development:

https://github.com/nektos/act

I'm curious what prevents you from writing your own actions in typescript now?

> the web editor has support for workflow syntax and will determine if your action is viable before committing.

I don't use the web editor, but more importantly it can't catch logical errors (missing required with: arguments, secrets that don't exist, environment variable names, etc).

> I'm curious what prevents you from writing your own actions in typescript now?

When I say "I want to write actions in typescript" I mean that I want to specify the entirety of my CI using a typescript program, without any YAML configuration. In particular, the jobs of a workflow themselves.

I have many jobs shared between build/test/release with slightly different triggers and configurations, but the only way to handle this in actions (especially when using imported actions) is by copying/pasting YAML. That wound up being untenable, and it's why I stripped out all action dependencies and wrote the automation to not use them such that all workflows

I've also had use cases for recursive workflows.

Act doesn't cover any of my use cases.

Have you considered writing a code generator that lets you define actions in TypeScript and have it generate the YAML for you?

This is absurd

We built this for my company. It's called YATM. Yet Another Thing to Maintain. ;)

Jenkins have Jenkinsfile in Groovy, but it's still kinda hard to debug it. Never tried

You could use TypeScript IaC like AWS CDK or Pulumi with FaaS like AWS Lambda or Cloudflare Workers.

End to end TypeScript from IaC to FaaS.

This is exactly what I'm working on at https://www.inngest.com. You can write statically typed workflows, run any code as an action (if it runs in a container, or use lambda otherwise), have complex conditionals, and coordinate between events (eg. pause until an event comes in that matches a condition, with TTL paths). There's a debugger to step-over workflows, and see the input/output as it happens. Coming soon, the ability to run actions locally during debug mode.

Workflows are statically typed in that they have event triggers - ideally with an event schema (which we generate for you) - and each "action" has static typing of inputs/outputs (plus additional workflow configuration for reusability). They're defined as code, and can be viewed or edited visually.

It also bundles an event hub, so you can automatically run workflows when events happen in real time. For example, if you want to run a churn flow on signup, create a workflow with a `signup.new` event trigger. The workflows can also coordinate between events, too, so in the churn workflow you can wait for an "interactivity" event from the user for up to 1 day, then time out and run some other flow/logic.

It's workflows, generalised. As if you put Github Workflows, Lambda, Segment, and Zapier in a blender.

If you want early access, you can always reach me at tony [at] inngest.com. I'm rolling out invites every week.

Do you have clients using this for CI/CD? It looks very interesting, if a bit more general-purpose.

CD (including ourselves), operational pipelines, sales pipelines, internal events/logic, scheduled flows, and machine learning are a few of the areas I'm seeing.

CI isn't _necessarily_ our current target. As of yet, there's no concept of "failed runs" in the classic CI sense. We highlight failed workflows but will retry actions by default. If things fail, we allow you to retry/edit the data, debug in line, etc.

It's definitely possible to use the failed workflow runs as a CI pipeline, but the UX would need an overhaul. Right now, we do user-event attribution to show you which workflows users are running in your system, their version, and the steps - which is needed for actual, live system workflows vs CI workflows.

You should post a Show HN when it's open for people try out. See https://news.ycombinator.com/showhn.html and https://news.ycombinator.com/item?id=22336638. Email me at hn@ycombinator.com with a draft text and I'll look it over for you beforehand. Same offer goes for anyone, just please don't expect a quick reply- that depends on the wild swings of the HN inbox. You can also see how we advise YC startups to do this at https://news.ycombinator.com/yli.html - the logistical parts are YC-only but the communication parts apply to everybody.

Ah, thank you Dang, I'll reach out when it's ready!! Very much appreciate you letting me know. It'll likely be several weeks before then - I'm super product focused and want to make sure the UX is down before a general release.

Use Teamcity. There you can use Kotlin.

I totally agree, and I’ve seen some completely nuts GH actions setups already where people seem to want to use every single possible action. What happens when the charging model no longer suits you? I’ve done self hosted Jenkins -> TravisCI -> CircleCI -> Self hosted GitLab on one of my projects in the past few years… I want to do as little CI provider specific stuff as possible.

For CI specifically, you should IMO check out https://dagger.io. It abstracts and generalizes CI in exactly the way you need: repeatable, locally runnable, and also cacheable on remote servers. Write once, run anywhere CI - what you have locally will be _exactly_ what runs remotely.

I've seen both sides of this. Heavily scripted Jenkins groovy files and weird YAML Gitlab pipe lines who admittedly are hard to deal with and understand especially for someone new.

IME I prefer the YAML files because it forces the "DevOps" guy to put most of the stuff in outside scripts, which can than be tested and run on a local dev machine.

Nothing worse than having to debug a freaking CI runner. I never got why a job should be more complicated than "fetch -> build -> test -> deploy" with maybe a bit of intelligence to handle build triggers and artifact management.

I don't want to deal with a web of groovy scripts with sed hacks everywhere and which involves 10 other jobs who does the same kind of unholy blasphemous hacks and who always break one another.

The Visual Studio team totally needs to steal your ideas and deliver implementations for sharing your workflow.

A graphical development environment and TS library set for something like UML wanted to be for OOP only for everything we need to do and want to do with our codes in any development environment from the enterprise VS group wallet grab setup we still run, to a "redmode" for working from Emacs.

It's not all as bad as XKCD says about creating new standards : the world of code writers is big enough now for many equivalent standards to thrive in coexistence. We're probably still too emotionally scarred from the very recent years when anything that smacks of making standards and arguing betweent them had much more serious consequences than possible today.

There is a serious market niche for that. There is SharePoint/Flow for Office, but it is overall dreadful software.

I think platforms like nextcloud try to build some workflow engines, but I don't know how far that is and in which format they are implemented. I don't like Zapier, but many people seem to like it.

One of the things I like about Actions is how much it's focused on automation rather than CI. The pain points I've had with Circle/GitLab/Travis have often boiled down to the fact that they are often very specifically about _testing software_, not _automating processes_, and not even _deploying software_.

On that last one, there's a potential bug in the deployment pipeline here – deploys could run simultaneously or some bad luck on runner speed could even see an older version of the code go out after a newer version. Combined with the automated database migrations this could be quite a big problem!

Actions thankfully solved this recently with the `concurrency` key that lets you form a serial queue by a given key such as the branch name.

I'm not sure the concurrency key can solve the issue of old code after new code, in the general case.

What happens if there are conflicting migrations on two "parallel" branches?

What happens in you bad luck situation when commits are pushed in rapid succession on the same branch?

With the concurrency key at the top level, actions run it commit order.

As for running migrations on the same database from multiple branches, not a good idea. Probably best to decide on the branch that is the release branch (maybe main/master) and only deploy from that.

In GitLab you can limit concurrency by using a resource group https://about.gitlab.com/blog/2020/01/21/introducing-resourc...

Does that address your need?

Edit with more context: We use CI for deploying to GitLab.com and use resource_group to prevent multiple jobs from running concurrently. What we lack is the ability to prevent multiple pipelines from running concurrently (resource_group is at the job level). It looks like concurrency for actions https://docs.github.com/en/actions/reference/workflow-syntax... can work on groups which is a bit nicer. There is some discussion about making this better for GitLab in https://gitlab.com/gitlab-org/gitlab/-/issues/217522

While doing it at the job level is handy, doing it at the workflow or cross-job level is likely to become necessary as the complexity of the workflow scales. CircleCI has similar issues, syncing on one job can be done with a plugin, but cross-job sync never really works, and in our 10 step workflow where we have deploys queueing up in quick succession, we often find ordering issues.

GitHub Action's, Semaphore's, or even Jenkin's, concurrency primitives are pretty capable so I'd probably go with one of those.

I've never felt like GitLab CI is more about testing software than anything else...

It's super flexible and you can do literally anything.

Yeah, I concur with that, and what's more I find the documentation so much better - Actions is maybe just as flexible (and if it isn't frankly that might be an improvement!) but the docs are extremely lacking. I do prefer GH for just general look/feel/ease of use though.

When I last used it (Aug 2020) seemed like they were trying to evolve out of a stricter build/test/release flow, like Travis, and into a looser/more general workflow system, like CircleCI. At that point, it felt like the workflows were a bit half-baked – getting caching right between workflow steps so that you could build a performant pipeline that didn't repeat unnecessary work was tricky if I remember correctly.

Apart from feeling like they were in that sort of transition, GitLab's docs were fine, better than Circle's, but I find GitHub Actions much clearer.

I love GH actions, but they're still a bit too sharp-edged for my taste.

Like...the last time I checked, workflows had no runtime macro for limiting execution to the default branch except explicitly by a specific name, and the closest you could get to generically checking "whatever the default branch is called right now" was either a template workflow that would set some static text for the name at creation that breaks if the default branch name is subsequently changed or a song and dance querying the API and setting an environment variable inside one of the workflow steps and then gating all subsequent steps on the result. This was a long time after they introduced editable default branch names and seems like such an obvious oversight.

Then there are weird quirks like the subshell file system permissions block that requires using sudo if you want to move files around within your repo clone from inside an invoked shell script.

One relatively simple thing I'd love is the ability to re-run a job where it fails, which is something other CI systems have. On a current project we have some flaky Cypress tests and it sucks to have to re-run the entire workflow from the start when one of the last steps of the job fails. I'm definitely not the only one wanting this [1].

They also offer triggering workflows with 'workflow_run' based on other workflows, but that only happens on the default main branch. We auto build testing environments on each PR and I'd love to be able to have better workflow management based on branches.

[1] https://github.com/actions/runner/issues/432

Looks like someone has built a reusable action for retries: https://github.com/marketplace/actions/retry-step

At my company, the biggest pain points we've run into with GitHub Actions are all centered around the many lacking aspects of permission handling.

- You can't pull in private dependencies published from other repos (for example, packages published on repo A used as a dependency on repo B) without using a private access token.

- You can't use git pulls from other repos (for example, repo B using `orgname/repoA#123456` as a dependency in package.json) without using a private access token, and it's a pain in the ass to make it work across workflow steps.

- You can't allow Dependabot to run as a trusted user, which makes it impossible to actually use any of the workarounds for the above issues with it.

- You can't create PRs to publish changes across repos (such as automatically keeping some set of files in sync) without using a private access token.

There are other complications, but those are the biggest ones.

The dependabot issue is insane, and the whole way this issue arose (it used to work before this limitation was introduced) indicates that the security team and the dependabot team at GitHub just didn't talk to each other.

(There's a workaround for the dependabot issue though, use pull_request_target instead and explicitly check out the sha of the branch. Then the run can access the secrets.)

I would also add "you can't rerun single jobs" and "actions can't call other actions" to the list of grievances.

GitHub Actions are such a great tool, I use them to schedule tweets for me.

Interesting, do you have a link to an example action for this?

This should get you started :D let me know if you need help setting it up https://github.com/reidjs/markdown-tweet-scheduler

Thanks for that, that looks great.

I've been looking for a way to post regularly without actually having to do it. tweetdeck is nice, but its still more hassle than filling a folder with markdowns.

me too! and Facebook posts, too.

Can anyone point me to an example of a Github action resulting in standing up a fully working backend with a resolvable DNS entry for manual pull request testing?

I can't point you to an existing one. However I'm confident I could put this together in a day.

Do you think it would be difficult? Github actions can be boiled down to running shell commands (with a bunch of other handy features), so it's quite versatile. At times it does require you to make workflows which are a bit convoluted but all in all I think it's pretty good.

As long as automation solutions (such as all ci-tooling) uses yaml to define logic instead of a proper programming language (or at least configuration language like dhall), I will try to stay away from them as much as possible.

I'm always hesitant to lock into solutions like these because they may not always be offered on the same terms.

For example, recently when they had to reduce access to GitHub Actions because miners were abusing it.

Where as my local configuration management (Ansible, Puppet etc.) script can always run anywhere and I can even run on my own build VM if I need too.

I did a triple-take "...at PostHog we've been avid users of GitHub since its early ARPANET days."

Yeah I don't get that, is it a joke? Pun? Some meaning that isn't how I'm (or your first two takes) reading it?

Even git's early days were well after the last ARPANET days.

Edit: I suppose they just mean 'since time immemorial', but in internets. Perhaps I'm just tired/'whooshed' but I think that was more confusing than they were going for!

Hey, I'm the author of the article, I sneakily put that bit in just to see who notices the absurd of it. ;) But sorry for the resulting triple-takes!

> I sneakily put that bit in just to see who notices the absurd of it. ;)

This give me ideas :-)

I noticed their hadolint action and couldn't help but think it falls a bit short in terms of flexibility and output. I wrote an action to improve these type of use cases that can be found here: https://github.com/jbergstroem/hadolint-gh-action

GitHub actions is really making me want to move our company back there from GitLab.

Does GitLab have a response planned?

That's cool :)

LOL a software company is more, so much more, than CI. I thought I was going to read something novel about using GitHub actions for tracking sales leads or customer success or something.

This does seem to be a pretty bog-standard usage of a CI/CD tool. I must be missing something.

Seems like the article and also comments here imply just that.

GitHub Actions aren't focused on CI, so they are much more useful.

> GitHub Actions aren't focused on CI, so they are much more useful.

Compared to what? ie.. Jenkins, CircleCI, Gitlab's CI/CD...none of them are "focused on CI", and can more or less do anything you want them to. I'm not trying to be argumentative, but I'm having trouble thinking of a CI/CD system that is more(or less) focused on CI than Github actions is - do you have some examples?

I'm not familiar with GH, but gitlab is definitely focused on building/running things from a commit in a repo to the point of making it awkward to do ad hoc actions or tasks not clearly tied to one repo. You can do it, but it's harder and has weird restrictions.

We started using Pipeline schedules in my past job to regularly trigger pipelines to e.g. rebuild Docker build images, clear caches and other sorts where you normally need shell access to a cronjob.

https://docs.gitlab.com/ee/ci/pipelines/schedules.html

Similarly, you can trigger pipelines from various angles https://docs.gitlab.com/ee/ci/triggers/ using the API.

If you are looking to combine it with events on-demand, the webhooks may come in handy. https://docs.gitlab.com/ee/user/project/integrations/webhook...

Agreed, some adhoc actions are project specific, though you can programmatically walk through them in API client code, for example searching for a group and triggering all project's pipelines.

https://python-gitlab.readthedocs.io/en/stable/gl_objects/gr... https://python-gitlab.readthedocs.io/en/stable/gl_objects/pi...

Reply on news.ycombinator.com