Incident with Webhooks

Maintaining was painful (strongly disliked when an Action would start failing for unknown reasons, forcing prioritisation of the Action over the app and users).

But setting up a useful CI/CD pipeline was the worst part. The tl;dr is installing everything and getting system tests (i.e. tests using a browser) to work was just excruciating, partly because of the minutes long cycle time between making changes in the yaml, committing, pushing, clicking and waiting for it to fail. The cycle time was the killer. (if you could reliably run GHAs locally, my opinion of them would be completely different - and I tried Act, but it had its own problems, mostly related to its image and dependences being a bit different to that used in GHA).

More details (linking to save repeating them): https://news.ycombinator.com/item?id=45530753

For my current work project, caching was actually slower than just doing fresh resource installs... I don't know if Github has some transparent caching proxies for npm, nuget, etc, but they run faster than the cache save/restore does.

We're using the github hosted runners for pull requests and builds... the build process with build and attach .zip files into a release and the deploy process runs on self-hosted runners on the target server(s). Tests for PRs takes about 2-4min depending on how long it takes to queue the job. Build/Bundling takes about 3-5 minutes. The final deploys are under a minute.

The biggest thing for me is to stay as hands off from the deployed servers as possible. Having done a lot of govt and banking work, it's just something I work pretty hard to separate myself from. Automating all the things and staying as hands off as I can. Currently doing direct deploys, but would rather be deploying containers.

It's valuable, but at what cost?

The day I forget to run tests before merging I'll set up CI/CD (hasn't happened before, unlikely, but not impossible).

My build process is gp && gp heroku main. Minor commits straight to main. Major features get a branch. This is manual, simple and loveable. And involves zero all-nighters commit-spamming the .github directory :)

Yeah, as someone old enough to have worked at mid-sized companies before cloud-everything became the norm, self-hosting is overly romanticized. People think you'll get a full infrastructure team dedicated to making sure your self-hosted Git/Artifactory/Jira/Grafana/whatever runs super smoothly and never goes down. In reality it ends up being installed by a dev or IT as sort of a side project, and once the config is hacked together (and of course it's a special pet server and not any kind of repeatable infrastructure setup with Ansible or Docker, so yes you're stuck on Ubuntu 12.04 for a decade) they let it "just run" forever because they don't want to touch it (because making changes is the #1 reason for outages) so you're constantly 2+ years behind the latest version of everything.

It's true that outages are probably less frequent, as a consequence of never making any changes, however when something does break e.g. security forces someone to actually upgrade the 5-years-since-end-of-support Ubuntu version and it breaks, it may take several days or weeks to fix because nobody actually knows anything about the configuration because it was last touched 10 years ago by 1 guy who has long left the company.

That’s already what Gitlab and gitea is doing

Run your own gitlab server then?

An other trick is to go into the "done" tab, and move at least 25 issues back to unread.

Then you can click the checkbox at the top and then "select all", and it'll mark the phantom notifications as read.

Yes private repos that are deleted leave notifications behind.

GitHub has been experiencing mass waves of crypto scam bots opening repos and mass tagging tens of thousands of users on new issues. Using the issue content body to generate massive scam marketing like content bodies.

Interesting that they went with a custom MIME type and a custom version header. I would have expected the version to be in the MIME type, but I feel like there is a reason behind this.

Thanks for this! I had been meaning to mark a phantom notification as read — and this snippet came in handy.

Same. Fraser Valley?

Here's the step-by-step guide to self-hosting git repositories:

Change directory to your local git repository that you want to share with friends and colleagues and do a bare clone `git clone --bare . /tmp/repo.git`. You just created a copy of the .git folder without all the checked out files.

Upload /tmp/repo.git to your linux server over ssh. Don't have one? Just order a tiny cloud server from Hetzner. You can place your git repository anywhere, but the best way is to put it in a separate folder, e.g. /var/git. The command would look like `scp -r /tmp/repo.git me@server:/var/git/`.

To share the repository with others, create a group, e.g. `groupadd --users me git` You will be able to add more users to the group with groupmod.

Your git repository is now writable only by me. To make it writable by the git group, you have to change the group on all files in the repository to git with `chgrp -R git /var/repo.git` and enable the group write bit on them with `chmod -R g+w /var/repo.git`.

This fixes the shared access for existing files. For new files, we have to make sure the group write bit is always on by changing UMASK from 022 to 002 in /etc/login.defs.

There is one more trick. For now on, all new files and folders in /var/git will be created with the user's primary group. We could change users to have git as the primary group.

But we can also force all new files and folders to be created with the parent folder's group and not user primary group. For that, set the group sticky bit on all folders in /var/git with `find /var/git -type d -exec chmod g+s \{\} +`

You are done.

Want to host your git repository online? Install caddy and point to /var/git with something like

    example.com {
    root * /var/git
    file_server
    }

Your git repository will be instantly accessible via https://example.com/repo.git.

I thought they were already on Azure?

At least I'm pretty sure the runners are, our account rep keeps trying to get us to use their GPU runners but they don't have a good GPU model selection and it seems to match what azure offers.

Well, of course the corporation wants to dogfood their platform.

Expecting more and more downtime and random issues in the future.

Its better for github to self-host.

Github hosts git repos, so it's an example of when self-hosting git repos on one's own servers could remain operational despite a github outage.

GitHub Pages are affected too

I self host a server for a website and I also compile executables in there and it's been running just fine for 2 years and it's not even like a big provider, a very niche one actually (mac servers)