GitHub Broken Download URLs
github.com
github.com
Sorry for being curious.. does Github have an office in Copenhagen? That would be cool.
If you're working remotely, are you employed by Californian standards/regulations/benefits or by local ones here in Denmark? If you don't mind me asking.
This was deployed Thanksgiving week? I realize Github isn't a consumer company, so it doesn't face the same pressures as Amazon, but I'm surprised there wasn't a code freeze so people can have a quiet holiday weekend.
Odd for something like this to slip through and not be rolled back immediately.
Unless it was intentional, in which case it would be even more odd to not communicate this widely beforehand.
With no announcements and no response to a now two day old bug report, I see two possibilities:
1)Their monitoring of their infrastructure and monitoring of issues is shockingly incompetent for a company of their size and importance (the fact that it is a US holiday is irrelevant.)
2)This was 100% intentional and they're purposefully looking "incompetent" to get people to shift to using other services for downloads.
My money is on the latter, given others in this discussion are reporting random download link failures starting a month or two ago. A huge number of projects seem to use GitHub as a sort of free file hosting service. I imagine the opex for both storage and bandwidth is a not insignificant amount of money and someone has been told to shoo the freeloaders off the grass.
Announcing they're ending free file hosting for unpaid projects would generate a lot of noise and PR. Instead they just make it unreliable, and people go elsewhere. Multiple people in this discussion have described moving downloads of Github in response, which is exactly what Github likely wants.
A change in the handling of URL schemes was deployed a couple of days ago that caused the regression being discussed here. Due to the amount of traffic that the archive endpoints see, and the high baseline of 404s on them, this regression did not cause an unusual increase of errors that would've caused our alerting to kick in. The change has just been rolled back, so the issue is fixed. We will investigate this issue further after the weekend and take the appropriate steps to make sure similar regressions don't happen in the future.
[0] https://github.com/github/feedback/discussions/8149#discussi...
> A change in the handling of URL schemes was deployed a couple of days ago that caused the regression being discussed here. Due to the amount of traffic that the archive endpoints see, and the high baseline of 404s on them, this regression did not cause an unusual increase of errors that would've caused our alerting to kick in. The change has just been rolled back, so the issue is fixed. We will investigate this issue further after the weekend and take the appropriate steps to make sure similar regressions don't happen in the future.
"Is that a service we can charge for?!"
But I don't understand how github didn't already know about the regression, from automated testing or error monitoring. I have high expectations for github, because they have met them.
They have error monitoring, but download URLs are one thing that's tricky to monitor for "errors" correctly because if two URLs 404, how do you know from your Grafana dashboard or whatever which are valid and which aren't? Having run a server whose only purpose is to serve static files myself -- you get a lot of 404s, all the time, it's no indication anything is wrong at all. This case would only be picked up by a dashboard if it fatalistically caused an error somewhere (like a 500) but by definition this wasn't ever a 500, it's a 404.
As you note, it's just a bug. Sometimes things you don't understand might actually surprise you, it turns out.
This was reported like it affected all download URLs based on a git tag, which also means all download URLs appearing on github "release" pages.
If so, I'd have expected there would have been some testing that would have caught this too. Of course, sure, bugs in tests happen too.
Obviously bugs happen because bugs happen. I stand by being more disturbed that it took two days to notice and revert to restore the regression than I am by the fact that a regression happened. Users noticed right away and tried to report the problem, it took two days after that for Github to comment on it and to revert, which seems problematic, no? Especially if the bug was really affecting as many download URLs as I think it was reported; if it was only affecting a minority of edge case ones, that's more understandable.
It's never possible to eliminate regressions. (It may be possible to reduce the rate of them of course). But whether by testing or by receiving error reports from users, it ought to be possible to notice all major regressions in less than two days.
The fewer external dependencies you depend on at the actual point of builds/deploys, the better. Since you're using a fixed version of the dependency anyway, you might as well self-host or include it in the ami or container.
Is there a list of pre-defined URLs supported by GitHub?
:(
If a project is using github to publish releases, where else are consumers of that software going to get them from?
Having all sources of everything that is packaged backed up is a must for the official repository of a competent distro, but even in that case there is no reason not to use github in normal operation.
It feels like people are using git wrong.
git clone --branch v1.0 --depth 1 https://github.com/example/example.git
It's not as fast as fetching a ZIP file, but it gets pretty close.
From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip.
The zip download method (download, decompress, build, compress into package, decompress onto system) is already silly enough, the first decompression part can easily be dropped.
In the context of distro packages (the bug report mentioned OpenBSD and Fedora) you might be building tens of thousands of packages of which thousands are likely to come from GitHub. A small difference becomes greatly magnified.
> From my count, this method only requires one dependency (git) whereas the curl + unzip method requires, well, both curl and unzip.
You’re forgetting that git itself depends on curl.
Heck, you could check if the tag you want already exists in the repo you have and skip the `git pull` a lot of the time.
I'm surprised something like this happened on a weekend though since I wouldn't expect anyone to be changing anything in the codebase (then again it could be some infra has just ran out of storage or etc)
Good Luck rolling back GitHub.
Activity Pub integration issue on Gitlab: https://gitlab.com/gitlab-org/gitlab/-/issues/21582
Discoverability will probably need more work though
I think rubygems has a single centralized download point.
I am not very familiar with other platforms, including OS distro packages. Do some of them use decentralized systems?
It's definitely more work than just throwing it onto GitHub though, something more convenient would be really cool.
My username ends in a hyphen. Apparently, that's no longer allowed, though my username appears to be grandfathered in.
Trying to give feedback about new experimental features lands me on the GitHub communities site, which is treated as a standalone app and thus requires you to log in via GitHub (it doesn't re-use the existing session token).
However, Communities won't let me sign up with my username since it has a hanging hyphen, and I can't change the username in the form. So I effectively can't sign up.
Support has not responded for over a month. Feels like things are inching toward getting worse with GitHub.
"I purposefully chose to create a nearly identical username to an existing user" isn't a great defense to someone saying the problem is between the chair and keyboard.
At best you didn't think of the possible confusion you'd cause.
So sorry, but no. Things have been fine. In places where the hyphen is ambiguous, I always make a point to mention it. It's never really been an issue, and the other individual has always been quite nice when the hyphen is omitted, kindly pinging me instead.
I'm not trying to defend against anything except poor attempts to paint me as somehow malicious...
Is that even surprising? Some names are just so common that people use them all the time and ends up as John, John_ , John-, John1, John11. John100
I don't really see what problem using a hyphen in a username could pose, unless there's some kind of filter being applied that doesn't take into account the previously permitted characters. I'd guess someone applied an [A-z0-9]+ without thinking too much about it because that's what the current username rules are.
I'm more surprised that there's a second authorization endpoint, Github could've just used their existing OAuth2 implementation to log users in if they didn't want to reuse the existing login code.
`[A-z]` -> https://regex101.com/r/IlhPiD/1 `[A-Za-z]` -> https://regex101.com/r/iWjwf2/1
After looking it up, `\p{L}` looks like it matches letters https://regex101.com/r/1UiG9S/1.
Please don’t shop unrelated concerns to threads that aren’t about that concern. GitHub breaking release URLs worldwide after a multi-hour global outage has no relationship whatsoever to your problem. I sympathize with your frustration, but pet peeve derails pollute HN discussions about every topic under the sun these days. Submit a post instead, and if it doesn’t get traction, so be it.
This isn't a pet peeve derailment. This is commentary on the multitude of issues GitHub has had in recent history, at least from my perspective.
A "pet peeve" is something I find annoying. This isn't that - it's a bug.
Further, my comment doesn't break the guidelines. I'm not commenting on the layout. I'm not flagrantly dismissing someone's work. If you don't like my comment, either keep scrolling or downvote and move on?
If you had violated a guideline, I would have contacted the mods rather than reply. The tragedy is that no rule can sufficiently be written that keeps people from concern shopping their personal issues into discussions on the most tenuous of links. “This is a post about a GitHub bug” opens the door for me to post about the hundreds of GitHub bugs I’ve encountered over time, and with thousands of users at HN, if we each do this, there’s so much less room left for discussion about the actual bug this post is about.
This affects Show HN, too: when someone posts their cool thing, everyone chimes in with all the other cool things that they like better. It’s incredibly disheartening and sets aside the purpose of the post – “a thing, discuss” – so that people can use that request as a launchpad to discuss other things instead, without making even the slightest effort to tie it back with relative comparisons to the Show HN topic itself.
There is no guideline that asks us to set aside our personal needs and desires in these comments and focus on what brings the most value to the original topic, no matter what we feel about other topics that happen to also be about GitHub. But I continue to hope, out loud and with salient arguments, that HN will step up and respect itself more than the guidelines require.
I imagine if support got back to you, they'd say "Yes, that was grandfathered in for backwards compatibilty, but that username is no longer allowed and won't always work with new services or functions, as you discovered. We recommend changing your username."
However, not getting a response is not encouraging, it's true.
the Executive Privilege for the common committer ! just keep your keys "valid" and don't talk to anyone we ban
up next -- Github social scores
In any case, the point was that Github.com just sucks, rather than everything cloud sucks. For the past year, they have been down a couple of magnitudes more time than I have spent managing my server.
I have spent orders of magnitude less time feeding my pet rock than the average dog owner spends feeding their pet.
The difficulty of keeping a service online depends on what it actually does. Not to mention, outages are generally caused by making changes. Changes which are required if a service is going to continuously improve
If you are again trying to make the point that they have a bazillion times the load, I will say again that it is self-inflicted, and thus part of the problem, and not a valid excuse nor justification. Other providers do not seem to have these problems anyway.
Comparing uptime between a production service and a single personal server however isn’t reasonable. You haven’t even said what your server actually does, only that it rarely goes down!
The complexity and load of github is the problem.
If every Arch package hosted its own source code, then even if each one of them has better uptime than GitHub, at least one of them is probably going to go down every single day (assuming their uptime is uncorrelated).
If you can personally centralize all of your own work on a server that you host, like a company-run Mattermost or something, and you have someone who can be on call to keep it up, do it. The result will, as you’ve described, be simpler and better than GitHub. But expecting every individual to run their own Git server is just stupid, because then my project is going to wind up depending on dozens of separately-hosted Git servers and I’m back to taking the intersection of their uptimes to compute the uptime of the whole system. There’s a sweet-spot here; GitHub is above the sweet spot, but a lot of “FOSS projects” (the kind that have a single maintainer) are below it.
Perhaps githubs value is elsewhere.
Especially for an organization like Arch Linux, the solution is really obvious; maintain your own mirror. Debian and Fedora do it, so can they.
But I agree: mirrors is the key!
That's what I'm putting into question, specifically for Github.