GitHub Major Service Outage
See https://status.github.com
> Monitoring
> May 31, 2017 3:08 PM
> GitHub have declared the outage resolved and we are starting to see incoming GitHub hooks. Builds are being triggered again. However we are still seeing failures with the GitHub API. This continues to prevent our webapp > from fetching data from GitHub. We are monitoring the situation and will ensure sufficient capacity for when their service resumes normal operations.
Doesn't seem to break anything, but it is a bit curious. May not be new though...I just happened to notice it today.
Fairly compact implementation too: https://gist.github.com/anonymous/33710bd9c7175a645dd0d72d1a...
I can only chalk it up to something like clock drift between the processing node and the database server.
Irritatingly I can't remember which site it was but I posted something somewhere a couple days ago and immediately after hitting enter the site marked what I'd said as submitted "a few seconds from now". I never fail to be amused that the fuzzy time library being used has code specifically designed to handle this edge case scenario. :D
Sure, if you use it to show comment age, you shouldn't ever see it, but I'm sure they fully support using it for countdowns, too.
EDIT: it's the 4th example under relative time for Moment.js (https://momentjs.com/).
So as not to spam with my reply to a similar comment, I'll link it: https://news.ycombinator.com/item?id=14452335
So from a UX we should simply discount time drift +-5 seconds simply says "blah blah just now" and anything larger might warrant as a warning to the author (Github can reject the commit and ask for confirmation). Although re-editing commit is a hassle for changing timestamp. It's more of a "please make sure you computer time is synced next time."
I don't quite remember but I think I might've been commenting on something on GitHub when I saw the time glitch.
Initially for a moment I thought "why not just have OCD local NTP tracking?" but then I realized that time glitching around (even at the millisecond level) can be disastrous. One way to solve this is to obsess about keeping up to date with NTP, but instead of updating the time, update a global reference to the offset. Then your time server is simply (system date)+-(saved offset), which should be super fast. And of course this can run on the node generating the HTML.
While parent object ids are provided on the command line, author and committer information is taken from the following environment variables, if set:
GIT_AUTHOR_NAME
GIT_AUTHOR_EMAIL
GIT_AUTHOR_DATE
GIT_COMMITTER_NAME
GIT_COMMITTER_EMAIL
GIT_COMMITTER_DATE
(nb "<", ">" and "\n"s are stripped)
In case (some of) these environment variables are not set, the information is taken from the configuration items user.name and user.email, or, if not present, the environment variable EMAIL, or, if that is
not set, system user name and the hostname used for outgoing mail (taken from /etc/mailname and falling back to the fully qualified hostname when that file does not exist).
A commit comment is read from stdin. If a changelog entry is not provided via "<" redirection, git commit-tree will just wait for one to be entered and terminated with ^D.
DATE FORMATS
The GIT_AUTHOR_DATE, GIT_COMMITTER_DATE environment variables support the following date formats:
Git internal format
It is <unix timestamp> <time zone offset>, where <unix timestamp> is the number of seconds since the UNIX epoch. <time zone offset> is a positive or negative offset from UTC. For example CET (which is
2 hours ahead UTC) is +0200.
RFC 2822
The standard email format as described by RFC 2822, for example Thu, 07 Apr 2005 22:13:13 +0200.
ISO 8601
Time and date specified by the ISO 8601 standard, for example 2005-04-07T22:13:13. The parser accepts a space instead of the T character as well.
Note
In addition, the date part is accepted in the following formats: YYYY.MM.DD, MM/DD/YYYY and DD.MM.YYYY.
* https://github.com/yeukhon/demos/commits/master/git-date
* https://github.com/yeukhon/demos/blame/a9fc9dfe6d35c5ffe14af...
* https://github.com/yeukhon/demos/commits/a9fc9dfe6d35c5ffe14...
You see I got a 3 minute (using local time) and then 4 hours ago because I set the timestamp manually in the most recent commits. So yes you can set time in the future / past.
Http response status codes rarely align with RFC guidelines.
I just think there should be an intent argument specifiable to the library to indicate that the duration in question refers to an event that has happened in the past.
In such a scenario, the library should mark the duration as happening "just now" and possibly flag a warning or raise an exception.
The reason I say this is that, humanly speaking, "Your message was sent 23 seconds from now" is amusing at best to developers who know what's happening (negative time delta) and linguistically confusing to general users ("was sent" vs "from now").
But frankly, I'd rather they have better uptime. Every couple months is too much. I pay them. My work pays them.
If their CEO is serious about zero downtime, how about he offers his paying customers a credit for time they cannot access the service?
Hmm. You pay them to uphold a contract. What does that contract say about SLAs and availability? Probably the same as the TOS that I agreed to when paying and those specifically say:
GitHub does not warrant that the Service will meet your requirements;
that the Service will be uninterrupted, timely, secure, or error-free;
that the information provided through the Service is accurate, reliable or
correct; that any defects or errors will be corrected; that the Service will
be available at any particular time or location; or that the Service is free
of viruses or other harmful components. You assume full responsibility and
risk of loss resulting from your downloading and/or use of files,
information, content or other material obtained from the Service.
I understand, it sucks when github is down. But we all get what we pay for and we all don't want to pay for more. And yes, I do have clients that meticulously mirror all their dependencies from outside sources and spend significant money on this - money that pays off in exactly these situations.
An upgrade from Team to Business is "only" a 2.3x price bump per dev. I have no experience with this though, my team is still of the Team plan and thus suffered from the outage today.
The vague rumour always seems to be 'DDoS attack I guess' but there's very little in the way of formal reporting as far as I can tell...
Maybe switch to bitbucket or other competition for a while?
Eg, if your Internet costs $100/mo, but you'd lose $100/hour when it's down during business hours, buy a fallback connection from a competing ISP. ;)
Wow! That actually exists in some places? ;-)
Infrastructure so often becomes a monopoly. I can't pay a competing bridge service to drive to work quicker, I can't pay a competing gas company to deliver gas via different pipelines to my house. And I can't pay a competing electric company that uses different wires.
I actually am lucky enough to live in a city where there are many competing high speed ISPs. But guess what? I've paid for fallback connections in the past and when one goes down, the other goes down, so I go out to lunch and see the guys working on the wires in the cabinet down the street. The wires that both my ISPs share. I suppose I could get a satellite ISP? That latency. True redundancy for infrastructure is actually very expensive in most cases.
Goal: zero downtime.
vs
Goal zero: downtime.
Works on contingency?
No, money down!Probably good idea to do rolling deployment. I will be surprised if they haven't for the kind of top engineering team they are running.
Even better idea: github should stop failing.
Serious Question: is there enough people that would pay for that 0.4% to support a business?
On the one hand, I see "Everything operating normally." at the top in green, and no flags or alerts.
On the other hand, the charts look good, but "App server availability" looks interesting, the right edge of the chart is pretty much at 0%.
98TH PERC. WEB RESPONSE TIME - 1134ms
4.3x?
>I pay them. My work pays them.
Github is raking in oodles of cash and they STILL can't keep their service up without going down, quoted from OP, "[e]very couple months".
It's not about "making a better one", nor is it about paying for the fancier/premium features; it's about the uninterrupted service, which Github keeps failing to provide.
> Feel free to make a better one
The GitLab team did that already so I use their service. ;)
Problems are to be expected. But as great as it is that they had multiple levels of backup, none of them worked. They hadn't even been tested.
Which features would you like to see in GitLab? We'd love to talk about it. You could also open a feature proposal issues in https://gitlab.com/gitlab-org/gitlab-ce/issues.
https://about.gitlab.com/2017/02/10/postmortem-of-database-o...
Not perfect, just working loads better for me and my teams