GitHub was down | Hacker News Reader

GitHub was down

status.github.com

216 pointsRetr0spectrum176 comments

Everyone is saying: just work on your local repo. But GitHub is way more than just git. There's bug tracking, code review, continuous integration, etc etc.

Making your organisation too dependent on a remote service can indeed be a scary prospect and I'm not sure what GitHub offers to mitigate this.

They offer a local instance of GitHub actually.

Which I can't imagine to be any more reliable than hosted GitHub.

But hey, I guess there's the comfort in know that when it goes down it's our fault!

Reliability can always be improved... It just costs more.

The argument is with a self-hosted solution you can choose your "danger times".

If you know you have a major deliverable coming up, you can choose not to touch your git server until it's at a somewhat safe moment to do so.

While it definitely sounds like a good idea, every single time I've seen it applied, the reason was the opposite. It was essentially "we've got so little control over our stuff that we'd rather not touch". Mostly goes along with lack of backups, lack of change management, and lack of automated rebuild procedure.

If you have "danger times", then any unplanned disruption in that time will hurt so much more.

I agree with you, and I'm generally in favor of having 2 or more ways of doing critical things if possible, just pointing out the response which can have some merit.

Redundancy provides availability more than reliability.

The odds of GitHub going down and your local copy/instance simultaneously is very low so availability is high.

Reliability is how often GitHub or your local instance goes down.

Your local instance of github and github.com are not redundant / interchangeable. If either of those goes down and you rely on it, the other one is not gonna help you much, so availability is not improved by using a local instance.

There's no ha for github enterprise. Single instance with hot failover.

Fossil [0] is a great alternative in some cases. It includes a bug tracker and wiki, and it's much simpler than git.

Not that I'd suggest using it for every scenario, but being aware of alternatives is always a good thing.

[0] http://fossil-scm.org/index.html/doc/trunk/www/index.wiki

> Making your organisation too dependent on a remote service can indeed be a scary prospect

At every company I've worked for, internal services have been less reliable than github. Certainly way less reliable than gmail.

I get that it's scary, in that it feels like you're giving up control over something important to your business. But I'll posit that you never actually had control, only the illusion of control.

> But I'll posit that you never actually had control, only the illusion of control.

That's a weird way of phrasing it. When you run your own services and they break you have total control and do have the power to fix it.

When you buy SaaS you are relying on someone else. You may very well have more reliability and uptime but you are nonetheless giving up control.

> When you run your own services and they break you have total control and do have the power to fix it.

Sure, but most of the time you just reboot the server or some other temporary hack and kick the problem down the road.

The power and skill to fix isn't worth much if you don't have the time.

That's exactly the point. Sure you have "control" but what good is all that control if it takes you 4 hours to track down the source of a problem and fix it (for example). Github has 100s of engineers (with heavily specialized knowledge that you don't have btw) working to fix any problems, I'd bet on them over myself and (maybe a handful of engineers) everytime ... and I've run my own subversion/git servers before.

Some folks just hate the feeling of not knowing what's happening and how long its going to take to fix, vs having direct access to work on problems themselves even though its not necessarily "better" in any sense of the word ... hence the illusion

100s of engineers working on an outage is a mess. It is more likely you have a handful of them on a given outage.

Yes, but problems are more likely to occur when engineers alter the system in some way. Now if you are in control, you can decide about windows of time in which you don't want something bad to happen (which is of course a reduced probability, not an absolute guarantee, but still the point stands).

Bad analogy time: Its like flying. You don't have control but your are better off without it (unless you are a pilot of course).

Your post gets to the false tradeoff that turned "devops" from a push for better interdisciplinary collaboration into a "2 jobs, 1 paycheck" role.

Software teams are often pathologically unable to create a deliverable that is scrutable and manageable by an operations audience. Building a product that can stand on it's own two feat without constant minding is work.

For all the testing dogma that files around, constant delivery has resulted in systems that are more brittle than ever. If someone with all the institutional knowledge is always there to catch the system when it fails, why bother making it easy to investigate failures? Why bother with analysis and building in fault tolerance when someone can worry about a long term solution for that failure mode when it causes them to get paged at 3 am?

So it becomes easy to say minders are necessary, and they must be developers. Business incentives mean that answer isn't always scrutinized as heavily as it should be, because it means spreading maintenance costs over time instead of an upfront investment in resilience and maintainability. Refusing to provide a toolset and manual for maintenance means some nobody with access to google can't fix 99% of the problems that could occur. That way the magic black box creators can ensure they're the ones getting paid to do it.

We don't have or need Windows engineers, kernel engineers, or Cisco engineers on call. Nor do we have nginx, postgres, cpython, exim, apache, php, mysql, Active Directory, Exchange or Office engineers on call. We use weird enterprise software from companies that have gone out of business, we don't have high-priority support contracts with many of the rest.

Software that needs minding from the developers is just bad software. That's technical debt they took on to get the product out the door.

> When you run your own services and they break you have total control and do have the power to fix it.

If I had total control of a service, I would make it have 100% uptime. Wouldn't you? The fact that there exists no service with 100% uptime indicates to me that nobody has total control of their services.

Similarly wrt having the power to fix my services. If I had unrestricted power to fix things when they broke, then I would use that power to fix all breakages immediately. Since nobody seems to be able to do that, I conclude that nobody actually has unrestricted power to fix breakages in their systems.

Or do you mean to say, I have some limited ability to control and fix the systems I run? I would agree with that. That's my whole point.

Internal has always been much less reliable for me as well, the difference is it's generally unreliable outside of office hours when I don't care, hosted services often go down in the middle of my workday, which is the middle of the night in America coincidentally.

Yeah. And a fact that may be one reason is that blaming a outboard supplier like github is easy, but blaming a colleague in company is more complex.

They offer GitHub enterprise you can setup internally if you want.

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

Gitlab?

Does anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance.

edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people.

edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.

Gitlab.com? BitBucket is the other major player that comes to mind, but googling "hosting git" returns enough results to melt ones mind.

If you're really concerned, you run it yourself. GitLab insider a docker container isn't hard to manage. You can even 1-click Gitlab onto a DO Droplet in just a few minutes: https://www.digitalocean.com/community/tutorials/how-to-use-...

Gitea[0] is extremely simple to host, if you want a self-hosted solution that's not as heavy as gitlab.

[0] https://gitea.io

Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).

Thanks for mentioning that. It seems like this bug wasn't reported yet, so I have filed a bug report.

Also, I don’t seem to be able to properly run it in docker (I’m trying to deploy it on kubernetes), do you, by random chance, have experience with deploying gitea?

Sorry, I don't know about running it in docker. My usecase is personal lightweight repos management, so I just download the binary and run it backed by SQLite. Never faced any issues with that. I never needed to go fancy with docker.

Consider running your own Gitlab instance.. I run one for $40/Mo on DigitalOcean, very happy with it.

Samba surely, wanna cry about having to set up file:// remotes?

Work locally? It's Git.

sure, but if your Build System / CI is based around github, then you are a bit poked.

thank goodness I'm on bitbucket :) One of the things they offer is your own local server synched with their cloud servers for redundancy.

> sure, but if your Build System / CI is based around github, then you are a bit poked.

then architect your build system be able to switch to another service with the flip of a switch, if its that important to you.

Github has added all sorts of proprietary integration points to prevent it from being this easy, like push based checkin notifications. Then developers get lure by the drag'n'drop CI servers that integrate with them.

Unfortunately people only care about API's now instead of standards.

With each GitHub failure people learn some good lessons. Redundancy is the key. Push to multiple remotes and fall back to backup remotes in your scripts.

If everyone has SSH access to a server somewhere, it's easy to just put a git repo there to let everyone keep working.

Obviously, this doesn't trivially scale to many repos/users/etc, which is why Github exists in the first place.

It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all.

However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.

It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.

It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server.

This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently).

What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.

Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.

Just because they sell it doesn't mean people shouldn't know better.

How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.

Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't.

If Github being down means you're not fixing your site today, I'd call it a runtime dependency.

I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.

see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.

Why it is down tomorrow?

(Hint: sans hug of death, because a runtime dependency brought it down.)

Incorrect. You need build-time dependencies to change an application, and runtime dependencies to run it.

Deploying/scaling is somewhere in the middle.

Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.

naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

I think the point is git is becoming the new svn for many people.

"Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth?

The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.

There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.

But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.

Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.

Not if you're using more than git from github, for example downloading releases as zip files for deployment.

There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.

Yeah definitely. I was just thinking about the basic use case where you deploy from it and that's it.

Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.

Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.

> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?

It's not a runtime dependency, it's a stupidity to rely on random repositories from the internets when building and deploying (remember the left-pad farca that halted work for half of the web programmers). You should always have your local copy of necessary sources or use repositories that can trivially be swapped with some other mirror.

We used to have the infrastructure for those things (in the form of binary and source package repositories), but it was mainly used by sysadmins, and programmers usually have an alergy for such infrastructure.

Speaking from an operations perspective, yes, it's a runtime dependency. From a development perspective, you might think of it as a "deploy-time" dependency. Think of it in contrast to compile errors. If your program has errors that your compiler can catch, then it will refuse to compile the program and you must fix the error before executing the program. A runtime dependency is one that may change after your program is compiled -- these are harder to test, especially when the program may fail depending on its environment. In the case of dependencies on services such as Github at deploy time, it's a "runtime dependency" for the operations programs. Since the operations programs are the ones with the runtime dependency, it's also in the operations team's purview (and responsibility) to revise their program(s) and eliminate the runtime dependency such that testing (and reproducibility of) deployment is deterministic and reliable.

> Your question seems to indicate that you have a runtime dependency on github.

Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.

But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing.

Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.

So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.

> you can instead push your artifacts to an s3 target

This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).

Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.

You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.

Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration and deployment. It's still early days, though, and there's a lot that a new user can get wrong in setting such a system up. Ansible telling servers to git pull from Github may give you better uptime once you factor in your own mistakes.

I agree.

Github is an off-site backup, and a nice interface for pull requests and code browsing.

There's just no reason to make github a critical part of your infrastructure.

> then some automated script will tell the cluster to pull from Github

GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.

Could you elaborate on how Kubernetes makes it easier to deploy code changes directly to the cluster? Are you talking about the Docker/container image build?

A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.

Exactly. There are no right/wrong or absolutes. It's a trade-off.. if it makes sense in someone's situation, it makes sense.

> github is down rarely enough

Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket.

Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).

AWS CodeCommit has worked pretty well for me. I generally trust AWS/GCP/Azure over others regarding availability.

You need to either ensure you control the resources you depend on, or plan for outages.

At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos.

Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.

just set up a server via webhook that pulls the repo everytime someone pushes to it. When GitHub goes down you can use that server.

Most items of ours that have a production dependency on GitHub actually point at a simpler (and much stupider) mirror. When commits are made pushes are mirrored to those repos. These repos can even be naively replicated since git's nature allows for this.

If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors.

Also make sure your tools don't lock on only one item.

At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked? Get your popcorn out. It will happen.

Is GitHub even allowed for govt contractor use? I'd be surprised, but don't know.

For some projects, yes. The biggest well known US examples are the 18F open source efforts and http://code.gov / http://code.mil open source efforts, most of which are public, open source on GitHub.

I'm pretty sure most, if not all, government agencies require on-premises hosting of code

There's e.g. https://aws.amazon.com/govcloud-us/details/ these days. Azure and Google have government offerings as well. Safe to say neither would be bothering if the government didn't use cloud products these days.

http://fortune.com/2016/09/02/us-government-embraces-cloud/

Looks like even the CIA is using some AWS.

"U.S. government agencies are moving to cloud computing and away from their own data centers faster than private corporations"

Github Enterprise

It was in lunchtime in New Zealand, so doesnt really matter. ;) I am just back with our world famous wellingtonian flat white, and looks everything perfect. :)

Reinvent VMS-style clustering on BSD/Linux boxes or just buy some VMS servers. They measured uptime in years to decades for tech that started in the 80's. There should be several, modern solutions like that given companies were developing them a long time ago when I first looked into such things. I just haven't surveyed to see what exists, is reliable, and is affordable.

They could also have a base version of the site that's read-only for source or binaries to allow customers access to the data to fall back to whenever problems happen in more complex functions. Keep it running in some form while they fix things. They might also similarly sell Git appliances w/ high-availability that themselves plug into multiple colo's or cloud providers mirroring changes in the repo to the Github site. Just a wild idea as I have no idea if that's marketable.

You can run a local copy on a server of something like Github. For example BitBucket Server (which used to be called Stash). I worked somewhere that was on GitHub but cried when it was down, then went to BitBucket and cried when it was down. It's a big point of failure.

Sublime Text also is depending a lot on github.com these days with Package Manager...

Use Docker.

When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.

Self hosted bitbucket and the like really. Or AWS CodeCommit. We are currently using both of those in our build pipelines.

> Run my own git server like the good 'ol days?

Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).

Blip, its back up. Nah, too much effort to host my own and I do need a coffee break from time to time.

Ignore this if you are on a big team of course.

if only we had a decentralized version control system...

Exactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.

Then what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.

Push and pull to your own server running Gitea/Gogs or Gitlab, and mirror to Github if you feel the urge? Or, if you insist on having your workflow around Github, do it the other way around: have your local server mirror your Github repositories. That way, if Github goes belly-up you can just switch to your local server for the time being.

If your workflow depends on centralization, run your own centralized service. If you can't live without the GUI, Github will even sell you Github Enterprise. Or just run Gitolite or one of the other tools.

You can set up another git server and webhook pull from the main one whenever someone pushes there, so you have a backup replica. Coders just need to setup two remotes for the repos and use the second one when main one is unavailable.

I always thought that an IM-style tool with online/offline presence information of remote repositories could be a fascinating tool for DVCS source control repositories.

Tried to prototype on XMPP at one point in college but never got very far in the prototype.

(That was back when I was a heavy darcs proponent too and had other crazy things like automated consensus branching as a great distributed way to coordinate efforts on such a distributed project, which would work with the darcs push/pull/merge model but not so well in git's.)

it's not about github though, it's about how people use git. nothing stops one from keeping a mirror, just in case similar situations happen

Yes, exactly. It's great that we can continue along editing and committing. The freedom that DVCSs bring has really changed the world of development. However, feeling the pain of not being able to access other repos shows both the value that Github brings and also the downside of siloed services.

Centralization is an artifact of human societies (programmers & organizations) decision making process not result of existence/lack of decentralization features.

Fossil [1] has always been my go-to DVCS for ages. Not only can I host it myself. I can also run it when completely offline. It offers Bug Reporting, a Wiki and both a web interface and CLI.

Created by Richard Hipp, creator of SQLIte, it's open source and free, has zero dependencies and works across all operating systems.

And even if you don't want to host it yourself, there's always chiselapp.com [2]

[1] http://fossil-scm.org/index.html/doc/trunk/www/index.wiki [2] http://chiselapp.com/

This is the 3rd major outage in a week, wow.

App server availability is now 98.3% over the past month, which seems pretty bad!

Good ol' one 9 uptime.

"One nine up-time" rolls right off the tongue! Must be a PR play!

Poor gals and guys. Godspeed to them.

Such incidents remind me of 2 things

1. No matter what 99.99999999% availability a service provides, its utterly useless if the time to get back up is unacceptable.

2. Do not have remote services, which you cannot fully control to be a part of your run time deployments.

Uptime % includes time to recovery. Since you're not up if you're still recovering...

Yes. And recovery time should be tracked separately from uptime is the takeaway imo.

But that still does not provide much inside into recovery time. Its good to know, the median recovery time.

The only time that uptime would not give insight* into recovery time is if the service is not being regularly updated (remember, zero-downtime deployments should never affect your uptime numbers), then, the uptime numbers improve over time to something really impressive but go in the toilet once the shit hits the fan.

I hardly think it's reasonable to consider services like GitHub or infrastructure like AWS as being irregularly updated...

I think GP's point is that you should assume the 0.{however many 0s}1% it's down is going to be a bad time for you, and then ask yourself how long you want to wait to be back up.

A long down time might be worse for you than a shorter down time more frequently.

I fail to see how.

99.99999999% uptime is 3 milliseconds of downtime a year. That's not even a TCP retransmission.

Well for a start, ten nines is a lot, isn't it?

If this service is up for five years, then it can go down for 15ms and still claim all ten over that period. Which is almost nothing still, but if you needed ten to begin with then maybe it's quite a lot for you.

All I'm saying is - and I think this is what the commenter originally twice above me meant - is that the rate doesn't give you all the information, you at least need a period as well.

I can't understand why people include GitHub as a deploy time dependency. Deploying should be about pushing an already built package to the destination server(s).

The idea that you push the source code then build it on the server makes no sense to me.

If I was github I would forbid (and punish) this kind of behavior.

If you want to host your own repos with user and access management, bugs, issue, task tracking and tons more, take a look at Facebook's Phabricator https://www.phacility.com (the self-hosted version, of course).

Tangential: Slack was down for a bit earlier today as well.

I'm seeing errors on Segment.com as well.

Ha ha, nice!

https://status.github.com shows

> 18:19 CDT Major service outage.

What's CDT?

Central Daylight Time

Seems like they're saying it's maintenance now?

  $ gl
  fatal: remote error:
    GitHub is offline for maintenance. See http://status.github.com for more info.

That is likely just their standard downtime message.

Only reason I brought it up because it was originally showing the all to familiar unicorn on the website, then it changed to also saying maintenance.

This seems like a pretty common response for a breaking incident for a scale app. Requests flow through to a failing system and trigger HTTP 500. Those requests may pachinko through the stack, making a variety of calls that can compound the degradation of a system weathering an unplanned failure state.

Engineers stop the bleeding by 503'ing requests at the perimeter or putting up a static maintenance page. This allows things like caches or DBs or app servers to cool off while a rollback or a revert goes out. Then, when the system is stable again, let requests flow through again (slowly, of course).

And now the office goes to the keg.

As the last time, pull/push access to repos works fine for me, only the web service doesn't work.

edit: confirming, push doesn't work either

Pushes are failing for me. (To one repo, at least.)

same

Pushes are failing for me with this error: "remote: fatal error in commit_refs". I'm seeing it across various repositories so it's not just one particular repo.

Gone now!

18:41 CDT Everything operating normally.

It's sometimes up ( perhaps bad to say this --> everyone will refresh )...

This is different from yesterday's GitHub Major Service Outage.[1]

[1] https://status.github.com/messages

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We purchased systems from Sun because of the quality of the HW and the stability of Solaris. Now a stack of cheap linux boxes with the right design (IP CLOS, BGP fabric, load balancing and containers) who cares if a switch fails or a rack goes out? Most issues today are caused my humans and bugs from software upgrades for new code.

Kind of?

How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape.

Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

Sounds like an argument against micro services?

I'd give a dollar to hear how you arrived at this conclusion.

Only if you don't stop to consider the alternative.

"The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."

I think you both have a point. The tools are mostly all here now (some in primitive forms), but no one person on earth knows how to use them all. From netfilter to kubernetes to sharded redis instances to nginx RTMP to embeded golang structs and git commit hooks.

It takes an amazing team to cover all the bases.

You are thinking small scale, and public web.

The fact that VAXClusters were already going out of fashion by that time, and the fact of IBM's parallel sysplex existing by that time already negate your point.

previous $employer had built a globally distributed multisite c&c processing system twice by then and were in the process of revamping it for a 3rd.. 1st on mainframes & remote serial compute devices in the 80s, then on unix workstations in the early-mid 90s.

They were by far not alone in dealing with this level of complexity for the time.

My ISP?

5 racks in 32 cities as POP. Each POP had 10 Sun Netra T1 1RU boxes behind SLB. All those were a service cache to offload the main DC which had...hum... ~100 or so Sun E450 (4x480Mhz Usparc3, 4GB RAM, 20x4GB or 9GB UltraSCSI3 drives, 2x1G NICs).

There was a cluster filesystem also. We had huge databases, etc.

This is just not that hard today.

In 1998

There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later.

There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

It was not a criticism so much as "well, it will be interesting to see what went wrong."

My bet is a human.

Is there, fundamentally?

Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either..

which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..

Agree. Hotmail IIRC had 9K or so servers. Yahoo had even more.

I think doing this at scale is simpler today because the networking is simpler, i.e., IP fabrics and VMs and containers. If you are building something today and want to use Layer2 I question your santiy.

Yeah I think there is. This is what wikipedia says Altavista

"As of 1998, it used 20 multi-processor machines using DEC's 64-bit Alpha processor. Together, the back-end machines had 130 GB of RAM and 500 GB of hard disk drive space, and received 13 million queries every day"

There are probably more read-only queries to gists per day than that.

Indeed. I don't think I even heard of the C10K Problem until '03 and it was still unsolved. Now we have servers doing multiples of that much traffic on off the shelf equipment.

> fairly simple

mmhmm

The future is here, it's just unevenly distributed.

Someone is doing all of the kinds of work necessary to verify that the servers are up, being useful, and nominally working the way the previous deployment did, but not everyone has the tools yet to do that (especially the last one).

We need a compete and complementary tool chain that everyone can use, and so far a lot of these are still business differentiators (i.e., proprietary)

Oh man was just browsing a 3 month old branch for a feature that was just revived today, did not clone... Please come back!

Run your own git server. Version-control locally, build automated systems to push triggered by uptime status.

And its back!

It seems to have been fixed now.

> 00:41 BST Everything operating normally.

95.8%. What would AWS think about that?

Does this mean we get the day off?

I'm down for that :D

Ugh, time to go to bed then.

anybody knows if www.atom.io would be affected by this?

And its back!

Not sure yet, but it seems to be due to AWS (S3 likely).

Rough year for GitHub. GitLab is licking their chops right about now.

at least they didn't delete the db.

Ohhh yea, shows how good my memory is.

There's still time!

Wow.

Once is an accident. Twice is a coincidence. Three times is an enemy action.

Both days a different error on pushing, making me think it wasn't down and it was my fault!

Reply on news.ycombinator.com