Github.com Down Again? | Hacker News Reader

Github.com Down Again?

github.com

116 pointsleesalminen77 comments

Partial outages of cloud services feel more and more common lately. AWS had degraded service a few weeks (months?) ago. Am I mis-remembering how robust everything except Reddit and Twitter used to be? It feels as if the internet caught a nasty cold at some point and just can't kick it.

Also, it seems way more common to find devs you work with to be over-idealized (focusing on “code beautification” or some other vanity metric vs. accomplishing a task). There’s nothing wrong with beautiful code, but I’ve seen the complexity it can add to an otherwise simple project.

To me, if the code is not "as simple as possible, but not simpler", it is not beautiful.

And there’s nothing wrong with that, but, I’ve seen some devs these days spend waaay too much time on “making it simpler” or whatever their ideal is, at the cost of error handling or you know, solving an actual problem. Personally, I don’t care what the code looks like as long as it’s resilient in the face of unexpected errors (ex: a user input of 0 to a division function is not “exceptional,” you should have anticipated that, while a service input of the same may be a breach of contract between the services and IS exceptional) and actually solves a problem.

You're contradicting yourself here already.

Resilient code is easy to read/grasp code. It should pass a 3 am test too.

I really think you're wrong to say there's no tradeoff here. Error handling makes code bigger and harder to read.

I’ve been noticing the same thing too. I wonder if the state of the job market over the past couple of years has anything to do with it. More people leaving and taking their knowledge with them, meaning issues are identified/fixed at a slower pace. I know I’ve seen it happen at the company I work at, but I wonder if big tech similar issues.

Thanks for sharing this insight, I hadn't put it together, but it's an interesting theory (or maybe lemma).

A colleague got an AWS error saying there weren’t enough resources available to provision an EC2 instance and to try a different region or AZ. Not sure how common that is.

Data centers are finite and I assume Amazon has some metrics on when to stop allowing new instances (even if they have capacity) so they can continue to serve existing customers.

I've seen that regularly with spot instances and new instance types. I assume spot instance availability is correlated to on demand capacity (why would AWS discount instances at spot prices of they were able to sell at the on demand rate).

Azure has been having capacity problems all year and some new instance types aren't available to customers or you need special usage requests approved

I've heard of it happening to colleagues twice in the last three years (though it has doubtless happened more frequently than that) - I think that last time was because they were requesting some ancient instance type that AWS is probably phasing out.

Depends on the region, availability zone, instance type, and time of day. There are always resources if you are flexible. I’ve only ever seen that message when dealing with spot instances or cutting edge hardware that was in the process of rolling out.

I’ve noticed a number of changes over the past couple of years to make it easier to manage mixed pools of compute resources - both spot and on-demand - a good way for AWS to spread load across different instance classes.

Well, there IS a cyberwar happening right now.

Is there a real-time map?

https://www.fireeye.com/cyber-map/threat-map.html

Or build your own: https://github.com/hrbrmstr/pewpew :)

A real-time map of cyber-attacks is actually nonsense.

You might visualize a small slice of DDoS attacks and similar routine automatically detectable botnet stuff, but nothing actually serious or interesting.

The trend towards cloud architecture has meant consolidating a lot of products onto just a handful of cloud solutions.

You trade a very bubbly flux of outages across countless bespoke deployments, hosts, and topologies for something with failures that are more rare but also more widespread when they occur.

Personally, I think what you’re noticing is that outages and failures are much more high profile than they used to be because it’s bigger news when these big bubbles burst than it was in that noisier flux where small outages were occurring all the time.

I think as dependence on web services increase there's also higher expectations about availability. You have giant services like Twitter, Facebook, and Google that are almost never down and start to expect that of other companies

Github is Microsoft. I expect the same from them as I do Google.

So, funny second hand story. Google’s accountants were using third party software hosted on GAE. They bitched to the third part about how the downtime of the software was unacceptable. The third party pointed out that each instance mentioned was a documented GAE outage.

After some internal discussions, the accountants apologized.

Google isn’t exactly a shining star.

Whenever GitHub disappoints me, I just remind myself how much better it is then Perforce.

Github.com can be reached using some AWS IP addresses, as well as some Microsoft Corporation IP addresses, and who knows what others. But one can also still use original GitHub, Inc. IP addresses on AS36459. I only use the AS36459 addresses for github.com.

Here are some Github, Inc. (AS36459) IP addresses for github.com

   140.82.112.3
   140.82.112.4
   140.82.113.3
   140.82.113.4
   140.82.114.3
   140.82.114.4
   140.82.121.3
   140.82.121.4
   192.30.255.112
   192.30.255.113

What, do you update your hosts file or control dns through pihole in order to achieve the selective A recs?

I have a RPi but I run NetBSD on it, not PiHole. AFAIK PiHole is just a modified dnsmasq and I dislike dnsmasq. I always used djbdns, never dnsmasq. Today, I use a custom root.zone loaded in nsd and a loopback bound forward proxy that stores the IP addresses for all the websites I visit in RAM. nsd listening on a loopback address just substitutes for a firewall rule directing traffic to the proxy. I use the HOSTS file for FTP server addresses.

These github.com addresses I listed are static for years at a time. As such, there is nothing to "update".

Thanks for the info/reply

No you're all mistaken - there hasn't been any down-time at all throughout all of September https://www.githubstatus.com/history.

Edit: GitHub making me look a fool by just updating it now (they didn't for the earlier outage...)

there is no outage in Github Sing Se

It was a bright cold day in April, and github.com was up…

It happened to me and I was also confused. I saw it wasn't on the status page, and I thought, GitHub must be like Amazon, unwilling to admit fault on the status page.

Just link how successful a team is with a metric, OKRs are great! Downtime is gone now someone’s promotion depends on it… /s

Oh, I mean, let’s redefine downtime so we can have nice numbers and everybody gets congratulated. /s

When you measure something for “consequences” you tend to destroy the value of the measurement.

Looking at downdetector.com right now and YouTubeTV, HostGator, DirecTV, YouTube, Paramount+, crunchyroll, slink, Philo and Roku all have big spikes in the past few minutes. Wonder how/why Github is related to all these streaming services.

I wonder if the current US West Coast heatwave is causing some shared cloud infrastructure to go down.

My guess is that there is zero correlation between all those products you mention, and Github. DownDetector is interesting, but I find it to have more noise than signal.

There is a noise floor but I’ve found it to be a pretty reliable early indicator of problems when the question was “is it just me or…” and far superior to any company’s status page.

I've seen two different error pages intermittently. One is a 500 page with a cartoon character holding a sign that says "500" and the other is a page titled "Unicorn!" with the message "We had issues producing the response to your request" and a link to a status page that says everything is fine.

those sound pretty exciting my error page just says:

"No server is currently available to service your request"

and the images are all broken.

I've never seen the non-unicorn one before:

https://i.imgur.com/rjw9ekL.png

Luckily, Git is decentralized, so that shouldn't be an issue.

I guess you don't use GitHub Actions or a CI build workflow ...

Exactly my point.

Sarcasm doesn't work on the internet - there's no way to read your tone.

More like it doesn't work on the internet with a general audience, because some people will need you to say something in a funny voice to know not to believe it.

In this instance I’d say it’s rather a question of how naive/innocent you think the poster must be for it to not be sarcasm.

It also doesn't work very well as "making a point" sarcastically, because that largely depends on pretending github only does git, and there's very little reason to do that.

The point being made is that we lose the decentralization benefits of Git if we tie ourselves to the proprietary functionality of centralized platforms, as the current event demonstrates. Obviously Git being decentralized doesn’t help much when GitHub is down. But why is that so? Because of GitHub’s proprietary centralized features. The comment was intended as food for thought that maybe we as an industry shouldn’t tie us so much to those features then.

Wouldn’t it be wonderful if when GitHub is down we could just move to a different GitHub instance? Isn’t it sad that we can’t, although it would be very feasible technologically?

But we're not losing the benefits of Git. We still have those. We're 'losing' the benefits of decentralized build servers, but we never had those to begin with.

This is a problem to work on, but it's not a Git problem.

I feel like we kinda do this already it’s just that both servers are owned by GitHub and you can’t tell that it’s happening.

Actually it works pretty well outside of HN.

> Sarcasm doesn't work on the internet

Ain't it cloud scale? Have you tried moving it to serverless?

Yeah. We also need to discourage this 'centralize everything' [0] nonsense and going all in on GitHub services as well.

At this point, it seems that GitHub is even more unreliable than a self-hosted solution.

[0] https://news.ycombinator.com/item?id=22867803

Tell that to people who deploy from Github.

It’s still intermittently down and most CI jobs are failing with network or authentication errors. A bit disappointing that the status page makes no mention of this.

It was also giving 500 errors about 14 hours ago for 30 mins and went undocumented

Based on the comments maybe they were just A/B testing the new 500 error page?

Based on data we can report that 90% of the customers do not like the 500 error page.

Again? Looks like it is always down at least twice a month. [0] Doesn't affect you, if you are self-hosting like GNOME, KDE, Linux, wireguard, Redox OS, Wine etc who are not going all in on GitHub services.

GitHub is still unreliable even after two years of calling it [1] with little to no improvement.

[0] https://news.ycombinator.com/item?id=31921324

[1] https://news.ycombinator.com/item?id=22867803

The errors seems to be rather intermittent. Metrist saw a grouping of them at about 4am pacific today, and then again 3pm. That said, in the last hour, less than 1% of Metrist functional tests failed.

For some reason I feel like I missed the timeline where "free" cloud services are hosted at the public library, and paid with a monthly subscription.

Specifically a registry of all available services worldwide, by location, and automatically billed and paid to the right entity. So if you want X service, pay for it, Y% goes to the software maker, the rest to the local host, and automatically gets installed at your location of choice.

Artifacts – C'mon Wit Da Git Down

https://www.youtube.com/watch?v=XimyEvDJWzA

I've been getting the unicorn for the past few minutes... reminds me of the Steve Miller Band logo. Nothing on their status channels as of yet.

West coast servers having trouble with the heat wave?

Yep, crashed within the last 5 minutes.

Yes, intermittent but even partial loads stall out. Was about to submit for the same reason.

Yes. Statuspage is not yet updated but down at least for the last 5+ mins

Seems to be timing out pretty often on my end for the last 2h.

Seems to be back up again now, was down for me too

Was down for a couple minutes, is back for me

Mine is working , web interface and CLI

Edit: northeast USA

I've seen more unicorns today...

A unicørn once bit my sister... No realli!

Come clean: that was just you, wearing a unicorn hair band.

its back.

It's down again.

Crashed but seems to be up again

Could an on-chain Github.com fork prevents those down times?

No.

Reply on news.ycombinator.com