...is it time to move away from Github?
Of course, I'm not actually internal to Microsoft or Github, so I have no idea and it's all opaque to me.
Also, I'm still anticipating a fuller report on the database issues they mentioned have been the root cause of many outages over the past few months.
This page https://web.archive.org/web/20190801000000*/https://github.c... has been[1] “500 internal server error” since late-December (globally it seems). Nobody cares (“not a priority” (c) support), nothing on githubstatus.
[1] blue circles on web.archive are errors too
It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.
If you want to see more the state of github "extras", you'd need to select "github actions" or "webhooks", which have a fair amount of downtime (about once a week or so, which seems about right).
Interesting how the most stable component of the company is the open source one of course ^^
Much if the work is just minimizing the impact of this changes by finding them before customers do.
This includes things like unit, integration testing, canaries, cellular/ zonal / regional deploys, auto rollbacks, multi-hour bakes, auto load tests, and much much monitoring. Not to mention cross team code reviews, game days, ops reviews.
It's also the highest performance software forge by objective measures:
Full disclosure: I am the founder of SourceHut.
And I believe sr.ht would beat out GitHub at their scale anyway. The services are an order of magnitude more lightweight. And the design is more fault tolerant: we use a distributed architecture, so one part of the system can go down without affecting anything else - as if GitHub's issues could go down without anything else being affected. And many of our tools are based on email, a global fault-tolerant system, which would allow you to get your work done more or less unaffected even if SourceHut was experiencing a total outage. We'd automatically get caught back up with what you were up to in the meanwhile once we're online, too.
I've spoken to GitHub engineers about some of the internal architectural design of GitHub, too, I'm confident that SourceHut's technical design beats out GitHub's in terms of scalability. And, despite already winning by a good margin, I'm still spending a lot of effort to push the envelope further on performance and scalability.
And then you go on to say it. I'm glad that SourceHut exists, and I like many of its principles, and it's probably better designed too, but walking into a thread where someone is having an outage and then claiming that you'd do much better is in poor taste no matter how good you are or how many of your services work offline.
Redundant power supplies
>pushing a bug into production
Nothing, but again, SourceHut is demonstrably better in this regard: because it's distributed, a bug in production would only affect a small subset of our system, and the system knows how to repair itself once the bug is fixed.
And I don't think I need to apologise for kicking Golaith while he's down. Someone said they want alternatives, so I pitched mine with specific details of how it's better in this situation, and that doesn't seem wrong to me. I would invite my competitors to do the same to me. We should be fostering a culture of reliability and good engineering - and if I didn't hold my competitors accountable, who will? "Here's an alternative" has more teeth than "I wish this was better."
Most of us could throw any of the open source solutions on a $20 Linode instance and probably have excellent uptime. How many active repos do you host, and on how many servers?
Can you talk about the geographic distribution of your 10 servers?
SourceHut is not the same scale as GitHub. This does not change the fact that SourceHut is faster and more reliable. We have an advantage - fewer users and repos - but still, that doesn't change the fact that we're faster and more reliable.
This has been objectively demonstrated as a numerical fact:
And yes, 9 of those servers are in Philadelphia (the other is in San Franscisco, but it's for backups, not distribution). That doesn't change the fact that, despite being more distant from many users, our pages load faster. In this respect, we have a disadvantage from GitHub, but we're still faster.
GitHub and Sourcehut are working at different scales. That doesn't change the fact that SourceHut is faster.
> we use a distributed architecture
> SourceHut is faster
I wasn't questioning that some of the web features are fast. I'm sure when Github was 10 servers their pages were fast too. I suspect if I threw Gitlab on a 9-server cluster on AWS they'd also be quick.
And yes, throwing GitLab on a 9 server cluster on AWS might be fast. But, I'm ready to bet you that SourceHut will be faster than it still, and I have a ready-to-roll performance test suite to prove it. And I know that SourceHut is faster than GitLab.com and GitHub.com, and every other major host, and you don't have to go through the trouble of provisioning your own servers to take advantage of SourceHut's superior performance.
While your tests are indeed objective, I don't think they're very useful. For example, why does your performance test ignore caching?
GitHub's summary page loads 27KiB of data for me unauthenticated, which is about 6% of the 452KiB you're displaying in your first table. The vast majority of developers who browse GitHub will not be loading 452KiB of static assets every single page load.
Anecdotally, GitHub's "pjax" navigation feels about as fast as SourceHut on my aging hardware.
https://news.ycombinator.com/item?id=22936448
SourceHut is at least 10x lighter weight and has a distributed, fault tolerant design which would allow you to continue being productive even in the event of a total outage of all SourceHut services.
When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.
That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.
That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.
ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.
You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.
That’s true modernism.
So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.
Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?
It's much less complicated (both from an admin standpoint, as well as a UI standpoint) than GitLab. I paired it with a Drone installation (also self-hosted) for CI and (sometimes) CD.
It all works great, and is way easier than I thought. If there's downtime, I'm (usually) in control of when or how long, as I have root on the box.
I'm also not giving my money to a giant military contractor (Microsoft, the owners of GitHub) any longer, which is a huge deal for me from a personal moral standpoint (YMMV).
Barring that you always have the tried and true (and for some reason abhorred by start-ups) option of running your own gitea or gitlab instance. Its not hard, and most of this stuff can be done in dockerless containers if you want.
If cloud servers are getting "overloaded" as some commenters say, you could even buy a few racks or u's of colo somewhere or use a cloud provider that isnt the most popular meme on YC. Vultr and ramnode are both good options and youd be supporting a small business, not Bezos next giga-yacht.
vultr is considered a small business now? crunchbase lists them as having 50-100 employees, and they seem to be owned by Choopa, LLC which some sources list as having 150 employees.
I chalk most of the early ones to moving services over to Azure.
Lately though, I don't know. Azure is running pretty close to capacity, so maybe it's part of the problem
https://github.blog/2017-10-12-evolution-of-our-data-centers...
The fact that you used "M$" indicates that you are predisposed to blame Microsoft for actions that are likely not from the parent, and discount any changes from the top down that have occurred within MS. And while I have a lot of issues with MS and Windows in particular, MS today is not the same as MS even a decade ago.