Hacker News Reader

Comment by dijit | Hacker News Reader

Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.

Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.

Especially if they can't hold up their end of the deal.

Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.

View on HN

Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?

It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.

What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?

There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.

You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.

Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?

The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.

Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.

> Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders

Genuinely could use a refresher here.

I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.

I have.

You could argue the scales are different, but computers are also faster now.

So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?

https://news.ycombinator.com/item?id=47947719

https://www.linkedin.com/in/jharasym/

> You could argue the scales are different, but computers are also faster now.

Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex.

Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.

The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.

You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.

> I have.

I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow.

GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak.

You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.

I'm not sure that resorting to personal attacks against the parent commenter for making a legitimate critique is the right, fair, sensible, or mature approach here.

Discarding legitimate criticism based on some self-determined criteria of intellectual superiority isn't a good look. It smacks of elitism and isn't something conducive to a productive and positive community discussion.

It is unhelpful, rude, condescending, and completely fails to address the underlying problem.

The commenter inserted his own personal bona fides (as a proxy for skill, experience, and knowledge) and use them to bolster his conclusion of culpability and incompetence of the GitHub team. If you take that risk, you should expect to be challenged if those skills are not up to par.

Put more simply: if you get into the ring, you’d better be prepared to take a punch.

Not a personal attack to fact check someone's claims.

I didn't bring their credentials into the conversation. They did.

"false equivalence" needs an equivalence claim to be false.

I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

GitHub spent a decade asking the world to host its code with them. They got what they asked for. You don't get to beg everyone to run services for you for ten years and then have "scaling is hard" be the answer. They should be improving, not regressing over time, and they have some of the worlds best engineers and a trillion dollar corporation behind them, they don't need my sympathy.

The original question is still open and nobody's engaging with it.

> I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

Don't you at least see how it's misleading to respond "I have" in response to a question about scaling GitHub-scale services?

Trying to caveat it with "the scales are different" misses the point. The parent commenter was talking about scale.

Ok, well, I work on systems quite a bit larger than Github, and I think they have a major reliability issue.

That’s not in dispute. The question is whether we should be supportive of the company’s efforts to improve reliability, or whether we should keep punching down. How would you feel if you were in a similar situation and outsiders breathlessly provided uninformed opinions about your problem and questioned your competence?

It’s all about the Golden Rule.

Executives have made a choice to not pay for top talent at Microsoft Azure and Github.

Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.

There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.

I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.

In a SWE job market like this, do you really want to be seen as the "conscientious objector"?

There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head

Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.

Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.

But it wasn't a pig. It was a reliable system, and then it increasingly became an unreliable one, in a way that is not explainable by the mere increase in demand. Whatever rearchitecture was performed, it was done and is apparently being perpetuated by software engineers who should be held accountable. Not necessarily guilty, or even directly at fault, but accountable nevertheless. "I am just an employee of a bad company" is not a valid excuse for an engineer.

eh...

https://github.blog/news-insights/company-news/oct21-post-in...

They tried to scale MySql and turn it into Cassandra and then they lost customer data (that they claimed to later recover).

Really? Only two possibilities?

yes I would tell them "you're underpaid, if you can, come to a company that appreciates your talents more".

What you said came across as an adverse judgment of their skill and talent. Is that not what you meant?

I've interviewed at many large tech companies including Microsoft, and Microsoft Azure was mostly a clown show. So, yes I judge the talent there. I'm sure there are some superstars that I'd love to work with, maybe I'd even work there and try to fix it for the right price, but god damn was it a stark contrast to other companies.

Are you hiring?

Yes google cloud is actively hiring.

#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.

Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).

Who is “maintain[ing] the service”? The workers, of course!

When did OP blame the people involved personally?

If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.

MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.

OP didn’t blame the staff. His focus is on the company.

Invoking individual workers well-being to defend a billion dollar company is also very strange.

A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.

If anything your argument against MS’s uniqueness makes OPs case stronger.

Tell us how.

I have all the empathy for people in the world.

A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.

A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.

A corporation is made of people, but its ethos is the product of decision-making. If a corporation is consistently, say, unethical, is it because they hire only unethical individuals? Or because unethical people somewhere along the chain of command make unethical decisions?

I'm not exactly sure what you're getting at with this question. It seems to still conflate corporate-level decisions with boots-on-the-ground work.

Are you suggesting that whatever decisions their upper-level management makes that you consider unethical irreversibly and irrevocably taints all the difficult and honorable work that their engineers and operations people are performing?

I’m saying their lower-level employees are probably honest, hard-working people like everyone else. But the detachment that comes from a large corporate structure makes the higher-ups decide things that aren’t as honourable.

“Corporations are made up of people” is a strange way to excuse the reality that the ‘bad’ things that corporations do are often decided by top management.

Ah. I didn’t intend to excuse the decisions of upper management when I said that. My intent was to counter the notion that a corporation and its workers can’t be analyzed independently.

A corporation is just a business formation, and businesses are made of individual people working for it. Those people’s motivations and efforts can, and often should, be evaluated separately from the decisions of management.

We agree, thank you for the clarification. Have a nice day!

If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.

GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.

To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.

Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.

I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.

I agree 100%! But this important distinction and nuance seems to be lost here.

Defending a multi-trillion dollar company you mean (Microsoft).

I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.

Does this logic still applies if the company is getting other benefits from having me as a user? (Genuine question, I can see arguments for both sides)

For example, if I am using the free tier of a service and "paying" by seeing ads, should I have similar expectations?

I'm not saying that's how users pay for github - in that case it's more subtle, for example by giving up control of some of their stack and bolstering github already near monopolistic network effect.

I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.

As far as I'm concerned, any benefit of the doubt I might have had for Microsoft is gone after this debaucle: https://news.ycombinator.com/item?id=47989883

You must've been fuming at email clients for the last 20 years hijacking people's signatures eh?

I don't use any email clients that mess with signatures, so I think I'm fairly consistent here, yes.

> Maybe that's good-will doing the work?

Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.

You're right, but that GitHub is dead.

Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.

Any goodwill they earned has been spent.

they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.

What money do GH make off open source projects on the free tier? I haven’t seen ads, micropayments to clone repos, etc?

It's the marketing budget. People only pay for it because they've used it for free.

https://xkcd.com/1150/

Oversimplification.

there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans

Apple, clothing brands, even some Microsoft.

I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.

But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.

At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.

Using "apologia" here is pretty embarrassing.

> Aside from the fact that defending a billion-dollar company is a bit strange

More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.

1. Telling that you think the only possible motivations are financial (getting paid, stockholder, or foolish expectations of a gift from Satya).

2. Maybe you know a bunch of people who work there, could be ex-colleagues etc. and you think overall it’s mostly good well-intentioned people there. Therefore you want to see them succeed, and also you might disbelieve that the company is deliberately being awful.

I don’t have any specifically warm feelings about a corporate legal entity, but I know people who work at various companies and partly for that reason I am not rooting for those companies to fail and I also don’t believe the least charitable explanations for all their failings.

Reply on news.ycombinator.com