Comment by rglullis | Hacker News Reader

Comment by rglullis | Hacker News Reader

Parent Full thread

> Maintaining a service, with the same availability as Github, is 3 FTE minimum.

Keeping Github is a complex issue, because they are trying to offer the same service to millions of people. It is a totally different beast than having to manage a service to thousands of people. Github needs multiple datacenters in multiple regions. A company self-hosting needs one server and can handle thousands of users. A lot less complexity, a lot less moving parts.

> when your bespoke system has more than a day of downtime?

What is "bespoke" about Gitea? Or Gitlab if you want an all-stop-shop solution? They can also do all the enterprisey things that one expects from Github.

Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.

Here's a bean counter comment: Your self hosted site goes down. What's the cost per hour of my 10,000 users not being able to use it?

The same as the cost per hour of GitHub going down, I agree, but GitHub will have 20 people working on fixing it. We'll just have to wait for you?

What do I tell my 10k users?

How about security? Are you going to manage the provisioning of those users, the access control? How about audits? How about scaling? Your magical .5 FTE will do all this on top of your daily duties?

The calculus is never "just give me a raise" – it's not paying 1Password 80k vs paying some IT person that much. It's just that that money supports a core function for 10k people.

A former-boss once told me when he became a CTO of a very large company – he said, "at our scale, most of our conversations revolve around risk rather than cost. we even start to wonder, will this vendor be able to cope with our level of demands? what if they go out of business? what if they want to quit? what is the cost to our organization in time and money to have to switch to a different vendor/product/etc?"

Who knows, a company of that size might save half that money in cybersecurity insurance premiums just by adopting 1Password. (guessing here...)

It's just way more complex than you think.

> We'll just have to wait for you?

Already answered in another place. Github is infinitely more complex to keep up than an internal company service. If Github goes down, can be for a few hours. Or worse, it fails in a way that affects only a handful of users and it takes them 11 days to respond your support ticket.

If an internal service fails, the sysadmin (not the "guy who was brought in to setup gitea") can run a troubleshoot playbook and in the worst case can have a whole new server from the latest backup running in some minutes. Even better, your internal monitoring service detects that the service went down and it can be reprovisioned automatically.

> Are you going to manage the provisioning of those users, the access control?

Sigh...

A company at this size would have already a SAML/LDAP/AD/SSO solution for their other services. You just integrate your service with it.

> most of our conversations revolve around risk rather than cost

I don't disagree. I just think that is just a way to say "I want to pay to buy some CYA".

I fully believe that someone managing a larger organization has other things to look at in terms of priority, but at the same time I think accepting this blindly leads to some complacency.

It's like software developers who are so used to their top-of-the-line workstations hooked to their 3-monitor and fiber internet who forgets that their users might be connecting to their website through a $50 phone on a 3G connection.

There are other ways to reduce your risks. Prioritizing open source is one of them, as that gives some sort of "built-in" protection against vendor lock-in. Giving more autonomy to your departments so they can independently choose their IT solutions to eliminate the chance of systemic risks. Adopting a solution that let's you both outsource or bring it in-house without expensive switching costs. And all of those could be applied no matter the size of the organization, yet it seems that everyone wants to think they are some Silicon Valley unicorn and feel justified in spending ~$1M/year in a password manager.

One of the variables nobody's mentioned is whether your clients will listen to excuses and accept the burden shift using a vendor implies.

>A company self-hosting needs one server and can handle thousands of users. A lot less complexity, a lot less moving parts.

Just because you have one server doesn't make you immune to downtime. If the server goes down at 8am EST (5am PST), who is on call to fix it? Or is the server just down for 3 hours until you get into work (assuming you live on the West Coast)? That's what I mean by 3 FTE. If this is core to your company and you require 24/7 uptime, then either one person is on call 24/7 or 3 people doing 8 hour shifts.

>Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.

And if it happens at 1:30am? Or if it happens while that Sysadmin is on vacation? I don't think you are considering that your solution has a bus-factor of 1 for a service that is depended on for a 1000 other people. The idea that an $80k/year cost is preposterous for a core service only applies to people who think humans are fungible, cheap robots who don't sleep, get sick or take vacations.

I'm not saying Gitea or Gitlab are bad products. Plenty of companies self-host Gitlab with their own teams. But the idea that it costs _much_ less than Github Enterprise once you get company sizes of 1,000+ is absurd. We haven't even considered what happens when Gitea or Gitlab starts serving you 500s because your company hits some use case that the OSS developers hasn't thought of. Who fixes that? Now you are looking at sponsoring development of that. Or do you fix that? Can you guarantee any SLA on fixing any bug in Gitea? That company is moving to Ubuntu 420.69 LTS. What's the timeline on getting Ubuntu 420.69 in CI? Is that .5FTE engineering hours?

Sorry, I was abusing the terminology. When I mean "0.5 FTE", I don't mean (necessarily) that you'd be hiring someone to work part-time. What I am saying is that if you have an IT team, the time that should get dedicated to this particular service should be at most 20h/week.

Presumably, your IT team will have many other projects to work on, some of them will be on-call, some of them will be on vacation, etc, etc. But when allocating the resources, I'd guess that 20h/week is plenty to such a service, even for a corporation of thousands of people.

Does that help?

Reply on news.ycombinator.com