Hacker News Reader

Comment by nijave | Hacker News Reader

I don't think it's necessarily just "personal use". The complexity quickly scales with the company size. Replacing Github with Gitlab might be a week-long project at a small company but could easily turn into a multi-year project at a large one.

It's not just a technical problem, either. Bigger companies tend to have more expertise-oriented teams (security, compliance, developer tooling, operations, internal infrastructure) which tends to make decisions more difficult than when a single person or team can do it themselves.

View on HN

> Replacing Github with Gitlab might be a week-long project at a small company

A week-long project initially. Now you have to install updates, set up and maintain secure access, reboot or troubleshoot when it dies, etc. Installing things is the easy part.

What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems?

The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivot to work on that, now the CI/CD server is down/slow/randomly breaking. But the sysadmin that knew the ins and outs of it left last quarter so the new sysadmins don't want to touch it. Oh, and management doesn't prioritize any of this stuff, so actually jumping two versions on the git server is a much bigger, fragile ordeal now than it was a month ago.

Exactly.

Our team of 6 sysadmins manages:

- DNS appliances, storage appliances, NTP appliances,

- hypervisors, Dev/stage/prod k8s clusters, some other k8s clusters

- dev/prod Elasticsearch/Logstash/Kibana clusters

- internal GitLab, Jira, Confluence, nautobot, OpenDCIM, a deprecated Twiki

- several internal custom apps

- Probably more I am forgetting.

Nothing gets patched consistently. Everything is neglected to a certain degree.

I'm not going to die on this hill, but that seems like a lot of complexity for a company with in-house skills, maybe even the worst of both (in-house vs cloud/managed) worlds.

In a previous job I managed a migration from bitbucket (!) to self-hosted Github Enterprize. As part of the migration I forced the creation of a monthly "backup restore test".

Every month we had a recipe to spin up a new instance of the Github appliance, and import the most recent backup into it.

A lot of systems do tend to get a little neglected over time, and of course different organizations have different priorities, but I insisted on this because I figured for that particular company the Github instance was one of the most critical components - if it is down people can't work.

You'd be surprised. Most dedicated sysadmins and DBA positions were done away with as part of the "devops" movement, especially as everything moved to the cloud and there were no longer physical systems to maintain. Developers are smart, they can just do that work too, right? It's all just typing. /s

Loads of non-profits and small businesses that have surprisingly demanding tech needs but can't afford / don't understand / can't manage / have only a part-time need, etc.

Its literally essential to have solid tech these days. I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept

It's not even close to essential, since by far the majority of businesses are running without anything close to what people on HN would consider adequate. It only becomes a problem if too many of your competitors have good tech and even then only if your industry depends on (software) tech. Companies like bakeries and building contractors can run fine with shitty IT and even if their competitors do IT better, no customer is going to drive further to get a bread from someone who has proper backups instead of from the closest shop.

> I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept

If a business does not have someone knowledge about IT, how can they know that they have 'deficiencies' with regards to IT?

Things are the way they are and they think that's 'normal' because they don't know any better.

Big business: We can look at our SIEM and endpoint management dashboards and determine that our end user machines are reaching end of useful life due to resource consumption and failure frequency

Small business: Please IT help my 5 year old laptop can't keep up and I swear it's not a virus. Yes, I updated everything

It's amazing watching friends/family outside of tech work. Within 2 minutes of watching someone's machine slow to a crawl, I open Task Manager on Windows and see they're out of RAM and their machine is heavily paging: Yeah, if you had another $40 8GB stick of RAM your machine would be significantly more responsive.

Sysadmins are expensive. >$100k/yr (in the US, at least) for a full time sysadmin when your other employees are taking $40-60k/yr is a pretty huge expense. If you're not a technology company, there's a good chance you'll look to just outsource it anyway (how are you even going to vet/hire someone in IT when you have no IT department already)

> What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems?

Even if they have one, that sysadmin could be incompetent or out of their depth.

Just started a new job where the last guy was decent enough, but what ended up being built up needs to basically be redone, e.g., a giant /21 network: no server VLAN/subnet, or separate network for network management interfaces or server IPMI.

> could be incompetent or out of their depth

Or just overworked. In theory you can self-host everything, but there's a always a time tradeoff.

Haven't worked at a startup yet with a sysadmin. They either have devops or SREs take care of infrastructure but those types tend to be cross functional so they're also handling infrastructure, operations, incident response, developer tooling, CI/CD environments, code performance issues (basically everything that's not application development).

One company I was at heavily used SaaS and AWS for everything so there weren't very many systems to manage. We'd just jam everything in Docker containers and use AWS ECS. Patching hosts was just rotating to a new AMI on AWS; patching applications was updating dependencies.

No kidding. I would have preferred to migrate the client to GitHub, but they were convinced that wasn't acceptable - had to run their own GitLab. Turned into a mess of managing Gitlab updates and BS rather than working on the product.

Reply on news.ycombinator.com