Hacker News Reader

Comment by moreira | Hacker News Reader

I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.

You were able to do it, but what happens to them when you're not around? Does their team have the required experience to handle it? That's the difference in cost. It's like DYI - yes, if I have all the skills and experience I can do everything myself incredibly cheaply, but... I don't. So I gotta pay.

But if your team does have the skills and experience, it's definitely worth looking into.

I do think people deeply underestimate what can be achieved with a single (or two, if you need a standby) dedicated Linux server these days. A single server can easily go up to 2TB of RAM and 128+ cores. Long before you ever get to a scale where that's a limitation, you'll have more than enough resources to figure things out.

View on HN

Funnily enough, small amount of servers that you want to utilize as much as possible is pretty much one of the original use cases for kubernetes.

My first deployment involved virtual machines, but we were essentially packing as much as possible into lowest amount of VMs, then doubled it up so that we had failover. This way we had clear visibility of how many resources we were using and could allocate them according to how much we had.

You were able to do it, but what happens to them when you're not around?

This is why you pay someone. Saving $950/month means it's well worth spending $500 for a day of someone's time occasionally. You don't have to do everything internally when you run a startup. Buying in services that you only need occasionally is well worth the money.

Are there contractors out there that will take on call shifts? Because it seems unlikely, and if your proposal is "put into a production a system that you will have to spend $500/day every time it goes down and wait 2-4 business days for a resolution" then you're a braver person than I am.

Obviously not. You don't pay someone to just set it up. You pay to help do what you'd do if you had a dedicated devOps teams. You pay someone to set up the system with your team so they understand it, train your team to use it, write some documentation about it, script a rollback procedure, maybe help on developing playbooks, etc.

Besides, there are people out there who offer on-call services for a small retainer.

This is optimistic to say the least. I've worked as an SRE for 5 years and apart from the others in the team the devs don't have nearly as much knowledge. There's no way I'd rely on them to fix an outage.

And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.

The idea is devops as _culture_ - I come in as an expert and set it up, then show them how I did it, then run through various disaster recovery scenarios so they learn it and can handle the vast majority of problems.

And you'd be surprised how little problems you might have - I've had many VMs with literally years of uptime running without issue.

Most people focus on "devops" as a job - and they never bother to teach the rest of the team anything about how stuff works. Worse than that, modern clouds encourage you to build opaque and complicated systems which probably only someone working on them full-time has a hope to understand...

If it was so easy for devs to pick up SRE companies wouldn't be struggling to find good people.

Culture doesn't mean they'll know how to fix <some weird edge case> at 3am in the morning. The SRE with constant ops exposure is likely to have a much better chance, if only because they (ought to) really know how to debug it.

> I've had many VMs with literally years of uptime running without issue.

I hope they all have fully patched base libraries and kernels, security auditing is getting to a much more common requirement these days even among very small companies. For example, anybody using Facebook Login.

And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.

This is why we document things at the company I work for. If you're serious about a project and want it to exist for a long time there will be things that only come up every few years, and you won't remember them. You can either write things down or you have to relearn how things work every time. Wroting things down is much easier.

I'm unconvinced learning what I did as a 14yo is more complex than modern devops.

> I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.

But this applies to everything. You also need Heroku or Kubernetes or whatever experience to maintain those systems, right?

The question is how much you need to know. You need much less knowledge to run something in Heroku than in k8s; dramatically less in the "onboard a CRUD app" case. I'd argue that running k8s effectively is no less knowledge-intensive than running VMs.

> I'd argue that running k8s effectively is no less knowledge-intensive than running VMs.

LOL - are you f'ing kidding?

Running VMs is much closer to running your app locally than containers and k8s - it's not that hard to do badly, and only marginally harder to do well.

Having done both, not kidding in the slightest.

> I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.

Exactly - $950/mo is nowhere near enough to pay for those skills if you don't have them. It's good value for money.

It's all scripted - so from bare ubuntu install to functioning environment and checked into git. So, it serves as documentation as well.

So disaster recovery is spin a new box, pull the most recent db backup in, and update dns - all scripted should I be gone.

I say it's over-rated. You don't need a huge amount of "sysadmin/devops", that's only now becoming a thing since we started calling it that. Before it used to be that backend devs just had intimate knowledge of how their service was running on a system, and most likely had to login and debug a multitude of issues with it. 99% of backend devs used to (maybe not so anymore now with "devops" et al) be more than capable of administering a system and keeping it chugging along as well as setting it up. It might not be 100% bulletproof or consistent or whatever, but more than enough for a company or service starting up.

We've lost that, and now everyone thinks we need "sysadmin/devops" for simple and moderately complicated deploys. Heck, most of the guides are out there, follow them. Also, ready-made images and docker containers are amazing these days with config built in. If you look-for or hire devops, you get K8s and all the "formal" items they'll bring with them. You don't need CI/CD for the first 6 months, just PoC the damn thing, get an MVP out and start getting users/customers, the rest will follow.

How long ago is your "before"? Because the pattern of "hand your jar file to the sysadmin to deploy" goes back almost as far as jar files do.

Reply on news.ycombinator.com