Running servers and services well is not trivial (2018)

Good enough according to whom? Inevitably there's always some unpatched server somewhere on the network someone forgot about that let hackers access data for millions of customers.

Existence doesn't mean much in times of quantitative easing; pick your favorite punching bag company in recent memory that is still extant. If you don't understand the economic situation backing all this it's hard to discuss your rebuttal further so we need to checkpoint there before I invest any time on more responses.

The issue is largely orthogonal to my own skillset because one needs to only view quality regressions in order to assert my point, not be the cause or hero of them (ain't philosophy generous?!). The times a 2 year experience system operator could go in and wreck havoc on a mainframe operation were not particularly common and the standards and practices (i.e. airgapped backups on tape, geographically dispersed sysplex etc) ensured business continuity were generally planned right in from the initial purchase order with the vendors' help once upon a time. In this decade people can and did launch massive financial exchanges on mongodb when it had serve and well known issues.

I would love to. See Dan Luu's big company pay statistics. Want to fund me to ignore that? I don't think anyone would (aka bubble, value creation detached from valuation). I'll just save what I can, try not to rock the boat too much for the people underwriting my salary in any direction good or bad, and hope to GTFO before complete burnout.

This is getting harder than it was even 5 years ago if you don't have decent mail volume

I also use to manage exim4/spamd long ago. Part of the problem I remember is that your email (sent from your server) can be marked as spam on someone else's inbox (e.g. you are blacklisted without knowing it). So while you haven't had issues on your side, it's hard to prove that you didn't have issues outside your box. This was especially common with VPS datacentres, as their IPs routinely showed up on blacklists.

Our project was greenfield 2.5 years ago, and in evaluating the ecosystem we ran into the work going on w Habitat after trying to figure out if there was a VM-analog to the stuff going on with k8s and containers (which was a very scary prospect to dive into.) Habitat seems to be a good tech stack for solving a variety of operational problems that you typically pull different products off the shelf and duct tape together -- particularly if you are focused on VMs.

It was a bit of a bumpy ride but the product has stabilized enough we have confidence in it and it turns out to be quite a good fit for the self hosting use-case. Basically, it's an all-in-one solution for packaging, isolation, service discovery, configuration management, and deploys. So it was relatively easy to transition our production deployment bits to a packaged up deployed solution to 3rd party servers, with all of the bells and whistles.

Our setup is a fairly minimal custom AMI that just installs the basic package dependencies for Habitat and configures the supervisor. Everything else is bootstrapped within Habitat, including our own proprietary configuration management service which allows a nice web-based GUI for configuring the ring state. The user data script just does some basic UNIX setup and DNS initialization and then does a bunch of Habitat service loads. By abstracting over all configuration via Habitat, a unified interface exists for configuring all the services across all the machines, regardless of their underlying configuration management approach, programming language, conventions, etc.

In this case, I'm not sure if it's abstraction-insanity as much as having common solutions to common problems. You know what would be an insanity of abstraction? Rolling all that yourself.

Given that there are better and worse abstractions, most are not suitable for most use-cases, a lot of times simpler and more centralized is actually more stable. But when you start needing high-availability-coordination you're going to be damn happy that there are things such as Consul, etcd and Zookeeper.

As for Salt; by all means, start off by having bash scripts in source control, which at some points becomes a pain-point and you'll realize you need some kind of configuration management to not go insane.

Several of these tools are just putting a framework and common language for tasks we have to do either way.

I think the problems are mostly around configuration that is so complex and redundant that we need programmatic abstractions to deal with it—we need something like a general purpose programming language to DRY up and simplify that configuration, but the powers that be are clinging desperately to YAML (despite encoding a shitty, half-baked AST on top of it a la CloudFormation or “generating” YAML with text templates a la Helm) as the human/configuration interface presumably because the “it’s as easy as YAML!” is such a good marketing schtick.

A certain amount of configuration complexity will always be there, but there’s still a lot of incidental complexity that could be cut away if we just generated these config a with a general purpose language.

It doesn't feel like we've come very far at all from the first time I realized I could use nginx as a forward proxy and have the code just make all service calls to localhost. That seriously reduced the complexity for the development team.

Some days I just want a load balancer that can route traffic to whatever servers are up now and then I'd just have Docker, a service registry, a load balancer and I'd be done for a while. The easy bits should be easy, and they can't even manage that.

Isn't that the point of the article? That infrastructure actually isn't easy, and you're just hating the fact that it isn't?

Isn't that sort of like someone in Sales complaining about the "state of development" and asking why they can't just press a button and have a custom app to sell, like, tomorrow?

Welcome to dev vs infrastructure vs ops vs devops.

DC/OS is dead, Jim. Well, kinda. Mesosphere changed their name to D2IQ and is now supporting kubernetes.

You can, however install the banzaicloud kafka operator with only a few commands and very little config and yield a production-grade kafka cluster pretty quickly.

https://github.com/banzaicloud/kafka-operator

Typically you use both, github for the git repo and Heroku for automatically deploying from github.

Good point, I didn't necessarily mean simpler primitives but rather "higher-order" primitives. We need to move to a higher level of abstraction on top of a new layer of primitives (even if those primitives themselves are more complex than the ones we have today).

I agree! But isn't that the beauty in abstractions? If you are using an unmaintained file system with a CVE, the beauty of the POSIX interface means you can just migrate to another file systems and basically all applications would just continue working. I'm not advocating for having nobody at the wheel to notice and address security vulnerabilities, just that those concerns should be isolated from the application code to a much greater extent than they are today.

> I don't think this is accidental complexity, it's essential complexity.

I think this is the kind of essential complexity which we faced when developing operating systems. OSes are fundamentally helpers, which don't solve the application problem, but make the solving easier (a good OS makes it easier by much).

So we can and should use the results which we got from OS development.

I agree that the parts are needed and the complexity essential. What I mean is that we need higher-order primitives that abstract away and shield us from the conplexity. Just like we don't need to spend time crafting TCP packets by hand, I don't think we should have to mess with transport protocols, writing RPC calls by hand, juggling yaml files to stand up Kubernetes clusters, and so on.

I've been working on building something like this for a year now [1] and it's certainly quite different from how we do backend development normally,but that's also the point :). The tricky part is to find abstractions that are general and non-leaky so they can be leveraged to build a wide variety of software. Feedback appreciated!

[1] https://encore.dev

> can you use the common languages - and if not, why?

Good question. I do not know the answer to that. But I think there must be some answer since there are all these different configuration languages, Ant, Gradle etc. They must have been created for a reason.

Yes, the Nix ecosystem makes running both servers and services much easier than any other tools I have used.

I describe the deployment I run with it here: https://news.ycombinator.com/item?id=21468506

Nix is not typed (it would be slightly better if it were), but everything is evaluated before it hits your servers which allows for lots of static checks.

I started writing a tutorial on NixOps here if you want to learn it: https://github.com/nh2/nixops-tutorial (only has 1 part so far, I'd like to show how to bootstrap a Consul cluster and distributed file system next).

This is a fair question if you’re lucky enough not to be familiar with devops tooling.

No, it’s not Go. Go is a general purpose programming language. What parent poster wants is more like Terraform, but better (I assume). With semantics that reflect devops actions.

You might write that tool in Go :) but the users wouldn’t necessarily know. Just like a gamer doesn’t need to know about C++.

I get a feeling about this as the one described in the article about git servers. Yes, static site generators make it easy to post new content written in plain text, markdown, and other markup languages. However, it's the initial configuration that can be tough. I spent a year playing around with Hugo. Along the way, I learned a little bit about HTML, CSS, SCSS, go-template language, shortcodes, configuration through toml, toml/yaml front matter, reStructuredText, markdown, and git submodules. When my blog was Jekyll-based, I had to maintain gems that were compatible with the version of Ruby & Jekyll used by GitHub.

None of this stuff is easy when you're starting out. Even when you've become used to whatever environment you've created, there are a lot of moving parts to monitor and manage.

It is quite likely your blog is serving spam in the best case and warez in worst

But that doesn't help if I want that 2FA key to access their mail, Gitlab instance, AWS servers, etc. What do I do when someone leaves and I want to be able to revoke it and take over their repositories and accounts?

If I'm a 1000 person company, I've got resources to spend on my corporate 2FA infrastructure.

What do I do when it's 10 people? 20 people? 50 people?