chucky_z

I've gotten servers from Hivelocity in the past, they like(d) to offer servers with the "enterprise" equivalent of these CPUs, I'm not sure if Intel still sells them, but there used to be a "xeon i3" basically, usually 4 core CPUs clocked to the moon, that supported ECC. They were really good!

Influx is so good for hobby-level projects. So good!!

For serious applications, it doesn't cut it. Trying to do more than a few TB a day is a waste of time outside of enterprise, which ain't cheap.

I plopped VictoriaMetrics in place of Influx for my cases and haven't even had a single hiccup.

this is a pretty common use-case for ci/cd tools.

azure devops, jenkins, concourse, drone, gitlab, ...

also a lot of etl tooling has this stuff (airflow comes to mind immediately)

there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability.

there's... a surprising amount of tooling in this exact space!

Symfony

if your company has a position called 'devops engineer,' they've failed as an entire company to resemble anything called devops.

It's really slick, and provides a client/server architecture with a bunch of extra security on-top. It generates 1:1 certificates between the api layer and the clients, using a full auth service.

I deployed it at a previous position and stopped SSH across all hosts entirely, and just used this.

It rotated all it's own certificates every 4 hours, and each SSH connection used a unique certificate.

I never had any issues with it, and I had the web UI straight up exposed to the internet. It enforces TOTP and user management with the OSS stuff is pretty trivial.

It also did something really cool, which is that I had multiple bastions federated together through a single API, instead of having singular multiple bastions. This meant no configuration to access hosts between prod/stage/test, all a single entry point, it was awesome.

I don't use it today in my current position for a number of reasons, but if I were to go back to a smaller company and had the choice, I would deploy it with no hesitation.

Literally anything. If I see someone with a lot of relevant forked repos, even if they're old, I take that as interest and something I can bring up.

If I see a repo of rcfiles I know they care about working efficiently. If I see abandoned stuff with more than 1 commit that's OK, that's something that was cared about at one point. These are just two super generic examples. Almost everything is a positive.

The only thing I don't like to see is repos with 1 commit, and nothing other than a README with the repo title in it. Not really negative, more of a 'cmon gimme more.'

FWIW, as a hiring manager, if someone has a portfolio I definitely judge them on it, and if it's good it allows me to bypass huge swaths of technical/coding interview stuff and dig much deeper into where/what I want. I always take it as a positive, even if it's old stuff.

The fact that, as a user, I can't tell a browser "hey, this internal PKI that you already don't trust, just go ahead and trust it no matter what because I tell you to," feels bad.

I’m so torn here. Personally I like this a lot and think it will really help enforce good practices and allow easier things like root/int key rotation. Professionally it sucks, as there are a ton of valid use cases for real certs in areas that require manual work and tracking them all is a hard problem. If internal PKIs were easier to make work across all OS and Browser combos I’d just use those instead.

IIRC, a 12oz coffee is ~$2.

The equivalent 12oz coffee milkshake is ~$4.

Guess which one is more popular?

That said, I do actually buy the non-sugary drinks because they're cheaper. I tend to buy Blonde Americanos to get the lowest bitterness espresso flavor from them, at the lowest price.

I get a bit frustrated by this view solely because internally, folks aren't _just_ your customer. They are your co-workers, and the same thing for them. Their co-workers are providing the platform.

The extra bit of empathy makes all the difference, because without fantastic personal communication, 'platform' could be a waste of time for everyone involved.

Because it’s cool! Also it’s extremely open source, has a cool tech blog, and feels like it has a lot of momentum

Awesome! For anyone who finds this through search, here's that whitepaper: https://dl.acm.org/doi/pdf/10.1145/3342195.3387524

I don't think it helps at a server level, but it does provide me some ideas, especially looking at some of the algorithms they're using.

My problems are mostly CPU related, whereas this paper seems to focus on Memory. I still can definitely get something useful from here. :)

The latter. We have a lot of metrics and data, but I need something consistent beyond "this box looks like its using a lot of CPU/memory/IO."

Our workloads are dynamic enough that a box using 20% or 80% are both within range of normal, with some single threaded workloads so per-core stats are also not great.

I'm also not looking to fix anything, just have the ability to say, with a high degree of confidence, "hey, our system detected this box looks funky, can you just pull it out and replace it?"

Please make an attempt to understand a technology before making comments like this. Nomad has no requirement to store kv's, nor secrets. It's an entirely separate thing from Consul and Vault. It simply integrates with Consul, and Vault. It even runs its own raft store, so each product's backend is totally separate (Vault dropped it's reliance on other backends as of 1.4 and can run it's own raft store now).

Nomad template stanzas are simply consul-template (https://github.com/hashicorp/consul-template#plugins), you could use `{{ with secret "" }}` and never touch Consul. You also have every function beyond service and the key* set of functions. You could build pretty static, or dynamic configurations using these blocks without ever touching Consul. On top of that, both Terraform templates and Levant work well for templating job specs out themselves, which contain template stanzas in them.

An example of something helpful would be if you wanted to drop a very small binary that changes with each deploy, you could use https://github.com/hashicorp/consul-template#base64decode and just change the contents each time you deploy the job.

If you wanted to use redis keys in your consul-template, simply drop a binary on each server as a plugin to consul-template, then for example: `{{ "my-key" | plugin "redis" "get" }}`.

Why not try running it yourself?

`nomad agent -dev` will get you a server running, then `nomad job init` will give you a full spec which you can run with `nomad job run example.job`.

This is what Nomad is though... It works without their other products and also natively integrates into them. Deploying Vault, Consul, and Nomad gives you a very nice experience.

Also with Consul 1.8 and Nomad 0.11 you’ll get Consul Connect with Ingress gateways which solves some of those problems you mentioned.

There's been a ton of improvements over time. That doesn't make it any less foot-gun-y, unfortunately.

I had the exact same problem with Kubernetes (and even just straight up containers), I just want to make sure folks are extremely aware of how big of a footgun it is, and to really, really, really test it well.

Yes, you know that small organization, Linux; who manages cgroups that everything else calls through with.

I have only tested it with Docker, again, that company Moby with this very little used software.

How dare their core systems have any systems that are not extremely simple to understand and have caveats.

Preemption is coming in the next release in OSS :)

Cloudflare, make sure you upgrade to 0.11.3, the new scheduling behavior is awesome for large clusters.

Also a massive warning to anyone wanting to use hard cpu limits and cgroups (they do work in nomad it’s just not trivial), they don’t work like anyone expects and need to be heavily tested.

Oh, interesting. What's the core difference between Tickscript and Flux? Was Tickscript what Kapacitor used for InfluxDB 1.7?

Vault is great for just a PKI, even if you aren't using it for anything else. There are some tools that just do PKI, but Vault works a real treat at it. Any Terraform backend that supports encryption + Terraform + Vault gives you such an amazing workflow. We use a mix of short and long certs, with different roles based on what's getting a cert.

For now, we have CRLs disabled on all short-lived backends, enabled on long-lived backends and we're actually looking at disabling storing short-lived certs in the storage system at all, and just cranking the TTL down to really truly short. We've tested it as low as 30m, but a more real-world max-ttl is 1 week, with individual apps setting it as low as they can handle. For reference we run more than 10 PKI backends, and adding one (or a bunch) more is just a little terraform snippet for us.

The way it works via hashicorp template land, is that you just plop

    {{ with secret "name-of-pki/issue/name-of-role" "common_name=my.allowed.fqdn" "ttl=24h" }} {{ .Data.certificate }} {{ end }}

into your Nomad template stanza, or use consul-template directly as a binary, or use vault agent with it's template capability. You can get the CA chain if required the same way, just hitting a different PKI endpoint.

Also, as of Vault 1.4, Vault's internal raft backend is now production ready, making it a snap to run.

Try running through a few of the Vault quick-start guides, and replicating them in Terraform as much as possible. There's a few things TF does not handle gracefully last I checked (initial bootstrap), but you can get around that by using a null_resource or just handling that outside Terraform.

I'd like to followup on this and say InfluxDB is fantastic -- with the caveat that you know exactly what you want, and use it with that in mind. It's great at storing some metrics with high volume, but I've found the OSS version falls over hard in some circumstances (high cardinality in tags absolutely murders performance, making it nearly unusable).

I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus.

Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.

Nomad + Consul is the happy path. Adding Vault into the mix is nice, but not required.

Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way.

"How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that.

Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community.

If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task.

My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.

Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?

It makes using org-level AppRoles with Vault really easy, but I wouldn't replace Vault with this. I use org contexts with CircleCI today to do this with a very generic AppRole that has access too 'secrets that should be in Vault, but are generally safe.' e.g.: a consul credential that can read/write from the terraform state area.

fasthttp is hilariously fast. it's also missing a ton of edge cases over net/http, but yea, it's very silly fast. i'd say just try it and see for your use-case, but expect 2-3x speedup over standard net/http based frameworks at a minimum.

So you went from full automation to... Copying and pasting by hand?

SQL for Smarties is the book that has proven, without a doubt, I am bad at SQL. https://www.amazon.com/Joe-Celkos-SQL-Smarties-Programming/d...

If you want to get undeniably good with SQL, this is the book.