t_sawyer

I moved to a Hetzner dedicated instance with cloudflare in front. Rails Postgres memcached all run on docker.

Vs

Heroku with cloudflare in front Postgres from heroku and memcached with an add on.

1. Very little. I put the app in docker a long time ago for local development but still used heroku for prod. I dumped my db and imported it into a docker run db on Hetzner. One thing to note: I did not expose the db to the internet. The app connects to the db over a docker network.

2. I do DevOps for a living.

3. It did not add to my learning.

4. $30/month for a large Hetzner dedicated instance and $12 for S3 (backups) vs $25 for the app (I added 2 more for 8 hours a day during busy season) and $25 for a job runner and whatever they were charging for DB. My total cost now is $42/month and on Heroku it was around $120 during busy season and $100 during non busy season.

5. Rails Postgres memcached and delayed job. I had to expand my app tier during busy season by up to 3 instances (winter). With my load testing the Hetzner instance handles much more than the 3 instances ever did in my load testing.

6. I added db dump cron jobs along with restic backups to S3.

There a few very important things to consider if you’re being asked to be on call: 1. Expected response time 2. Number of times called on average 3. Average time spent working to fix a problem.

#1 and #2 are pretty obvious. These things actually eat into your life because you’re actually working.

#3 is hard to calculate how much that’s worth. If I’m required to be logged into vpn and starting to dig in within let’s say 10 minutes. That means I cannot reasonably leave my house to: eat dinner with my family, go to lowes to grab some pvc because the pipe for my sump pump started leaking, walk my dogs, take my kids outside to teach them how to ride a bike, etc.

I feel I should be compensated for having to be ready to go and not having the freedom to live my life. That in itself is an interruption.

I posted this on the Reddit thread too but:

Yeah I was through 4 or 5 “steps” before salary was discussed. And I ultimately think that’s why they didn’t move forward.

Lesson learned. I wasted quite a bit of applying, doing an initial interview, a second interview with a teammate, doing weird aptitude tests that had me comparing images, a personality test, a coding test while a manager watched, and finally talking to an hr person about salary requirements.

I posted a similar worry about the ENV's.

Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.

I'm interested if application ENVs were stolen. If they stored Github APIs in plain text I'm sure our ENVs were too.

I went ahead and reset my planetscale passwords and moved my app out of Heroku. But I only had two apps hosted there I feel for anyone with a large number of apps on there.

Um… I pay for screens. As long as my account is within that screen limit it shouldn’t matter who is using it.

I see the function of SRE as a place to pass off an app when an app team (DevOps included) need to develop another app.

SRE sets standards so a centralized team can manage the ongoing maintenance and on call break/fix of many apps in an organization while development teams move on to something else.

DevOps should be embedded on the team. Doing development and taking care of pipelines/iac when needed. If these people are not embedded on the team, it’s just more of the same with a new title.

I get paid a lot because I can write api endpoints, front end code, iac, pipelines, and understand networking enough to make our apps SOC2 or HIPAA compliant.

There’s a ton of people who dip or use chewing tobacco… no second hand smoke there.

Do you support cannabis being illegal? This is a pretty similar stance. Personally, I don’t think tobacco should be illegal. Do I partake? No. But I certainly don’t want to jail people for partaking.

Vault is only replaceable in cloud. Idk of any on-prem products that have anywhere near Vaults functionality.

Yes, ingress like Traefik can auto renew certs.

My problem with swarm is that the overlay networking was super flaky. Dead containers still being registered on the network cause every X requests to fail (since it’s round robin) because it’s routing to an overlay IP that doesn’t exist.

Further, I could never find storage adapters for cloud storage that continued to be maintained. Rexray was the longest standing but hasn’t had a commit since Jan 2019.

Except... gymnastics won't end up being dominated by a bunch of people like Simone Biles.

Female sports will continue to be dominated by biological males if we continue to let this happen.

They don't need to eat a slice of Heroku's pie. Heroku pays AWS for each of their customers already. Their infrastructure runs on top of AWS.

You think 10% of a towns population will die from COVID in a couple weeks?

Has that percentage of people in any county in the USA actually died from COVID in over a year?

Florida who has been pretty laissez-faire about COVID has 5,980 deaths in Miami-Dade County (their highest amount of deaths in a county). 2.717M population in 2019.

0.22% of their population.

How do they charge egress for this?

Idk why this take isn’t more popular.

Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem.

Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.

iDrac and similar devices can provide “physical” access without physical access.

People have been using them in collocations for years. Collocations also provide hardware swap services. They can replace a power supply for you.

Same. My only option here is spectrum. 940/40 is what I pay for.

I also watch my bill like a hawk and I run a Speedtest exporter with Prometheus and grafana to make sure I’m semi getting what I pay for.

Lol but they’ve got no problem with Spotify hosting Eminem songs since it’s inception? This is ridiculous.

His point is why the internet has a single point of failure -> Cloudflare. They offer DDoS protection for free.

YouTube existed without Google. Google bought them.

You responded to this better than I did.

I participated in my local explorers program because my mom was a dispatcher. Go on enough ride alongs to see that 90% of the job is dealing with domestic disputes or drunk and disorderly people and you’ll never want to be an officer

I answered that question. It’s a separate job to investigate.

Regular cops are first responders not investigators.

90% of the job is dealing with drunk and disorderly people or descalating domestic disputes. That knocks a lot of High IQ/EQ people off the list.

My mother was a dispatcher for 15 years. I was in our local areas explorers program. It’s exactly what you see on Live PD. Traffic stops, drunk and disorderly people, drug busts, or domestic disputes. Cops do almost no detective work. Investigators do that.

I’m a huge K8s advocate but if all you’re looking for his a repeatable way to spin up DO droplets then use a user-data script.

User data is a bash script that can be automatically run when the machine first spins up.

You could pass that script via digital oceans cli or even a tool like terraform.

This just isn’t true. I’ve never used Tensorflow but Kubernetes is great.

We moved to it from docker swarm because docker swarm still has a lot of glitches with its overlay network. Rolling upgrades would leave stale network entries and its impossible to reproduce. Sometimes it happens sometimes it doesn’t.

With a managed solution, Kubeadm, or RKE it’s not hard to deploy anymore. All our infrastructure is in code, is immutable, and if you’re careful can be deployed into any kubernetes cluster.

Just like Docker has been great for easily deploying open source products, kubernetes is great for doing the same thing when you need to deploy horizontally. It’s easy for OSS to provide a docker image, a docker compose file for single node deploy, and Kubernetes yaml for a horizontal deploy.

Yes. DDG actually pulls search results from Bing and a few other searches. They don’t actually scrape the web.

My only experience with RabbitMQ is managing an Openstack environment. In that environment, it's a huge resource hog and we had to put it on 3 separate bare metal instances to keep it stable.