Does your startup need complex cloud infrastructure?
hadijaveed.me
hadijaveed.me
I currently have distilled, compact Puppet code to create a hardened VM of any size on any provider that can run one more more Docker services or run directly a python backend, or serve static files. With this I create a service on a Hetzner VM in 5 minutes whether the VM has 2 cores or 48 cores and control the configuration in source controlled manifests while monitoring configuration compliance with a custom Naemon plugin. A perfectly reproducible process. The startups kids are meanwhile doing snowflakes in the cloud spending many KEUR per month to have something that is worse than what devops pioneers were able to do in 2017. And the stakeholders are paying for this ship.
I wrote a more structured opinion piece about this, called The Emperor's New clouds:
If you use it as IaaS, it's a lot quicker to get prototypes working than if you use anything else, including VPS's from other providers.
Google Cloud in particular has very few vectors for lock-in, and follows more principle of least surprise.
But once you have prototyped, you should ask the question about rebuilding it somewhere that is cheaper.
Near infinite scalability of disk drives is nice, and snapshotting, and cloud in general can allow you to extend your prototype into taking production load and allowing you to measure what you will need; but leaning in to "cloud magick" (cloud run, lambdas, etc) will consume almost as much time to learn and debug as just doing it the old school way anyway. In my lived experience.
The biggest problem is the so called cloud native stuff which is both more expensive and more complex. There are contexts where it makes sense but for startups they are doing more harm than good.
Still, my most benevolent interpretation of current reality is, rather than saying "that cloud native stuff is crap", accepting that there are cases where it may make sense.
For instance, large companies might have trouble hiring a good ops team because they have in general trouble hiring and retaining talent (another conversation topic).
Ops people are a scarce good because univs do not train people for that and most people prefer coding. I am leaving the work devops out because the market completely perverted its meaning.
(my take on the devops funeral: https://logical.li/blog/devops/ )
https://survey.stackoverflow.co/2022/#developer-profile-deve...
Only around 11% of the whole devs identify as devops specialist or cloud infrastructure engineer.
This is why I am saying ops people are a scarce good (unfortunately) from a data driven perspective. Of course my daily life confirms it.
The connection of this to the replies above it: I am not sure if this kind of junior people would be easy to retain in a large corporate environment. We certainly can do that in niche consulting.
While the VMs that run some parts of the system have been rock solid giving zero issues... Should have just thrown the stuff on one of them or added third one. Cost would have been same.
You mentioned Python backend, so literally just replicate build script, directly in VPS: "pip install requirements.txt" > python main.py" > nano /etc/systemd/system/myservice.service > systemd start myservice > Tada.
You can scale instances by just throwing those commands in a bash script (build_my_app.sh) = You're new dockerfile...install on any server in xx-xxx seconds.
For Python backends I often deploy the code directly with a Puppet resource called VcsRepo which basically places a certain tag of a certain repo on a certain filesystem location. And I also package the systemd scripts for easy start/stop/restart. You can do this with other config management tools, via bash or by hand, depending on how many systems you manage.
What bothers me with your question is Pip :-) But perhaps that is off topic...?
Will not run on FreeBSD, for example.
s/host OS independence/a certain level of host OS independence
And getting containers to run depends on the OS - if you don't control the host, leads to major ping-pongs.
Even within Linux (Ubuntu, Debian, RHEL, etc) when you are distributing multiple related containers there are details to care about, not about the container itself but about the base OS configuration. It's not magic.
No, you're tied to operating systems using a Linux kernel that supports the features necessary for running images.
You can -> Podmaaan
https://podman.io/docs/installation#installing-on-freebsd-14...
ATM experimental
Not true:
https://podman.io/docs/installation#installing-on-freebsd-14...
ATM experimental
Oh, and not only build their app, they can take it a step further and setup the entire new vps and app building in one simple script!
I split it into multiple scripts that get called from one, just for my own sanity.
I use Puppet for distribution of users, firewall rules, SSH hardening + whitelisting, nginx config (rev proxy, static server, etc), Let's Encrypt certs management + renewal + distribution, PostgreSQL config, etc.
The profit from this is huge once you have say 20-30 machines instead of 2-3, user lifecycle in the team that needs to be managed, etc. But the time investment is not trivial - for a couple of machines it is not worth it.
But does that matter to security researchers or the public? No. Apple needs to get their bounty program in order regardless of the reason it's broken.
Ultimately, this blog post is just another example on the already large pile[1][2][3][4][5]
1: https://arstechnica.com/information-technology/2021/09/three...
2: https://mjtsai.com/blog/2021/07/13/more-trouble-with-the-app...
3: https://medium.com/macoclock/apple-security-bounty-a-persona...
4: https://theevilbit.github.io/posts/experiences_with_asb/
5: https://shail-official.medium.com/accessing-apples-internal-...
What do you find frustrating about cloud-init? I'm relatively new to it.
I feel that about the cloud as a whole, tbf. It's incredibly painful to me.
This isn't bad for a small team, but it becomes increasingly painful as you scale, but it's really hard to make it work smoothly for bigger teams (the sysadmin team becomes a bottleneck for everyone's deployment, deployments slow to a crawl so everyone builds these enormous, buggy releases, testing becomes a once-a-month thing instead of a continuous thing, etc). And the teams that do it well basically end up reinventing a big chunk of the cloud without any of the benefits of a standard, well-documented, widely-understood cloud platform anyway.
I deploy to pared down bare metal, but I use containerization for development, both local and otherwise, for me and contributors.
So much easier than trying to get a local machine to be set up identically to a myriad of servers running multiple projects with their idiosyncratic needs.
I like developing on my Qubes daily driver so I can easily spin up a server imitating vm, but if I’m getting your help, especially without paying you, then I want development for you to be as seamless as possible whatever your personal preferred setup.
I feel containerization helps with that.
Pipy is an informal source of software that has low security levels and was infested with malware many times over the years. It does not provide security updates: it provides updates that might include security-related changes as well as functional changes. Whenever you update a package from there, there is a chain reaction of dependency updates that insert untested code in your product.
Due to this, I prefer to target an LTS platform (Ubuntu LTS, Debian, RHEL...) and adapt to whatever python environment exists there, enjoying the fact that I can blindly update a package due to security (ex: Django) without worrying that it will be a new version which could break my app. *
Furthermore, with Ubuntu I can get a formal contract with Canonical without changing anything on my setup, and with RHEL it comes built-in with the subscription. Last time I checked Canonical's security team was around 30pax (whereas Pipy recently hired their first security engineer). These things provide supply-chain peace of mind to whoever consumes the software, not only to who maintains it.
I really need to write an article about this.
* exceptions apply, context is king
There's tons of examples, you are learning a durable skill, and 90% of the time (for personal stuff), I had to ask myself: would I really ever deploy this on something that wasn't Debian?
Boom: debian-lts + my_package-0.3.20240913
...the package itself doesn't have to be "good" or "portable", just install it, do your junk, and you don't have to worry about any complexity coming from ansible or puppet or docker.
However: docker is also super nice! FROM debian:latest ; RUN dpkg -i my_package-*.deb
...it's nearly transparent management.
1. Not needing to run my own PPA server (not super hard, it's just a little more friction than using Docker hub or github or whatever)
2. Figuring out how to make a deb package is almost always harder in practice for real world code than building/pushing a Docker container image
3. I really hate reading/writing/maintaining systemd units. I know most of the time you can just copy/paste boilerplate from the Internet or look up the docs in the man pages. Not the end of the world, just another pain point that doesn't exist in Docker.
4. The Docker tooling is sooooo much better than the systemd/debian ecosystem. `docker logs <container>` is so much better than `sudo journalctl --no-pager --reverse --unit <systemd-unit>.service`. It often feels like Linux tools pick silly defaults or otherwise go out of their way to have a counterintuitive UI (I have _plenty_ of criticism for Docker's UI as well, but it's still better than systemd IMHO). This is the biggest issue for me--Docker doesn't make me spend so much time reading man pages or managing bash aliases, and for me that's worth its weight in gold.
Really, my package is currently mostly: `Depends: git, jq, curl, vim, moreutils, etc...` (ie: my per-user "typically installed software"), and I'm considering splitting out: `personal-cli`, `personal-gui` (eg: Inkscape, vlc, handbrake, etc...), and am about to have to dive in to systemd stuff for `personal-server`, which will do all the caddy, https, and probably cgi-bin support (mostly little home automation scripts / services).
I'm 100% with you w.r.t. the sudo journalctl garbage, but if you poke at cockpit https://www.redhat.com/sysadmin/intro-cockpit - it provides a nice little GUI which does a bunch of the systemd "stuff". That's kindof the nice tag-along ecosystem effects of "just be a package".
I'm definitely relatively happy with docker overall, but there's useful bits in being more closely integrated with the overall package system management (apt install ; apt upgrade ; systemctl restart ; versions, etc...), and the complexity that you learn is durable and consistent across the system.
> 1. Not needing to run my own PPA server (not super hard, it's just a little more friction than using Docker hub or github or whatever)
Docker actually has more infrastructure requirements than alternatives. For instance, we have some CI jobs at work whose environments are provided via Nix and some whose environments are provided by Docker. The Docker-based jobs all require management of some kind of repository infrastructure (usually an ECR). The Nix-based jobs just... don't. We don't run our own cache for Nix artifacts, and Nix doesn't care: what it can find in the public caches we use, it does, and it just silently and transparently builds whatwver else it needs (our custom packages) from source. They get built just once on each runner and then are reused across all jobs.
> 2. Figuring out how to make a deb package is almost always harder in practice for real world code than building/pushing a Docker container image
Definitely depends on the codebase, but sure, packaging usually involves adhering to some kind of discipline and conventions whereas Docker lets you splat files onto a disk image via any manual hack that strikes your fancy. But if you don't care about your OCI images being shit, you might likewise not care about your DEB packages being shit. If that's the case, you can often shit out a DEB file via something like fpm with very little effort.
> 3. I really hate reading/writing/maintaining systemd units. I know most of the time you can just copy/paste boilerplate from the Internet or look up the docs in the man pages. Not the end of the world, just another pain point that doesn't exist in Docker.
> 4. The Docker tooling is sooooo much better than the systemd/debian ecosystem. `docker logs <container>` is so much better than `sudo journalctl --no-pager --reverse --unit <systemd-unit>.service`. It often feels like Linux tools pick silly defaults or otherwise go out of their way to have a counterintuitive UI (I have _plenty_ of criticism for Docker's UI as well, but it's still better than systemd IMHO). This is the biggest issue for me--Docker doesn't make me spend so much time reading man pages or managing bash aliases, and for me that's worth its weight in gold.
I don't really understand this preference; I guess we just disagree here. Systemd has been around for like a decade and a half now, and ubiquitous for most of that time. The kind of usage you're talking about is extremely well documented and pretty simple. Why would I want a separate, additional interface for managing services and logs when the systemd stuff is something I already have to know to administer the system anyway? I also frequently use systemd features that Docker just doesn't have, like automatic filesystem mounts (it can do some things fstab can't), socket activation, user services, timers, dependency relations between units, descri ing how services that should only come up after the network is up, etc. Docker's tooling really doesn't seem better to me.
I was mostly comparing Docker to system packages, and I was specifically thinking about how trivial it is to use Docker Hub or GitHub for image hosting. Yeah, it's "infrastructure", but it's perfectly fine to click that into existence until you get to some scale. I would rather do that than operate a debian package server. Agreed that Nix works pretty well for that case, and that it has other (significant) downsides. I'm spiritually aligned with Nix, but Docker has repeatedly proven itself more practical for me.
> Definitely depends on the codebase, but sure, packaging usually involves adhering to some kind of discipline and conventions whereas Docker lets you splat files onto a disk image via any manual hack that strikes your fancy. But if you don't care about your OCI images being shit, you might likewise not care about your DEB packages being shit. If that's the case, you can often shit out a DEB file via something like fpm with very little effort.
I'm not really talking about "splatting files via manual hack", I'm talking about building clean, minimal images with a somewhat sane build tool. And to be clear, I really don't like Docker as a build tool, it's just far less bad than building system packages.
> don't really understand this preference; I guess we just disagree here. Systemd has been around for like a decade and a half now, and ubiquitous for most of that time.
Yeah, I don't dispute that systemd has been around and been ubiquitous. I mostly think it's user interface is hot garbage. Yes, it's well documented that you can get rid of the pager with `--no-pager` and you can put the logs in a sane order with `--reverse` and that you specify the unit you want to look up with `--unit`, but it's fucking stupid that you have to look that stuff up in the man pages at all never mind type it every time (or at least maintain aliases on every system you operate) when it could just do the right thing by default. And that's just one small example, everything about systemd is a fractal of bad design, including the unit file format, the daemon-reload step, the magical naming conventions for automatic host mounts, the confusing and largely unnecessary way dependencies are expressed, etc ad infinitum.
> Why would I want a separate, additional interface for managing services and logs when the systemd stuff is something I already have to know to administer the system anyway?
I mean, first of all I'm talking about my preferences, I'm not trying to convince you that you should change, so if you know and like systemd and you don't know Docker, that's fine. And moreover, I hate that I have to choose between "an additional layer" and "a sane user interface", but having tried both I've begrudgingly found the additional layer to be the much less hostile choice.
> I also frequently use systemd features that Docker just doesn't have, like automatic filesystem mounts (it can do some things fstab can't), socket activation, user services, timers
Yeah, I agree that Docker can't do those things. I'm not even sure I want it to do those things. I'm talking pretty specifically about managing my application processes. But yeah, since you mention it, fstab is another technology that has been around for a long time, is ubiquitous, and is still wildly, unnecessarily hostile to users (it can't even do obvious things like automounting a USB device when it's plugged in).
> ... dependency relations between units, descri ing how services that should only come up after the network is up, etc. Docker's tooling really doesn't seem better to me.
Docker supports dependency relations between services pretty well, via its Compose functionality. You specify what services you want to run, how to test their health, and how they depend on each other. You can have Docker restart them if they die so it doesn't really matter if they come up before the network (but I've also never had a problem with Docker starting anything before the network comes up)--it will just retry until the network is ready.
Docker's tooling is better in its design, not necessarily a more expansive featureset. It has sane defaults, so if you do `docker logs <container>` you get the logs for the container without a pager and sorted properly--you don't need to remember to invoke `sudo` or anything like that assuming you've followed the installation instructions. Similarly, the Compose file format is much nicer to work with than editing systemd units--I'm not huge fan of YAML, but it's much better than the INI format for the kind of complex data structures required by the domain. It also doesn't scatter configs across a bunch of different files, it doesn't require a daemon-reload step, the files aren't owned by root by default, they're not buried in an /etc/systemd/system/foo/bar/baz tree by default, etc.
Like I said, I don't think Docker is perfect, and I have plenty of criticism for it, but it's far more productive than dealing with systemd in my experience.
Option 2: use Poetry
How is this different than a Dockerfile that is creating the venv? Just add it to beginning, just like you would on localhost. But that is why I love to code Python in PyCharm, they manage the venv in each project on init.
Sorry, but you've got no idea what you're talking about.
You can also run OSI images, often called docker images directly via systemds nspawn. Because docker doesn't create an overhead by itself, its at its heart a wrapper around kernel features and iptables.
You didn't need docker for deployments, but let's not use completely made up bullshit as arguments, okay?
That doesn't necessarily mean there aren't Pro's to Docker, but one Con to Docker is - it's absolutely overhead and complexity that is not necessary.
I think one of the most powerful features of Docker by the way is Docker Compose. This is the real superpower of Docker in my opinion. I can literally run multiple services and apps in one VPS / dedicated server and have it manage my network interface and ports for me? Uhmmm...yes please!!!! :)
Your most powerful feature is literally a hostfile that docker generates on container start that's saved at /etc/hosts + Iptables rules
Edit: and if you don't want them, use Network-Mode: host and voila, none of that is generated
There are cons beyond performance. For example Docker complexity - you need to learn a new filetype, a new set of commands, a new architecture, new configurations, spend hours reading another set of documentation. Buy and read another 300 page O'Reily book to master and grasp something that again has Pro's and Con's.
For me? It's not necessary and I even know some Docker Kung-Fu but choose not to use it. I do use Docker Desktop occasionally to run apps and services on my localhost - it's basically a Docker Compose UI, and I really enjoy it.
No. Docker doesn't "wrap" anything, and it certainly does not wrap Linux. Please reconsider looking at the documentation. It uses native kernel features. SystemD does a similar thing.
> For example Docker complexity - you need to learn a new filetype, a new set of commands, a new architecture, new configurations, spend hours reading another set of documentation
I can't say I agree.
...and bypass the host firewall by default unless you explicitly bind stuff to localhost :-/
I don't particularly love or hate docker, but when I realized this, I decided to interact with it as little as possible for production environments. Such "convenient" defaults usually indicate that developers don't care about security or integrating with the rest of the system.
Not really, no. Docker just uses functionality provided by the Linux kernel for its exact use case. It's not like a VM.
> it's absolutely overhead and complexity that is not necessary.
This is demonstratively wrong. Docker introduces less complexity compared to system native tools like Systemd or Bash. Dockerfiles will handle those for you.
> I have no idea what I am talking about
I wouldn't say that. You seem to have strong puritarian opinions tough.
Yes it does, the Docker runtime (the daemon which runs under root) is horribly designed and insecure.
A lot of system applications on a standard Linux machine run as root or run with rootful permissions. This problem is solved by sandboxing, confining permissions and further hardening.
> requires sudo to use
Yes. However, this is a security plus and not a disadvantage.
> turns off all system firewalls
This statement makes no sense.
> has no way of doing security updates for containers.
I don't know what you mean by this.
(Theoretically you could build bare images without pulling in Alpine or Ubuntu, but literally almost nobody ever does that. If you have the skills to build a bare Docker image then you don't need Docker.)
You're not duplicating an entire distribution, just the user land that you want. Typically we use minimal user lands that just have certs and /etc/passwd and maybe `sh`. And to be clear, this is mostly just a disk overhead, not a CPU or memory performance overhead.
> Theoretically you could build bare images without pulling in Alpine or Ubuntu, but literally almost nobody ever does that
Yeah, we do that all the time. Google's "distroless" images are only about 2MiB. It's very commonly used by anyone who is remotely concerned about performance.
> If you have the skills to build a bare Docker image then you don't need Docker.
Building a bare Docker image isn't hard, and the main reason to use Docker in a single-host configuration is because Docker utilities are just far, far saner than systemd utilities (and also because it's just easier to distribute programs as a Docker images rather than having to deal with system package repos and managers and so on).
I would love to do away with the Docker layer, but first the standard Linux tooling needs to improve a lot.
Forget about "clunky overhead" - the running costs are < 10%. The dockerfile? You don't even need one. You can just pull from the python version you want e.g. Python1.11 and git pull you files from the container to get up and running. You don't need to use container image saving systems, you don't need to save images, or tag anything, you don't need to write set up scripts in the docker file, you can pass the database credentials through the environment option when launching the container.
The problem is after a year or two you get clashes or weird stuff breaking. And modules stopping support of your python version preventing you installing new ones. Case in point, Googles AI module(needed for gemini and lots of their AI API services) only works on 3.10+. What if you started in 2021? Your python - then cutting edge - would not work anymore, it's only 3.5 years later from that release. Yeah you can use loads of curl. Good luck maintaining that for years though.
Numpy 1.19 is calling np.warnings but some other dependence is using Numpy 1.20 which removed .warnings and made it .notices or something
Your cached model routes for transformers changed default directory
You update the dependencies and it seems fine, then on a new machine you try and update them, and bam, wrong python version, you are on 3.9 and remote is 3.10, so it's all breaking.
It's also not simple in the following respect: your requirements.txt file will potentially have dependency clashes (despite running code), might take ages to install on a 4GB VM (especially if you need pytorch because some AI module that makes life 10x easier rather needlessly requires it).
life with docker is worth it. i was scared of it too, but there are three key benefits for the everyman / solodev:
- Literally docker export the running container as a .tar to install it on a new VM. That's one line and guaranteed the exact same VM, no changes. That's what you want, no risks.
- Back up is equally simple; shell script to download regular back ups. Update is simple; shell script to update git repo within the container. You can docker export it to investigate bugs without affecting the production running container, giving you an instant local dev environment as needed.
- When you inevitably need to update python you can just spin up a new VM with the same port mapping on Python 3.14 or whatever and just create an API internally to communicate, the two containers can share resources but run different python versions. How do you handle this with your solution in 4 years time?
- If you need to rapidly scale, your shell script could work fine, I'll give you that. But probably it takes 2 minutes to start on each VM. Do you want a 2 minute wait for your autoscaling? No you want a docker image / AMI that takes 5 seconds for AWS to scale up if you "hit it big".
I have found that going all-in with certain language/framework features, such as self-contained deployments, can allow for really powerful sidestepping of this kind of operational complexity.
If I was still in a situation where I had to ensure the right combination of runtimes & frameworks are installed every time, I might be reaching for Docker too.
We have a simple cloud infrastructure. Last year, we moved all our legacy apps to a Docker-based deployment (we were already using Docker for newer stuff). Nothing fancy—just basic Dockerfile and docker-compose.yml.
Advantages:
- Easy to manage: we keep a repo of docker-compose.yml files for each environment.
- Simple commands: most of the time, it’s just "docker-compose pull" and "docker-compose up."
- Our CI pipeline builds images after each commit, runs automated tests, and deploys to staging for QA to run manual tests.
- Very stable: we deploy the same images that were tested in staging. Our deployment success rate and production uptime improved significantly after the switch—even though stability wasn’t a big issue before!
- Common knowledge: everyone on our team is familiar with Docker, and it speeds up onboarding for new hires.
For example, if you have a program that uses wsgi and runs on python 2.7, and another wsgi program that runs on python 3.16, you will absolutely need 2 different web servers to run them.
You can give different ports to both, and install an nginx on port 80 with a reverse proxy. But software tends to come with a lot of assumptions that make ops hard, and they will often not like your custom setup... but they will almost certainly like a normal docker setup.
The bad cloud infrastructure is when people try to use every single thing AWS sells and their whole infrastructure is at super high levels of abstraction that they could never migrate to another platform. K8s isn't that at all.
However, I ran kubeadm on a hetzner server and it's just sat chugging along forever basically. I use the cluster to run ephemeral apps where I build and deploy 1 golang service, a couple of node services in about 60 seconds ( with cache, obviously ).
As someone old enough and skilled enough to do the same with puppet, why bother when it's simpler easier that even the kids who don't understand TLS can do it with k8s?
With k8s you get a way of saying 'WHAT YOU WANT' without 'HOW TO DO IT', and this is applies not only to the actual infra aspect, but the people maintaining it too. Any cloud platform and devops worth their salt can maintain a k8s system. Good luck finding someone to understand what that 'custom Naemon' plugin is doing.
You Kubernetes people get triggered very easily. I was already lucky to have found several juniors that worked in this kind of thing with minimal training. The 'custom Naemon plugin' is 30 lines of bash and you can adapt it to any monitoring system.
Of course this is scary and complicated. I might consider switching to 'Kubernetes operators', which sounds simpler :-)
I think everyone just has to acknowledge that there are use cases for both. Also Kubernetes and "classic" configuration management via Ansible (or others) are orthogonal to each other. So these discussions are somewhat misguided in the first place.
For example: you might want to deploy a VM or auto-install and configure a physical machine with custom tooling and something like Ansible or Puppet and _then_ configure said machine as a Kubernetes node that handles the actual workloads. In other cases some Dev might want to install and run an application without the k8s layer using Nginx as a webserver. In this case, too, Puppet/Ansible might or might not be involved in configure the application but only handle the "OS layer" if there is such a thing. And in yet other cases you get away with a simple cloud-init script that makes your machine a k8s node and leave out other configuration management tools altogether.
Guess what: All of this is fine. Evaluate solutions based on what you need, not what other people working in giant corporations urge you to use. And then go and build it, ideally having fun doing it.
Representing either tool as a one-size-fits-all is misguiding at best and seems to be overly simplistic to the complex problem of deploying your applications.
I am generous in the context for generosity. Turns out that engineering is not about being generous but rather about choosing the most efficient solution for problems that in the end need to be business driven. This requires evaluating requirements, context and tradeoffs. That takes a cold, rational mind more than generosity.
> K8s really does address real problems around deployment and it's very well thought out
It's great where it makes sense. It's less than great elsewhere.
Not everything is SaaS, not everything needs scaling, not everything needs 99.99% of uptime, not everything needs a CDN, not every company is VC backed operating at high risk / high reward, etc, etc. Context is better than ideology. If you read the article I posted you will see that stated clearly.
While I am a happy cloud infrastructure user in private, I have to go through some extra hoops to deploy applications at work, regardless of if k8s is used or not.
Two examples that I came across
- "Test" mean if it passes on CI, it is good. Failing to run test on local? Who do development on local anyway?
- Teams so reliant on "AI" because this is the future of coding. "how to sort a list in python" became a prompt, rather than a lookup on the official documentation.
This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.
I tried to explain this topic in the article linked above. Not sure how far I succeeded.
You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
It is definitely less terrible than trying to unfuck tangles of terraform / terragrunt / yaml / bits of cloud infra.
(30 nodes)
I'm glad it was painless for you. In my experience, it was not, and most of the reasons were beyond my control.
If all Linux is to you is a place to run some application software, these choices are mostly irrelevant. As long as the software you care about continues to run, the other things are just picayune details. If this comes off as derisive, I apologize, because I'm actually broadly endorsing that view of things, as much as it is possible to achieve. But if you start really taking advantage of the things which the distribution provides out of the box and recommends, especially around large-scale multi-system operation, you end up buying into the distibution's choices. When a large organization you're a part of does it too, now the sunk costs really start to mount. As the Linux ecosystem continues to evolve, especially in different directions than the distribution chose at the time, the cost of migrating to later releases grows. This is all a good reason to me to not marry oneself so tightly to those particular choices, but that isn't always feasible with deadlines and compliance requirements and so on bearing down on the sysadmin.
There's also an even bigger problem that can arise, the distribution can just end, such as the termination of CentOS, leaving lots of people hanging. In that case, I know some who started to pay Red Hat for RHEL, but most seem to have moved on to other distros, like Ubuntu. That kind of migration has a lot of the same issues, too, once again leaving me to recommend not to lean into the particulars too much.
You mean management interfaces and repo mirroring stuff provided by the OS vendor, like cockpitd and Satellite and whatever?
And yeah if you package stuff against, e.g., the Python libs included in the distro (or EPEL), you essentially need to maintain a repo as a downstream repo of the distro, then rebuild the whole repo with whatever subsequent release as a new upstream when it's time to upgrade. That kind of thing is doable but it's substantial integration work, and if it's aomething you do once a decade nobody is ever going to be fluent in it when it's time to be done.
I think I'd rather just maintain two repos— one against the latest stable release and one against the upstream rolling release (Fedora Rawhide, Debian Unstable, openSUSE Factory or Tumbleweed, etc.)— and upgrade every 6 months or whatever than leap the wider chasms between LTS releases.
And yeah the Python and Python libs shipped in a distro are generally there for the distro's integration purposes, which may involve different goals and constraints than app developers usually have. Building against whatever a distro ships with is not always the best way, as your painful migrations demonstrated.
If you are doing something serious you probably want to chose suppliers in such a way that you can demonstrate you have security and business continuity under control. That means you probably want to use RHEL, Suse or Ubuntu, distributions for which commercial support exists.
(Ubuntu is particularly interesting because you can start with an LTS release for free and activate commercial support if business goes well, without changing your processes.)
You can think about this beforehand or wait until customers require some kind of certification and the auditors ask you for your suppliers list + the business continuity plan, among other things. You will face this if you deliver to a regulated market or if your customers are large enough to self regulate this kind of thing.
LTS not good enough? Well, cloud native does not have LTS comittement and Pipy does not provide security fixes separated from logical changes.
Try to keep your Terraform code stable for two years in AWS, or try to understand the lifecycle of AWS Glue versions from the docs. Or trust that Google will not discontinue their offers :-)
I mean, maintaining software is never easy or effortless but I respect the effort done by LTS Linux providers - they sell stability and security for a fraction of what you pay for cloud native.
There is no need to adapt things that work as they are already.
If possible, it's nicer to pick established, mature software for as much of your stack as you can, so that there's less of a difference in APIs over longer time frames. But it's not always possible.
IMO, the lure of an LTS is that you don't need to keep testing if your computer is still working every week when a set of updates come. Not that things that your software depends on the details remain frozen. If your software depends on the details of something, you should add it as a dependency.
What should be a gradual migration as new things develop turns into a singular nightmare.
Once in a decade you get something like a breaking upgrade of nginx, or the glibc debacle of 2003. That may take a person-week to fix[1], what can hardly be called "herculean".
1 - If you go with 1 person * 1 week, if you try to go with 7 people * 1 day, it will suddenly cost 7 person-weeks. But the only way upgrading is such a hurry is if you borked a lot of things prior to it.
Many of my bad experiences were because of corporate policies and lack of proper prioritization at levels above system administration. However, the sysadmin does have some choice in the matter, especially when greenfielding. You can turn stability into a vice if you're not careful.
If all my machines are FreeBSD 4.11, I don't care if my scripts don't run on Linux or Solaris or SCO or even FreeBSD 4.8 or 14. I might care someday, but not today.
Maintenance scripts need to run on all the versions in the fleet (usually), but setup scripts can often be limited to the latest version, because why not use the latest OS if you're setting up a new machine.
If you're distributing software, yeah you've got to support a lot of variation. If you're at a shop that runs lots of different flavors, you have to support lots of variation. But a lot of people just pick a flavor and update the scripts as needed when the flavor of the day changes.
Trying to keep dependencies and running services as tight and small as possible helps a lot with keeping up to date on security. Don't need to update things that aren't installed, and may not need to update things that are installed but not running (but sometimes you do).
Other Unix distros don't have first-class support with Nix so you may need to take some additional care when working out your script (especially the part of it that installs Nix), but if you don't need to set up services this way you can write portable scripts with few limitations that will work across all Linux distros, macOS, probably FreeBSD and maybe NetBSD.
I've never been so lucky as to work at a place that used any Unix flavors other than Linux and macOS, though.
I was pinched myself: Security.
- With the cloud threats, everything needs to be constantly up-to-date. Docker images make it easier than permanent servers that need to be upgraded. We used to upgrade every week, now we’re upgraded by default. So yes, sometimes our images don’t start with the latest version of xyz. But this is rare, downgrade is easy with Docker, and reproduction on a dev engine easier.
- With the cloud threats, everything needs to be isolated. Docker makes it easy to have an Alpine with no other executable than strictly necessary, and only open ports to the required services.
I hate the cloud because 4GB/2CPU should be way enough to run extremely large workloads, but I had to admit that convenience made me switch.
The versions that are not locked can be a test or dev environment that constantly updates and checks for errors.
Security threats are a thing, how we do and don't use technologies as well which ones can also factor in to how much is exposed.
But there's a middle ground here too. To me there's a HUGE gap between Kubernetes distributed systems and shell script free for all.
it was a big reason why we moved to containers at the bare minimum, because its quick and easy to spin up and destroy and you are guaranteed what runs locally runs on prod. no more "well it worked on my system!".
Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it.
You can achieve this with Docker as well, if the arrangement is not too complex.
The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.
Sometimes the job descriptions are boastful in their reference to those technologies, and other times you can detect some level of despair.
I’m still very much an ansible noob, but if you have a repo with playbooks I’d love to poke around and learn some things! If not, no worries, I appreciate your time reading this comment!
How do you control access to this setup?
How do you deploy on a different provider to Hetzner?
How do you access logs on this setup?
How do others maintain this setup?
How do you run backups?
How do you run cron jobs?
How do you deal with an offline node?
How do you expose a new ingress?
How do you provision extra storage on this setup?
If any of those is answered with 'something homegrown' or 'just write a script' then you have all the reasons k8s is worth it.
- https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
- https://www.hetzner.com/hetzner-summit --> "Managed Kubernetes Insights and lessons learned from developing our own Kubernetes platform"
If I was to write an idempotent script for each native resource I would finish in some years :-)
You chose whatever monitoring system you like the most.
For offline nodes you use whatever the level of criticity of your node justifies. This is something people struggle to understand: not every business needs 99.99% uptime. That said, I never had a downtime in Hetzner. On Digital ocean I had one short forced reboot in 4 years. YMMV so protect yourself as much as necessary.
Deploying on a different provider than Hetzner is the same as deploying on Hetzner except the part of launching the machine which is trivial to script - the added value is making the machine work and Ubuntu/Debian/RHEL are the same everywhere. You don't have vendor lock in with this.
If K8s works for you, enjoy it. Nobody is telling you to stop :-)
While I absolutely agree with you and your approach, would you mind elaborating what kind of configuration compliance you are referring to in this statement? I suppose you do not mean any kind of configuration that your Puppet code produces as that configuration is "monitored", or rather managed, by Puppet.
This case is actually pretty simple.
Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.
If there is an error the return code of the Puppet agent is different from that of the situations above.
Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.
Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.
This is active configuration management rather than what people usually call provisioning.
Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.
At my place of work we went the "traditional" way of running Puppet locally. It has been our experience that Puppet failures due to user misconfiguration or some such do not require our immediate attention (e.g. after hours), so we just check Puppetboard a few times per day to identify failing nodes.
Another reason why we use Puppetboard to monitor Puppet nodes is that every alert that our Icinga monitoring system produces is automatically interpreted as an incident which needs immediate attention. We are currently in the process of changing that so we are able to process non-critical alerts in a saner way.
Anyway, interesting to see how a fellow Puppet user manages their setup. Keep it up!
Feel free to reach out on Linkedin if you need some more details. More than happy to share.
The additional benefit is devs can run all the same stuff on a Linux laptop (or Linux VM on some other platform) - and everyone can have their own VM in the cloud if they like to demo or test stuff using all the same setup. Bootstrapping a new system is checking in their ssh key and running a shell script.
Easy to debug, not complex or expensive, and we could vertically scale it all quite a ways before needing to scale horizontally. It's not for everyone, but seed stage and earlier - totally appropriate imo.
Both are Apache 2 and the Flatcar folks are excellent to work with
Running and configuring VMs isn't hard to do correctly, it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
Yup, and I'm glad your experience has been different from mine but mine has been that tired and stressed people are anything but disciplined, so nipping a few "I'll just apt-get ..." in the bud goes a long way. So does Reverse Uptime (or its friend, Chaos Engineering)
If it interests you, both major git hosts (and possibly all of them) have and endpoint to map a username to their already registered ssh keys: https://github.com/mdaniel.keys https://gitlab.com/mdaniel.keys
It's one level of indirection away from "check in a public key" in that the user can rotate their own keys without needing git churn
Also, and I recognize this is departing quite a bit from what you were describing, ssh key leases are absolutely awesome because it addresses the offboarding scenario much better than having to reconcile evicting those same keys: https://github.com/hashicorp/vault/blob/v1.12.11/website/con... and while digging up that link I also discovered that Vault will allegedly do single-use passwords, too <https://github.com/hashicorp/vault/blob/v1.12.11/website/con...>, but since I am firmly in the "PasswordLogin no" camp, caveat emptor with that one
I've been running my SaaS first on a single server, then after getting product-market fit on several servers. These are bare-metal servers (Hetzner). I have no microservices, I don't deal with Kubernetes, but I do run a distributed database.
These bare-metal servers are incredibly powerful compared to virtual machines offered by cloud providers (I actually measured several years back: https://jan.rychter.com/enblog/cloud-server-cpu-performance-...).
All in all, this approach is ridiculously effective: I don't have to deal with complexity of things like Kubernetes, or with cascading system errors that inevitably happen in complex systems. I save on development time, maintenance, and on my monthly server bills.
The usual mantra is "but how do we scale" — I submit that 1) you don't know yet if you will need to scale, and 2) with those ridiculously powerful computers and reasonable design choices you can get very, very far with just 3-5 servers.
To be clear, I am not advocating that you run your business in your home closet. You still need automation (I use ansible and terraform) to manage your servers.
Did you read the article or just the headline?
Scroll down to the bottom, under the section "A few considerations" and try not to laugh.
"A few considerations" turns out to be a pretty significant chunk of security work ESPECIALLY if you are storing/transmitting highly sensitive information.
How do you handle something like HIPPA compliance when you're in this situation?
There are 2 types of programmers: those that think they've seen everything and those that know they've seen next to nothing. And as such, these absolute takes are tiring.
I'm a dev who hasn't seen anything related to that. Since you bring it up, can you give some pointers on why something like a MySQL db coupled to a monolithic backend isn't good enough? What shortcomings did you experience?
All of the things raised in the article seem possible to solve without the need for microservices.
First, this has nothing to do with microservices. Needing cloud infrastructure and building microservices are 2 orthogonal things.
Second, it has nothing to do with the tech you're using. MySQL is irrelevant. So is a monolithic backend.
What IS important is the security and infrasture behind the data your storing. Clinical data (and data captured in EMR's) is easily some of the most sensitive stuff you'll come across (unless you work in govt). The idea that I wouldn't use off-the-shelf, already-tested solutions specifically for this problem with a cloud provider is nuts. I pay Azure peanuts compared to what I'd have to pay a full-time person to manage multiple environments, security updates, provisioning new infra, etc. And that's not even considering the actual process you need to go to connect to outside systems.
Most integrations want you to have a SOCS audits and stuff. What happens when there is a breach? Do you have the personnel on staff to understand and troubleshoot the issue? Remember the "we have your data and will release it for bitcoin" hacks? That's only made possible by these systems sitting in closets in someone's facility.
And trust me, this isn't just a "large enterprisey" problem. It's a "everyone who wants to build an app in this space" problem.
So you can use MySql (if you can host it compliantly) and I'm building what you could theoretically call a "monolithic" backend and it's working well. I use MSSQL on Azure though.
When something bad does happen, is the cloud company liable?
Im a developer who is a CTO for a healthcare company (not like a big corp or anything) and also administers an Office 365 tenant while building out custom apps and an EMR. The office side of things is so much harder to get secure.
I'm sorry, are you saying securing patient data is simple? No offense, but you might be the only person on this planet to share this sentiment and there's a reason why.
So, it's simpler to secure sensitive information in a database, secure your hosting, maintain security updates to those hosts, undergo audits, keep up with changing regulations, keep up with the latest threat vulnerabilities, staff a full response team in case something happens, etc?
Not trying to be rude, but it's obviously not simple.
What's crazy about your answer is that we had a whole host of "Bitcoin for your data hacks" that were only made possibly by setups your describing.
>By far the hardest thing about that project was the complexity of the medical codes-
Yes, this is also complex. But a totally different problem in a totally different space.
To be fair of the things you've described, if you can swing it, you should be doing most of this regardless for a business setup. Specific to HIPAA would be the auditing and 'changing regulations' (and depending on client needs, you'll likely have other audits for business needs).
I'm going through a gap analysis for HIPAA now; would you mind sharing what impactful changing regulations you've seen in the past 5 years?
Not sure how to respond to this. Are you saying I should go out and hire 2-3 people to set up a ton of infrastructure and maintain it for me instead of relying on the professionals at Azure (who specialize in this) and it's done automatically at a fraction of the cost? We went through 5 years of "bitcoin for your data" fraud in exactly the situation your describing.
I don't need to hire anybody as of now. None.
> I'm going through a gap analysis for HIPAA now; would you mind sharing what impactful changing regulations you've seen in the past 5 years?
This is my point. I don't know and don't care. I don't have to worry about it at all. I don't have to worry about updating the handful of apps and servers that connect to all the different integrations we use because this field is siloed into a 1,000,000 little pieces. I don't have to worry about PHI getting leaked out of some server I forgot to update somewhere or misconfigured because I made a mistake while installing it or setting it up the first time. That stuff is all handled through Azure's existing cloud infrastructure. It's literally tailored to healthcare solutions. No single person (or 2 or 3 or even 4) full time people could come close to what they offer at the cost.
I understand that you're using Azure's existing infrastructure to handle your logistical technical management, but I was here asking if you had to make any changes to keep abreast of changing regulations. There seems to be practical business decisions that need to be made that HIPAA impacts, such as what data constitutes PHI (has that changed? Maybe you had to go back and change what data you were keeping because of the above regulation changes- I don't know if that could be the case, that's why I'm asking, I'm not aware of what I don't know). If Azure is somehow keeping track of all "changing regulations" for you (including business needs) and you've never had to worry about it, that's good to know. I would still be interested in any specific details if you're aware of it.
> but I was here asking if you had to make any changes to keep abreast of changing regulations.
No, we haven't. Not yet.
> If Azure is somehow keeping track of all "changing regulations" for you (including business needs) and you've never had to worry about it, that's good to know. I would still be interested in any specific details if you're aware of it.
I get your question know. So, when I was referring to Microsoft and HIPAA it was primarily around this side of things: https://learn.microsoft.com/en-us/azure/compliance/offerings...
You do bring up a good point and I shouldn't have implied otherwise that it can handle everything for you. So yes, there is a ton of other stuff that isn't magically handled by you such as identifying PHI and stuff. That being said, they have a whole suite of analytical and machine learning tools that will help you do this.
But since you mentioned policy changes, https://www.cms.gov/priorities/key-initiatives/burden-reduct... this is big and will have wide-reaching consequences and things like the ability to export patient data isn't necessarily baked into Azure.
BUT, they do have this healthcare platform they're building like this stuff https://learn.microsoft.com/en-us/dynamics365/industry/healt... that I would imagine would provide a bit more coverage on those types of changes than something you're building yourself.
Here's a deidentification service that can be integrated: https://learn.microsoft.com/en-us/azure/healthcare-apis/deid...
Someone else above had mentioned the complexity of medical coding and I don't know what you do or what you're working on but that's another really interesting part of the puzzle. And starts to get into why it's so hard for one system to communicate with each other in healthcare.
The most important things to consider (IIRC) were ensuring that the data was encrypted at rest and in flight, and that access to the data was audit logged and properly authorized.
We had an audit every so often. None of this was hard. Just tedious. It does help to have a HIPPA expert advise.
I don’t think public cloud vs self hosting makes a massive difference. Of all the problems such a project faces, that is not close to the top one.
Keeping machines patched and up to date is also not terribly hard.
Anyway, I’m not saying you’re totally wrong. Our project may have had more hidden risk than I realize. But it’s my opinion based on that experience.
Right now, I'm the CTO of a medium-sized healthcare company. We're building our own EMR to replace the one we're currently using ON TOP of building out some line-of-business integrations that can help modernize other parts of our office.
Part of that is grabbing data from an FTP EDT source from an HIE, storing that, processing it and then reporting. Our EMR has a bulk data download that we roll throuhg each night, processing data, building reports, etc. These integrations also tie into existing apps we use like Microsoft Teams, Microsoft Forms, Power BI, etc.
With the EMR we're building, I was able to pull on some help early on, set up all environments in Azure (dev, test, prod), all databases, background services (which we use A TON), blob storage, certificates, etc. I can count on one hand the number of times I've had to touch it since.
Prior to me coming on, all our data was stored on a server we hosted ourselves. It was a simple shared drive that constantly needed to be patched and updated. Went down ALL the time. And became a nightmare to manage on top of the 20 other pieces of technology we needed to use to get by. You know what I did? Copied the entire share to OneDrive and shut down the server and I was done. Never had to think about it again. And it's versioned. That's another benefit of cloud infrasturcture.
I'm a single dev at a healthcare company that has dozens of things going on all because I can rely on Azure's cloud infrastructure.
And that's not even counting the additional healthcare services they offer like FHIR servers, deidentifications services, pulling out snomed, med, and diagnoses codes from history and physicals, etc.
I couldn't come close to this if I was tasked to do it myself. And the problem is that healthcare changes constantly. So you need to be able to be nimble and fast. Being able to offload those sort of challenges has been super helpful in that regard.
It's not a silver bullet. My biggest issues NOW are people related. Links in emails are the hands down the biggest attack vector I have to worry about (for better or for worse).
As far as the coding complexity, while a totally different animal, is another huge challenge as you mentioned. And it's not just "how do I translate this to a billing code" it's being able to make sense of unstructured clinical documentation, being able to report on it and analyze it, and most importantly share it. An encounter with a patient could potentially have to collect upwards of 2000 data points that are changing based on the patient, the diagnoses, or what's happening the world (Covid for instance). It's an insanely challenging problem which it sounds like you have experience with.
I’m not opposed to the cloud. I run my current (non-HIPAA) project on Render, and it is really convenient. But, I also run a number of things on VPSs, and they aren’t difficult at all other than the up-front friction. They have been rock solid for us. I think it’s mostly a function of how simple we keep our setup. The cloud is certainly more convenient when managing a big team with lots of dynamic allocations of resources. But, VPSs (which some consider to be then cloud), and physical servers get more shade than I think they deserve.
You can go really far as a business on a single physical server and with a second backup server. With a bit of care, deployments can be simple and reliable, too.
I need to remember most people aren't as bad as I am on the infra side of things.
> You can go really far as a business on a single physical server and with a second backup server. With a bit of care, deployments can be simple and reliable, too.
You're right. A lot of what pushed me towards the cloud was that I wasn't building a single app. It was a collection of small, line of business type of stuff + an in progress EMR + a ton of Office 365 integration so it always made sense to go straight for Azure. As well as just not having the experience it sounds like you do.
It's when one starts getting sucked down the "cloud native" wormhole of all these niche open source systems and operators and ambassador and sidecar patterns, etc. that things go wrong. Those are for environments with many independent but interconnecting tech teams with diverse programming language use.
I which there was something that did just that, because kube comes with a lot of baggage, and docker-compose is a bit too basic for some important production needs.
But I think for many "kubernetes" means your second paragraph. It doesn't have to be like that at all! People should try settling up a k3s cluster and just learn about workloads, services and ingresses. That's all you need to replace a bunch of ad hoc VMs and docker stuff.
I can have something with nice deployments, super easy logs and metrics, and a nice developer experience setup in no time at all.
What I actually got was a half an hour tutorial from the guy who set it up, in which he explained the whole concept (I had no clue) and gave me enough information to deploy a server, which I did with zero problems. I had automatic deployment from `git push` working very quickly.
To me this seemed like a no brainer. Unless you literally have one service this is waaay easier to use.
Granted I didn't have to set it up - maybe that's where the terrible reputation comes from?
Seriously, I think a lot of people do things the hard way to learn large scale infrastructure. Another common reason is 'things will be much easier when we scale to a massive number of clients', or we can dynamically scale up on demand.
These are all valid to the people building this, just not as much to founders or professional CTOs.
I don’t quite get if people do it for interest, for love of the tech, or if they are technocratic and believe in levelling up their skill to get k8s on their CV like you say.
All I think is “this looks painful to manage”!
I run a k8s cluster at home. Part of it yes, is to apply my existing skills and keep them fresh. But part of it is that kubernetes can be easier long term.
Ive got magical hard drive storage with rook ceph. I can yoink a hard drive out of my servers and nothing happens to my workloads.
I can do maintenance on one of the servers with 0 down time.
All of my config for what I have deployed is in git.
I manage VMS and kubernetes at work, and im not going to pretend that kubernetes isnt complex, but it's complex up front instead of down the road. VMs run into complexity when things change. I'm sure you can make VMS good but then why not use something like kubernetes, you will have to reinvent a lot of the stuff that's already in kubernetes.
It's a hammer for sure and not everything is a nail, but it can be really powerful and useful even for home labs.
Kubernetes does not force you into the distributed systems hell, you can go that route later, or never.
b) In what world is upgrading your entire platform ever a single step. Even for a basic Python app you still have Python itself plus dependencies. And then of course whatever front end web server you're using.
Few but not no people will ever run into problems at the kind of scale k8s operates at. Plus, learning how it "expects" the programs running inside its Pods to behave is kind of like learning how Django or Rails "expect" a web app to work - it's a more complicated style than just writing your own totally custom, hermetically-sealed Python apps for your personal use, sure, but it also comes with a slew of benefits in case you ever do hit that level of scale and want to move over.
Or, maybe you look over the app you're writing and say "Fat chance." In which case you can justify e.g. not making everything an API endpoint, keeping a ton of state mucking about, etc. But I still feel that's an improvement over not even realizing the questions are being asked.
Is that really true anymore? Even self hosting k8s these days (e.g with rke/rke2) is a single yaml file and one command to deploy an entire cluster.. Maybe back when we all used kubespray and networking was more complicated (to the user at least) etc.. But today? I don't think so.
Using a hosted offering is even easier, literally a couple of clicks, a ./gcloud-cli or terraform apply -- again not very hard and all the cloud providers provide you with example code you just need to plug some machine sizes etc into..
Dev setup? Install orbstack and click 'kubernetes' and you're done, your IDE (likely) will automagically pick up your kubeconfig and you can go right ahead creating services, deployments, jobs, whatevers...
You get basic loadbalancing, health checks, centralized and nearly out of the box logging and monitoring and tracing.
You get a streamlined build process (create a container image, have an image build, create your helm chart, done)
Your RAID commment is quite far away of what k8s makes k8s
These things aren't mutually exclusive though. I've spent the last few years working with kubernetes at work and running a 'simple'(but with tons of containers and weird edge cases / uses) unraid server at home for all of my needs. At some point I flipped over from 'jeez kubernetes is just too much, almost nobody should ever use this' to 'wow I have to migrate 99% of my home services to a cluster, this is driving me nuts.' I haven't quite gotten around to that migration, but I do think that k8s cluster for services / temporary storage / parallel jobs and separate unraid box that runs NFS (and doesn't do much else) is going to be a great setup for a home lab.
I dunno why, but the k8s in my workplace keeps breaking in painful ways. It also has an endless supply of breaking points that makes life painful for anybody that depends on what runs in it, but aren't detected by the people that manage it.
Honestly, that second part is an exclusivity there, but I have never seen people "just keeping adding things to it" on practice.
aka, you're front loading the complexity.
You can even think of it as paying insurance premiums upfront. You get to "make a claim" if the requirements do grow into the sort of need that suit such a cluster/complex setup.
It's a bit like factorio with the extra dopamine hit of getting to unbox stuff.
You don't need k8s for all of that, but there's not a simpler solution than k8s that handles as much.
Life is full of pain. Deal with it.
Some people seem to have no concern with the needs and timetables of the would be customers but instead burn through cash building fancy nonsense.
It's like going in to a car mechanic for tires and then finding out it took 3 weeks because the guy wanted to put on low rider hydraulics and spinner hubcaps for his personal enrichment.
The worst part is it's inherently ambiguous to the next people. They don't know if the reason something is there is because it's needed or because it's just shiny bling.
It's so insane. Like hiring a long haul truck to pick up a sandwich
I've got this allergic reaction to bullshit and fetishize successful products and customer satisfaction. I think we've both changed; I'm different than I was 20 years ago and so is web development.
Tight applications with minimal tools that can be pivoted and changed swiftly which require competence and finesse to administer where you don't create developer debt, these are out of fashion.
All profitable hacker spaces professionalize as romantic magic becomes a liability.
I'm a middle aged divorced man, not divorced from a person, but from a profession and I've been trying to date around with new loves.
Having seen some of these half-rolled, first-time-understood k8s deployments, and the multi-year projects to unravel the mess that was created, overflowing with anti-patterns and other incorrect ways of doing things, I think I would prefer a narrower scope of true experienced professionals (or at least some experienced pros that can help guide the ship for their mentees) working on and designing k8s infra.
And for those that don't need it (the vast majority of startups, small businesses, regular-sized businesses, etc), just stick to the easier-to-use paradigms out there.
This is a case where “things will be much easier when we scale to a massive number of clients” turned out to be true.
Should you pick a complex framework from day one? Probably not, unless your team has extensive experience with it.
My objection is towards the idea that managing infrastructure with a bespoke process and custom tooling will always be less effort to maintain than established tooling. It's the idea of stubbornly rejecting the "complexity" bogeyman, even when the process you built yourself is far from simple, and takes a lot of your time from your core product anyway.
Everyone loves the simplicity of copying over a binary to a VPS, and restarting a service. But then you want to solve configuration and secret management, have multiple servers for availability/redundancy so then you want gradual deployments, load balancing, rollbacks, etc. You probably also want some staging environment, so need to easily replicate this workflow. Then your team eventually grows and they find that it's impossible to run a prod-like environment locally. And then, and then...
You're forced to solve each new requirement with your own special approach, instead of relying on standard solutions others have figured out for you. It eventually gets to a question of sunken cost: do you want to abandon all this custom tooling you know and understand, in favor of "complexity" you don't? The difficult thing is that the more you invest in it, the harder it will be to migrate away from it.
My suggestion is: start by following practices that will make your transition to the standard tooling later easier. This means deploying with containers from day 1, adopting the 12 factors methodology, etc. And when you do start to struggle with some feature you need, switch to established tooling sooner later than later. You're likely find that your fear of the unknown was unwarranted, and you'll spend less time working on infra in the long run.
One approach that I’ve considered is to start with the standard tooling (k8s + gitops) from day one, but still run it in a single VM. Any thoughts?
Unfortunately it's HN so people are more likely to do everything in bash scripts and say a big "fuck you" to all new hires that would have to learn their custom made mess
Also leaning towards putting it behind a Cloudflare tunnel and having managed Postgres for both k3s and application state.
Counterpoints anyone?
Have been running k3sup provisioned nodes on Hetzner for services and even a Stackgres managed Postgres cluster on another node (yes, it backs up to the cloud). And it's been great. Incredibly low cost and I do not have to think about running out if compute or memory for everything I need for a tiny startup.
If you do want to self-host, k3s could also be an option, like a sibling comment suggested. It's simpler to start with, though it still has a learning curve since it's a lightweight version of k8s. I reckon that you would still want to run at least 3 nodes for redundancy/failover, and maybe a couple more for just DB workloads. But you can certainly start with one to setup your workflow, and then scale out to more nodes as needed.
These are the only things I have ever been comfortable using in the cloud.
Once you get into FaaS and friends, things get really weird for me. I can't handle not having visibility into the machine running my production environment. Debugging through cloud dashboards is a shit experience. I think Microsoft's approach is closest to actually "working", but it's still really awful and I'd never touch it again.
The ideal architecture for me after 10 years is still a single VM with monolithic codebase talking to local instances of SQLite. The advent of NVMe storage has really put a kick into this one too. Backups handled by snapshotting the block storage device. Transactional durability handled by replicating WAL, if need be.
Dumbass simple. Lets me focus on the business and customer. Because they sure as hell don't care about any of this and wouldn't pay any money for it. All this code & infra is pure downside. You want as little of it as possible.
This is the most expensive way to build cloud services. When people talk about the cloud being more expensive than on-prem this is often the reason why. If you're just going to run VMs 24/7 there are better options.
You may never need to split your monolith! Stripe eventually broke some stuff out of their Rails monolith but it gets you surprisingly far.
You are not going to get easier to debug than a Django/Rails/etc monolith.
I bit of foresight on where you want to go with your infra can help you though; I built the first versions of our company as a Django Docker container running on a single VM. Deploy was a manual “docker pull; docker stop; docker start”. This setup got us surprisingly far. Docker is nice here as a way of sidestepping dependency packaging issues, this can be annoying in the early stages (eg does my server have the right C header files installed for that new db driver I installed? Setup Will be different than in your Mac!)
We eventually moved to k8s after our seed extension in response to a business need for reliability and scalability; k8s served us well all the way through series B . So the setup to have everything Dockerized made that really easy too - but we aggressively minimized complexity in the early stages.
And yet, funnily enough, the book on Monoliths says to break things up into smaller services! It says your data should be stored in its own service (possibly multiple services, if you need multi-paradigm access [e.g. relational, full-text search, etc.]). The user experience should use its own service. And, at very least, you should have another service in between (this is where Django and Rails usually fit). Optionally, it says, you will probably want to have additional services as well (auth, financial transitions, etc.)
VPS technology has come a very long way and is highly reliable. The disks on the node are set up in RAID 1 and the VM itself can be easily live migrated to another machine for node maintenance. You can take snapshots etc.
To me, I would only turn to cloud infra not for greater reliability but more for collaboration and the operational housekeeping features like IAM, secrets management, infra-as-code etc, or for datacenter compliance reasons like HIPAA.
I run a small, bootstrapped startup. We don't have enough money to pay ourselves and I make a living doing consulting on the side. Being budget and time constrained like that I have to be highly selective in what I use.
So, I love things like Google cloud. Our GCP bills are very modest. A few hundred euros per month. I would move to a cheaper provider except I can't really justify the time investment. And I do like Google's UI and tools relative to AWS, which I've used in the past.
I have no use for Kubernetes. Running an empty cluster would be more expensive than our current monthly GCP bills. And since I avoided falling into the micro-services pitfall, I have no need for it either. But I do love Docker. That makes deploying software stupidly easy. Our website is a Google storage bucket that is served via our load balancer and the Google CDN. The same load balancer routes rest calls to two vms that run our monolith. Which talk to a managed DB and managed Elasticsearch and a managed Redis. The DB and Elasticsearch are expensive. But having those managed saves a lot of time and hassle. That just about sums up everything we have. Nice and simple. And not that expensive.
I could move the whole thing to something like Hetzner and cut our bills by 50% or so. Worth doing maybe but not super urgent for me. Losing those managed services would make my life harder. I might have to go back to AWS at some point because some of our customers seem to prefer that. So, there is that as well.
Boring, but useful.
In all seriousness, that is "no load". I know it fits 99% of all startups, and many larger companies too, but that's kind of the point.
I wouldn't do it differently though, I think it's a perfectly fine architecture :)
> > 20-100 transactions per second
> "no load"
But that will cost more than $10/month.
I've not seen it with Go because I haven't worked with Go in a production capacity; but I've seen C# handle thousands of RPS per node.
Friends don't let friends use ruby|python|perl|php|...
Thankfully. ;)
I have had great success with a very simple kube deployment:
- GKE (EKS works well but requires adding an autoscaler tool)
- Grafana + Loki + Prometheus for logs + metrics
- cert-manager for SSL
- nginx-ingress for routing
- external-dns for autosetup DNS
I manage these with helm. I might, one day, get around to using the Prometheus Operator thing, but it doesn't seem to do anything for me except add a layer of hassle.
New deployments of my software roll out nicely. If I need to scale, cut a branch for testing, I roll into a new namespace easily, with TLS autosetup, DNS autosetup, logging to GCP bucket... no problem.
I've done the "roll out an easy node and run" thing before, and I regret it, badly, because the back half of the project was wrangling all these stupid little operational things that are a helm install away on k8s.
So if you're doing a startup: roll out a nice simple k8s deployment, don't muck it up with controllers, operators, service meshes, auto cicds, gitops, etc. *KISS*.
If you're trying to spin a number of small products: just use the same cluster with different DNS.
(note: if this seems particularly appealing to you, reach out, I'm happy to talk. This is a very straightforward toolset that has lasted me years and years, and I don't anticipate having to change it much for a while)
One big advantage of the operator is that its custom resources are practically kind of standard by now. This means helm charts for a lot of software ship those and integrating that piece of software into your monitoring is a matter of setting a few flags to true. The go to solution for a k8s monitoring setup is https://github.com/prometheus-community/helm-charts/tree/mai...
But yeah, if I only wanted a thing, Ebs works.
rsyslog + knowing what the fuck you are doing is much better.
But still, no matter what, the odd customer demands they need all these complexities turned on for no discernible reason.
IMO it’s a far better approach with any platform to deploy the minimum and turn things on if you need to as you develop.
Incidentally, I’ve been exposed to “traditional” cloud platforms (Azure, GCP, AWS) through work and tried a few times to use them for personal projects in recent years and get bewildered by the number of toggles in the interface and strange (to me) paradigms. I recently tried Cloudflare Workers as a test of an idea and was surprised how simple it was.
I thought the same thing until recently. Apparently there's a "Docker Swarm version 2" around, and it was the original (version 1) Docker Swarm that was deprecated:
https://docs.docker.com/engine/swarm/
Do not confuse Docker Swarm mode with Docker Classic Swarm which is no
longer actively developed.
> Docker secrets are only available to swarm services, not to standalone containers. To use this feature, *consider adapting your container to run as a service. Stateful containers can typically run with a scale of 1 without changing the container code.*
(Emphasis mine. From https://docs.docker.com/engine/swarm/secrets/ )
I was brought in to help get a full system rewrite across the finish line. Of course the deployment story was pretty great! Lots of automated scripts to get systems running nicely, autoscaling, even a nice CI builder. The works.
After joining, I found out all of this was to the detriment of so much. Nobody was running the full frontend/backend on their machine. There was a team of 5 people but something like 10-15 services. CI was just busted when I joined, and people were constantly merging in things that broke the few tests that were present.
The killer was that because of this sort of division of labor, there'd be constant buck-passing because somebody wasn't "the person" who worked on the other service. But in an alternate universe all of that would be in the same repo. Instead, everything ended up coordinated across three engineers.
A shame, because the operational story letting me really easy swap in a pod for my own machine in the test environment was cool! But the brittleness of the overall system was too much for me. Small teams really shouldn't have fiefdoms.
Puff! Talk about microservices! Or is it macropeople?! :-)
Since it's B2B we don't need zero downtime, updates at midnight are all right.
A day before rollout they go through the staging server and the test environment, so no surprises the next morning.
Before updates, the backups kick in, so if we need to recover from a bad update we can roll back.
Sounds all 2000 and not very fancy but boring and profitable cuts for us
My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime.
What i can do, can everyone else do when i'm on holiday.
How fast are you back if your server burns down tomorrow? How often have you tested that?
Are your devs waiting regularly on things?
Yes, we collect server metrics - that's pretty old-school
> How fast are you back if your server burns down tomorrow? How often have you tested that?
25 Minutes - we test it once a year and we have third partys to check it. It's called an audit. They also check other cyber security related stuff.
> My ci/cd is doing a system test because everything is in containers. I can do full e2e tests and automatic rollouts without a downtime.
We have a staging system for this.
> What i can do, can everyone else do when i'm on holiday.
We also have documentation; is this really a big thing?
> Are your devs waiting regularly on things?
Code Reviews, these take time
---
Are these real problems organizations have?
Huh. I never said to roll one's own [hardware] infrastructure, although it even makes sense if having a GPU cluster.
1. It took the end of ZIRP era for people to realize the undue complexity of many fancy tools/frameworks. The shitshow would have continued unabated as long as cheap money was in circulation.
2. Most seasoned engineers know for the fact that any abstractions around the basic blocks like compute, storage, memory and network come with their own leaky parts. And that knowledge and wisdom helps them make the suitable trade-offs. Those who don't grok them, shoot themselves in the foot.
Anecdote on this. A small sized startup doing B2B SaaS was initially running all their workloads on cheap VPSs incurring a monthly bill of around $8K. The team of 4 engineers that managed the infrastructure cost about $10K per month. Total cost:$8K. They made a move to 'cloud native' scene to minimize costs. While the infra costs did come down to about $6K per month, the team needed new bunch of experts who added about another $5K to the team cost, making the total monthly cost $21K ($6K + $10K + $5K). That plus a dent to the developer velocity and the release velocity, along with long windows of uncertainty with regards to debugging complex stuff and challenges. The original team quit after incurring extreme fatigue and just the team cost has now gone up to about $18K per month. All in all, net loss plus undue burden.
Engineers must be tuned towards understanding the total cost of ownership over a longer period of time in relation to the real dollar value achieved. Unfortunately, that's not a quality quite commonly seen among tech-savvy engineers.
Being tech-savvy is good. Being value-savvy is way better.
On AWS, fargate containers way are more expensive than VMs and non fargate containers are kind of pointless as you have to pay for the VMs where they run anyway. Also auto scaling the containers - without making a mess - is not trivial. Thus, I'm curious. Perhaps it's Lambda? That's a different can of worms.
I'm honestly curious.
As said, most of their workloads were on cheap VPSs before. Moved some to 'scale-to-zero' solutions, reduced the bloat in VMs, fixed some buggy IaC, also moved some stuff to the serverless scene. That got a decent ~20% reduction.
You can even run the whole thing locally.
We actually just did a Show HN about it:
Do startups really need complex cloud architecture?
Inspired, I wrote a blog exploring simpler approaches and created a docker-compose template for deployment
Curious to know your thoughts on how you manage your infrastructure. How do you simplify it? How do you balance?
What I quite like about your repo:
- there is a separate API and background job instance
- there is a separate web image, to not always couple front end deployments to back end
- there are specialized data stores like Redis (or maybe RabbitMQ or MinIO in a different type of project)
- Dozzle seems nice https://dozzle.dev/ (I use Portainer mostly, but seems useful)
- starting out with a monolithic back end but making it modular with feature flags (e.g. FEATURE_REPORTS, FEATURE_EMAILS, FEATURE_API), so that you can deploy vastly different types of workloads in separate containers BUT not duplicate your data model and don't need to extract shared code libraries (yet) and if you ever need to split the codebase into multiple separate ones, then it won't be *too* hard to do that
- having a clear API (RESTful or otherwise) as the contract between a separate back end and front end deployment, so that even if your SPA technology gets deprecated (AngularJS, anyone?) then you can migrate to something, unlike when doing SSR and everything being coupled
- the same applies to NOT having the same container build process have both the front end and back end build (I've seen a Java project install a specific Node version through Maven and then the build dragging on cause Maven ends up processing thousands of files as a part of the build)
- using the right tool for the job: many might create full text search, key-value storage, message queues, JSON document storage, even blob storage all with PostgreSQL and that might be okay; others will go for separate instances of ElasticSearch, Redis, RabbitMQ, something S3 compatible and so on, probably a tradeoff between using well known libraries and tools vs building everything yourself against a single DB instance
- in my experience, many projects out there are served perfectly fine by a single server so Docker Compose feels like the logical tool to start out with, if multiple instances indeed become necessary, there is always Docker Swarm (yes, still works, very simple), Hashicorp Nomad or K3s or one of the other more manageable Kubernetes distros
- self-hosted (or self-hostable) software in general is pretty cool and gives you a bunch of freedom, though using managed cloud services will also be pleasant for many, more expensive upfront but less so in regards to your own time spent managing the stack; the former also lends itself nicely to being able to launch a local dev environment with the full stack, which feels like a superpower (being able to really test out breaking migrations, look at what happens with the whole stack etc.)
- having some APM and tracing is nice, something like Apache Skywalking was pretty simple to setup, though there are more advanced options out there (e.g. cloud version of Sentry, because good luck running that locally)
- having some uptime monitoring is also very nice, something like Uptime Kuma is just very pleasant to use
- heck, if you really wanted to, you could even self-host a mail server: https://github.com/docker-mailserver/docker-mailserver (though that can be viewed as a hobbyist thing), or have MailCatcher / Inbucket or something for development locallyYou don't need to, you can just enable/disable certain features during app startup, based on what's in the environment variables/configuration, though many frameworks have built in functionality for something like that, for example: https://www.baeldung.com/spring-conditional-annotations
If I wanted to allow toggling access to the API, then I'd have an environment variable like FEATURE_API and during startup would check for it and, if not set with a value of "true", then just not call the code that initializes the corresponding functionality.
It's really nice when frameworks/libraries make this obvious, like https://www.dropwizard.io/en/stable/getting-started.html#reg... but it might get harder with some of the "convention over configuration" based ones, where you have to fight against the defaults.
> I'd also imagine there would be calls between features from within the same codebase, do those become network calls?
It depends on how you architect things!
There's nothing preventing you from using the service layer pattern for grouping logic, and accessing multiple services in each of your features as needed, and poking the different bits of your data model (assuming it's all the same DB).
If you are at the point where you need more than the same shared instance of a DB, then you'd probably need a message queue of some sort in the middle, RabbitMQ is really nice in that regard. Though at that point you're probably leaning more in the direction of things like eventual consistency and giving up using foreign keys as well.
> And how does this interact with your Docker Compose/single server recommendation?
Pretty nicely, in my experience!
When developing things locally, you can enable all of the needed FEATURE_* flags on your laptop, then it's more like a true monolith then.
Want to deploy it all on a single server when the scale is not too big? Do the same with Docker Compose, or maybe have separate containers on the same node, each with one of the features on, so the logs are more clean and the resource usage per feature is more obvious, and the impact of one feature misbehaving is more limited.
The scale is getting bigger? Docker Swarm will let you scale out horizontally (or Nomad/K8s, maybe with K3s) and you can just move some of those containers to separate nodes, or have multiple ones running in parallel, assuming the workload is parallelizable (serving user API requests, vs some centralized sequential process).
At some point you'll also need to consider splitting things further in your database layer, but that's most likely way down the road, like: https://about.gitlab.com/blog/2022/06/02/splitting-database-...
Rather, I have decided to opt for Supabase instead. Probably over the long time it may cause issues for my startup - but even more realistically my startup is going to fail and my increased developer velocity by using simple tooling like this will allow me to figure out why my idea doesn't work in in a shorter amount of time, so I can go on to my next pursuit.
To be honest I think even using docker is overengineering.
Focus on product market fit (PMF) and keep things as straightforward as possible.
Create a monolith, duplicate code, use a single RDBMS, adopt proven tech instead of the “hot new framework”, etc.
The more simple the code, the easier it is to migrate/scale later on.
Unnecessary complexity is the epitome of solving a problem that doesn’t exist.
Over-time this “clean” abstraction adopts a bunch of optional parameters based on the upstream API routes, leaving you with an omni-function that is more convoluted, and thus harder to change, than if the API routes weren’t overly optimized from the get-go.
As a personal rule, I’ll let myself copy something 3 times before taking a step back and figuring out a “better” way.
Low operational costs are essential for a hardware business if you don't want to burden your customers with an ongoing subscription fee. Otherwise the business turns into some kind of pyramid scheme where you have to sell more and more units in order to keep serving your existing customers.
I have a moral obligation towards my customers to keep running even if the sales stop at some point.
So I always multiply my cost for anything with 10 years, and then decide if I am willing to bear it. If not, then i find another solution.
I sometimes wonder how many of these post boil down to "I don't want to learn k8s can I just use this thing I already know?".
My team of 6 engineers have a social app at around 1,000 DAU. The previous stack has several machines serving APIs and several machines handling different background tasks. Our tech lead is forcing everyone to move to separate Lambdas using CDK to handle each each of these tasks. The debugging, deployment, and architecting shared stacks for Lambdas is taking a toll on me -- all in the name of separation of concerns. How (or should) I push back on this?
Why did the tech lead decide to move everything to lambda when you only have 1k DAU? Can they be reasoned with or is it lambda or the highway?
You can pull put the stats and do comparison, note the wasted time, how it's not beneficial but rather detrimental. Note how long it now takes to debug for such a small codebase, then extrapolate that out.
Having tons of lambdas is a massive pain in terms of debugging. Cloud watch is not that great to debug, and the debug tooling tends to be rather expensive, like data dog so not too much is invested. Or it's too resource intensive to setup open telemetry.
But an application built in the high pressure environment of a startup also has the risk of becoming unmanageable, one or two years in. And to the extent you already have familiar tools to manage this complexity, I vote for using them. If you can divide and conquer your application complexity into a few different services, and you are already experienced in an appropriate application framework, that may not be such a bad choice. It helps focus on just one part of the application, and have multiple people work on the separate parts without stepping on each other.
I personally don't think that should include k8s. But ECS/Fargate with a simple build pipeline, all for that. "Complex" is the operative word in the article's title.
And at that point you've assembled a stack just as complex as doing it all inside a single k8s cluster.
Also, if you're fair... not all those AWS acronyms you're listing would be displaced by the single k8s cluster. (Maybe you weren't arguing to swap out complexity, rather that the complexity floodgates were open already anyway?)
Very common in fact since many k8s clusters are air-gapped except for a single inbound edge node.
For new projects that, with luck, will have a couple hundred users at the beginning it is just overkilling (and also very expensive).
My approach is usually Vercel + some AWS/Hetzner instance running the services with docker-compose inside or sometimes even just a system service that starts with the instance. That's just enough. I like to use Vercel when deploying web apps because it is free for this scale and also saves me time with continuous deployment without having to ssh into the instances, fetch the new code and restart the service.
All businesses need to think about scalability, regardless of their size. If you're a startup, you likely want to be frugal with your infra costs, while still having the ability to quickly scale up when you need it. Those "simple" approaches everyone loves to suggest have no way of doing this.
But wait, don't you want some form of redundancy/failover in case one of these servers catches fire? Alright, let's double this then. Make sure to setup your load balancer as well, which should probably run on a separate server.
But wait, don't you also want some kind of staging environment, so that you can certify your releases before deploying them to production? Alright, let's double this again.
And so on, and so on... Eventually you'll end up rebuilding the same features of those complex gimmicky tools, but do a much worse job at it, and you'll also have to maintain your custom tooling mess.
Of course, if your company fails after a few months, none of this is worth considering. But if you plan to exist for the next few years, I would argue that your productivity would be considerably higher if you had just chosen that gimmicky tool from the start, or a very short time after it.
So, you had technical scalability in one system but if the customer base grew quickly every other bottleneck would be revealed.
There is more to business operations than technology, it seems.
Like all things, there's a good middle ground here-- use managed services where you can but don't over-architect features like availability & scaling. For example, Kubernetes is an heavy abstraction; make sure it's worth it. A lot of these solutions also increase dev cycles, which is not great early on.
Yes. This is the basis of privilege separation and differential rollouts. If you collapse all this down into a single server or even lambda you lose that. Once your service sees load you will want this badly.
> SQS and various background jobs backed by Lambda
Yes. This is the basis of serverless. The failure of one server is no longer a material concern to your operation. Well done you.
> Logs scattered across CloudWatch
Okay. I can't lie. CloudWatch is dogturds. There is no part of the service that is redeemable. I created a DyanmoDB table and created a library which puts log lines collected into "task records" into the table paritioned by lambda name and sorted by record time. Each lambda can configure the logging environment or use default which include a log entry expiration time. Then I created a command line utility which can query and or "tail" this table.
This work took me 3 days. It's paid off 1000x fold since I did it. You do sometimes have to roll your own out here. CloudWatch is strictly about logging cold start times now.
> Could this have been simplified to a single NodeJS container or Python Flask/FastAPI app with Redis for background tasks? Absolutely.
Could this have been simplified into something far more fragile than what is described? Absolutely. Why you'd want this is entirely beyond me.
Your dynamodb solution isn't foolproof. It has throughput limited to the partition granularity -> in your case the lambda name. It's also relatively expensive and fairly slow to query in bulk (DDB is designed for OLTP).
I don't have direct experience here, but I expect slapping grafana on top of any disk basked source is likely to be cheaper, faster, and have better ergonomics. Once your logging is too much for a disk to handle (this will be later than you would've outgrown ddb, but before you would've outgrown cloudwatch) then you can bring something fancy in.
The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event. I also wouldn't, for example, log API requests using this mechanism. We're nowhere near this being an issue as 20-30 lambdas is not a particular problem for us. Choose a good naming convention and build your own deployment infrastructure and it's no sweat.
> relatively expensive
Large object compression and/or offload to s3 is baked into our dynamodb interface library. Not that this matters as almost all log records end up being less than 4kb anyways.
> slow to query in bulk
Which is why time is part of the key. You're not often looking back more than an hour. There's bulk export back onto campus servers if you wanted that anyways. TTL is default 1 day. Running a "tail" is absurdly cheap, much cheaper than CloudWatch's laughable rate for their similar feature, a miss is 1/2 a read unit, and a hit is almost never more than 2.
> slapping grafana
I didn't need "observability." I need current state and recent deltas. This is particularly true when any changes are made. Otherwise my logs are pure annoyance and don't generally provide value. We optimized for the exceptionally narrow case we felt the cloud underserved in and left it at that.
Elsewhere in thread you say:
> The event volume is not particularly large as we tend to process things in batch and rarely on the edge of an event.
So the service is not actually under load, and it runs in batches so (temporary) failure is not actually a concern.
> This work took me 3 days. It's paid off 1000x fold since I did it.
Since Lambda was introduced less than 10 years ago, what you're saying here is that it'd be full time job for you for the past 10 years to maintain this (3000 days instead of three) if you have not gone the serverless way, which I find doubtful.
> Could this have been simplified into something far more fragile than what is described? Absolutely.
Considering the hyperboles in the rest of your comment, this sounds more like snark than a considered opinion.
What relation does any of those have with load?
(And also, why are people so kin on doing privilege separation by giving full privilege to a 3rd party and asking it to limit what each piece of code can do?)
Downside is its a one to one system. But I just use downsized servers.
Bare minimum, script out the install of your product on a fresh EC2 instance from a stock (and up-to-date) base image, and use that for every new deploy.
We run Spacelift workers with Auto Scaling Groups and pick up their new image ~monthly with zero hassle since everything is automated.
Raw EC2 is just part of the story...
Edit to add: I also recommend using Amazon Linux unless you _have_ to have RHEL / Cent / Rocky or Ubuntu. Just lean into the ecosystem and you can get so many great features (and yes, I ACK the vendor lock-in with this advice). A really cool feature is the ability to just flip on various AWS services like the systems manager session manager and get SSH without opening ports a-la wireguard.
obviously, it's not cloud-native... but if you are using AWS EC2 it works
Scaling (and relatedly, high availability) are premature optimizations[0] implemented (and authorized) by people hoping for that sweet hockey stick growth, cargo culting practices needed by companies several orders of magnitude larger.
[0] https://blog.senko.net/high-availability-is-premature-optimi...
I recently attempted to move to a completely static site (just plain HTML/CSS/JS) on Cloudflare Pages, that was previously on a cheap shared webhost.
Getting security headers setup, and forcing ssl, and www - as well as HSTS has been a nightmare (and still now working).
When on my shared host, this was like 10 lines of config in an .htaccess file before.
Let’s say I’ve got a golang binary locally on my machine, or as an output of github actions.
With Google Cloud Run/Fargate/DigitalOcean I can click about 5 buttons, push a docker image and I’m done, with auto updates, roll backs, logging access from my phone, all straight out of the box, for about $30/mo.
My understanding with Hetzner and co is that I need to SSH (now i need to keep ssh keys secure and manage access to them) in for updates, logs, etc. I need to handle draining connections from the old app to the new one. I need to either manage https in my app, or run behind a reverse proxy that does tls termination, which I need to manage the ssl certs for myself. This is all stuff that gets in the way of the fact that I just want to write my services and be done with it. Azure will literally install a GitHub actions workflow that will autodeploy to azure container apps for you, with scoped credentials.
This will most certainly cost you more than $30, but you can do it.
[1] https://keepassxc.org/docs/KeePassXC_UserGuide#_setting_up_s...
Depends on your defintion of simplest. In terms of set-up probably someting like https://dokku.com/ . It's a simple self-hosted version of herokku, you can be up and running in literally minutes and because its compatable with herokku you can re-use lots of github action/ other build scripts.
In terms of simple (low complexity and small sized components) just install caddy as your reverse-proxy which will do ssl certs and reverse proxy for you with extremely little, if any config. Then just have your github action push your containers there using whatever container set-up you prefer. This is usually a simple script on your build process like "build container -> push container to registry -> tell machine to get new image and run it" or even simpler just have your server check for updated images routinely if you don't want to handle communication between build script and server. That's the bare minimum needed. This takes a bit longer than a few minutes but you can still be done within an hour or two.
Regardless of your choice it shouldn't take more than 1 working day, and will save you a lot of money compared to the big cloud providers. You can run as low as €4.51/month with hetzner and that includes a static IP and basically unlimited traffic. An EC2 instance with the same hardware costs about $23 a month for comparison (yes shared vs dedicated vCPU, but even the dedicated offer at hetzner is cheaper, and this is compared to a serverless set-up where loads are spikey, which is exactly how we can benefit from a shared vCPU situation).
If you don't like ssh you can have a gitlab runner on your VM which will redeploy your stuff on git push / git tag / whatever you want
It does automatic certificates.
Everyone is building like they are the next Facebook or Google. To be honest, if you get to that point, you will have the money to rebuild the environment. But, a startup should go with simple. I miss the days when RAILS was king just for this reason.
The added complexity is overkill. Just keep it simple. Simple to deploy, simple to maintain, simple to test, etc. Sounds silly, but in the long run, it works.
In my time at my current job we've scaled PHP MySQL and Redis from a couple hundred active users to several hundred-thousand concurrent users.
EC2+ELB, RDS (Aurora, Elasticache). Shell script to build a release tarball. Shell script to deploy it. Everyone goes home on time. In my 12+ years I've only had to work off hours twice.
People really love adding needless complexity in my experience.
No, people love thinking their experience is the same as everyone else's.
Have you ever worked in healthcare? Do you have any idea what sort of requirements there are for storing sensitive information?
>n my 12+ years I've only had to work off hours twice.
Well that settles it. Then no one on the planet should need cloud infra if yuo didnt.
And please, please don't tell me you've spent the last 12 years at the same place and have the gall to extend that to all software development.
I presented my story of how we've actively kept our architecture simple, and noted we've had very few issues. I did not say our architecture is the architecture for everyone.
Then I said
> People really love adding needless complexity in my experience
If the complexity is legally mandated, as in healthcare, it's by no means "needless". Legal compliance is a need.
If the complexity is justified, has merit or value, it's not "needless".
However, I've known a fair number of people who work on complicated kubernetes driven architectures that give them non-stop grief, and whose user base max out at ten-twenty active users.
My point is just don't make things more complex than they need to be.
Sounds pretty absolute to me. I mean, when asked "Does your startup need complex cloud infra" (which is a loaded question) and you say "No. YAGNI" that seems pretty unequivocal and not really fair to say I misinterpreted it.
> My point is just don't make things more complex than they need to be.
I agree. I just don't care for the absolute language (that I and others use sometimes). It made learning when I was just getting into this field really tough. My answer to that exact same question would be "It depends".
I like the cloud but it is overused and misused a lot imo.
Postgres for everything including queuing
Golang or nodejs/TypeScript for the web server
Raw SQL to talk to Postgres
Caddy as web server with automatic https certificates
- No docker.
- No k8s.
- No rabbitmq.
- No redis
- No cloud functions or lambdas.
- No ORM.
- No Rails slowing things down.
The goal is not for the technologies used to be simple or boring.
The overall architecture is simple, the technologies used are powerful.
Your little startup will become large, and fast.
That hacked together single server is going to bite you way sooner than you think, and the next thing you know you’ll be wasting engineer hours migrating to something else.
Me personally, I’d rather just get it right the first time. And to be honest, all the cloud services out there have turned a complex cloud infrastructure into a quick and easy managed service or two.
E.g., why am I managing a single VPS server when I can manage zero servers with Fargate and spend a few extra bucks per month?
A single server with some basic stuff is great for micro-SaaS or small business type of stuff where frugality is very important. But if we shift the conversations to startups, things change fast.
Part of the reason they weren’t successful was because my managers insist on starting with microservices.
Starting with microservices prevents teams from finding product-market fit that would justify microservices.
What would be the simple yet robust infra for data eng? Not thought a lot about it for now, so I am curious if some of you have would have any insights.
In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".
Decisions are sticky so take context into account.
- A lot of companies and startups can get by with a few modest sized VPSs for their applications
- Cloud providers and other infrastructure managed services can provide a lot value that justifies paying for them.
Want to run bare metal? OK, guess you're running your databases on bare metal. Do you have the DBA skills to do so? I would wager that an astounding number of founders who find themselves intrigued by the low cost of bare metal do not, in fact, have the necessary DBA skills. They just roll the dice on yet another risk in an already highly risky venture.
Faulty input killing your logic - I saw this plenty, would Lambda really help here?
[0] https://kamal-deploy.org [1] https://kamalmanual.com/handbook
We’ve tried deploying services on K8s, Lambda/Cloud Run, but in the end, the complexity just didn’t make sense.
I’m sure we could get better performance running our own Compute/EC2 instances, but then we need to manage that.
In reality, there is a strong bias in favor of complex cloud infrastructure:
"We are a modern, native cloud company"
"More people mean (startup/manager/...) is more important"
"Needing an architect for the cloud first CRUD app means higher bills for customers"
"Resume driven development"
"Hype driven development"
... in a real sense, nearly everyone involved benefits from complex cloud infrastructure, where from a technical POV MySQL and PHP/Python/Ruby/Java are the correct choice.
One of the many reasons more senior developers who care for their craft burn out in this field.
One domain, an idea, an easy-to-use development stack for a bootstrapped as well as funded startup is more than good enough to locate product-market fit.
Alway remember this quote by Reid Hoffman “If you are not embarrassed by the first version of your product, you’ve launched too late.”
It's simple and can scale to complex if you want. I've had very good experience with it in medium size TS monorepos.
[0]: https://sst.dev
From what I understand he employees a dedicated system administrator to manage his fleet of VPS (updates, security and other issues that arise) for 1000s of USD per month.
Costs in my case is not the highest priority: I can spend a month learning the ins and outs of a new tool, or can spend a few days learning the basics and host a managed version on a cloud provider. The cloud costs for applications at my scale are basically nothing compared to developer costs and time. In combination with LLMs who know a lot about the APIs of the large cloud providers, this allows me to focus on building a product instead of maintenance.
Operating a bunch of simple low-level infrastructure yourself is not simpler than buying the capabilities off the shelf.
I'd say it is more like: Using a trolley to move some stuff across the street is more simple than using a fleet of drones.
Certainly you should not be running your own K8S cluster, but using Google Cloud Run is simpler than keeping your own server running. Even using Google Cloud Kubernetes Engine with autopilot is simpler than keeping your own server running.
So it's a risky thing to brag about right now.
Check:
Man the infrastructure was absolutely massive and so much development effort went into it.
They should have had a single server, a backup server, Caddy, Postgres and nodejs/typescript, and used their development effort getting the application written instead of futzing with AWS ad infinitum and burning money.
But that's the way it is these days - startup founders raise money, find someone to build it and that someone always goes hard on the full AWS shebang and before you know it you spend most of your time programming the machine and not the application and the damn thing has become so complex it takes months to work out what the heck is going on inside the layers of accounts and IAM and policies and hundreds of lambda functions and weird crap.
I built out a POC and was running it on bare metal for serious workloads under my desk at GE (12-factor). Management practically scrambled to get me cloud access. My setup was ephemeral and could be easily reproduced anywhere. The software was easily deployed on, or integrated with, cloud services. I just shrugged.
I didn't care where my code ran, to them it was some epic priority to get it in the cloud and generate extra expenses.
It’s really brilliant. Sun would have been the one to buy Oracle if they’d figured out how to monetize FactorySingletonFactoryBean by charging by the compute hour and byte transferred for each module of that. That’s what cloud has figured out, and it’s easy to get developers to cargo cult complexity.
I'd like to see things work on Erlang/Elixir in Production and whether that works better, since the BEAM is very good at preventing individual processes from dominating the server and is also very good at recovering from errors.
And S3. S3 is just a wonderful beast. It's just so hard to get something that is so cheap yet offers practically unlimited bandwidth and worry-free durability. I'd venture to say that it's so successful that we don't have an open-source alternative that matches S3. By that I specifically mean that no open-source solution can truly take advantage of scale: adding a machine will make the entire system more performant, more resilient, more reliable, and cheaper per unit cost. HDFS can't do that because of its limitation on name nodes. Ceph can't do that because of it bottleneck on managing OSD metadata and RGW indices. MinIO can't do that because their hash-based data placement simply can't scale indefinitely, let alone ListObjects and GetObjects will have poll all the servers. StorJ can't do that because their satellite and metadata servers are still the bottleneck, and the list can go on.
Of cource it highly depends on the skills of the team. In a startup there could be no time to learn how to do infrastructure well. But having an infrastructure expert in the team can significantly improve the time to market and reduce the developer burnout and the tech debt growth rate.
You shouldn't need to assemble a plane when your startups journey can be expected to only last a few kilometers and you really only need to carry a few boxes.
The result: It's still up after 5 years. I never looked back after I created the project. I do remember the endless other projects I did that have simply died now because I don't have time to maintain a server. And a server almost always end up crashing somehow.
Another thing, Pieter Levels has successful small apps that relies more on centralized audiences than infrastructure. He makes cool money but it's nowhere near startup-expected levels of money/cash/valuations. He is successful in the indie game but it'll be a mistake to extrapolate that to the VC/Silicon Valley startup game.
1. New technology is bad and has no merit other than for resume.
2. Use old technology that I am comfortable with.
3. Insist that everyone should use old technology.
hahahahahahahahahaha. Yes, back in the days when all you could do on a website is read the text.
Old technology was 1000% not simpler. What an insane & absolute statement to make in an enormous field just because you can't make a solid argument.
Really? EC2 instances are waaay overpriced. If you need a specific machibe for a relatively short time, sure, you can pick up one from the vast choice of available configurations, but if you need on for long-running workloads, you'll be much better of picking up one from Hetzner, by an order of magnitude.
For one of the many examples, see this 5-year old summary (even more true today) by a CEO of a hardware startup:
https://jan.rychter.com/enblog/cloud-server-cpu-performance-...
Most of crap hitting servers is old exploits targeting popular CMS.
WAF is useful if you have to filter out traffic and you don’t know what might be exposed on your infra. Like that Wordpress blog that marketing set up 3 years ago and stopped adding posts and no one ever updated it.
vulnerability scanning of your images.
Fargate
RDS
Yeah, that won't scale to a million QPS, or even 10 QPS. But way more businesses fail because they never achieve 100 Queries Per Day, instead of failing because they fell over at 10 or 1,000 or 1,000,000 QPS.
I mean, hell, Twitter (back in the day) was famous for The Fail Whale.
Getting enough traffic is harder and more important than your "web scale architecture" for your startup. Making actual cash money off your traffic is harder and more important than your "web scale architecture" (ideally by selling them something they want, but making cash money through advertising or by impressing VCs with stories of growth and future value counts too).
There is precisely _zero_ chance that if you ever get within 2 or 3 orders of magnitude of "a million QPS" - that the code you and your cofounder wrote won't have been completely thrown away and rewritten by the 20 or 100 person engineering department that is now supporting your "1000 QPS" business.
I need it on my resume for every 2 year stint and 2-3 people on the team to vouch for it
You’re saying “hey, let everyone know you worked on a tiny company’s low traffic product and how about you just don’t make half a million a year," all to save the company I work at a little money?
until companies start interviewing for that its a dumb idea, I’m rarely making green field projects anywhere and other devs also are looking for maintainers of complex infrastructure
99.99% of the time. No.
For many of the "complex" things like lambdas there are frameworks like Serverless that makes managing and deploying it as easy (if not easier frankly) than static code on a VM.
Not every workload also scales at the same time, we have seen new things that got very successful and crashed right out the door because it could not properly scale up.
I agree that you don't need an over engineered "perfect" infrastructure, but just saying stick it on a VM also seems like it is too far of a swing in the other direction.
That ignores the cost side of running several VM's vs the cost of smaller containers or lambdas that only run when there is actual use.
Plus there is something to be said about easier local development which some things like Serverless and containers give you.
You may not need to setup a full k8s cluster, but if you are going with containers why would you run your own servers vs sticking the container in something managed like ECS.
> But here's the truth: not every project needs Kubernetes, complex distributed systems, or auto-scaling from day one. Simple infrastructure can often suffice,
When the hell can we be done with these self compromising losers? Holy shit! Enough! It doesn't save you anything doing less. People fucking flock to bot-Kubernetes because they can't hack it, because they suck, because they would prefer growing their own far worse far more unruly monster. A monster no one will ever criticize in public because it'll be some bespoke frivolous home grown alt-stack no one will bother to write a single paragraph on, which no one joining will grok understand or enjoy.
It's just so dumb. Theres all these fools trying to say, oh my gosh, the emperor has no clothes! Oh my gosh! It might not be needed! But the alternative is a really running naked through the woods yourself, inventing entirely novel unpracticed & probably vastly worst less good means for yourself. I don't know why we keep entertaining & giving positions of privilege to such shit throwing pointless "you might not need it" scum sucking shits trying to ruin things like so, but never ever do they have positive plans and never ever do they acknowledge that what they are advocating is to take TNT to what everyone else is trying to practice, is collaborating on. Going it alone & DIY'ing your own novel "you might not need" to participate in a society stack is fucking stupid & these people don't have the self respect to face up to the tall dissent they're calling for. You'd have to be a fool to think you are winning by DIY'ing "less". Fucking travesty.
Amen.
> I don't know why we keep entertaining & giving positions of privilege to such shit throwing pointless "you might not need it" scum sucking shits t
Amen.