MRSK vs. Fly.io
fly.io
fly.io
My writeup, in case it helps others: https://www.vmii.org/blog/2023/03/12/kubernetes/
> The prevailing wisdom is that Kubernetes is overkill for a small Rails/Elasticsearch app, but overkill is my life philosophy.
It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something. Everything has to work in minutes. Next, what inevitably happens is that services shut down, pricing changes, or something stops working, and they have no way to debug it or move off it. And then we get customer support complaints and posts on HN about it. These developers look for the next shiny 3rd party service that solves the problem immediately and repeat the cycle, without ever learning anything that helps in the long term.
Quite the opposite. K8s are a big opaque ball of magical complexity to most of us devs for sure (as is heroku). However, for 100% of my use cases nothing should be more complex than the database.
I’m not opposed to learning about it, but reducing complexity wherever reasonable has paid off for me.
It's more than the learning curve, though. It also requires resources for itself (the control plane) so it's not really suitable for small projects unless you use a managed offering (which has the same downsides as you mentioned in your second paragraph).
Also you make it sound like all Kubernetes alternatives are 3rd party, paid, hosted services, which is far from the truth.
I can still run it locally for dev via Docker and I use Devspace to make it easy to bring up a dev env.
But one should not forget that you also had to build up a lot of vendor specific know how in the past. Someone had to configure your F5 BigIP and your Juniper Router and the Cisco Switch and of course the Dell or HPe boxes you bought.
I take more concerns with k8s immature ecosystem which is kind of reinventing classic unix stuff for distributed computing. And that just started and you've to lifecycle components with breaking changes every few weeks. And people took issues with updating Ubuntu LTS releases every two years. Now they have to update some component every week.
Look I can and have done all these things, but it's just not worth my time to do them for my little apps. I'd rather be talking to customers and shipping features at this point in my career.
I don't know. I had a pretty good thing going prior to k8s too, just some rsync and `ln -sfn` and it was easy, simple and very fast, but like you said, upgrading Ubuntu and PHP and other services becomes the problem there. Couldn't do that without downtime.
Trade-offs.
It barely takes a weekend to learn and play around with managed k8s from cloud providers or with minified k8s tools like k3s, k0s etc.
Because my project doesn't need the benefits that K8s offers. Why choose the more complex solution when I don't need it? And judging from experience and what others have shared, most projects will not need it.
> no way to debug it or move off it
If a managed service is down, the host company debugs it. That's the whole point of a managed service, so I don't have to do devops.
Why would there be no way to move off of it? PaaS is so easy to onboard and deploy, you can move to other services easily.
> It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something
Why don't you go all the way and build your own physical servers instead of using these magic cloud machines? That way you can go deep into it, and if they have problems, you can debug yourself.
If you enjoy building and maintaining your own K8s clusters and you feel it benefits you, great for you. But don't be so condescending to people who don't feel the same and choose the simpler infra solution because they'd rather go deep into building their application rather than spending time with K8s. Calling them spoiled for that is just obnoxious.
So what? It just takes a month long time investment to be used to kubernetes(many people already have some experience from working for employers), and after that it takes one or two days extra days to make some project as easy to deploy as heroku. So basically it is a fixed learning cost. Obviously if someone is looking to get something up as soon as possible and doesn't have kubernetes experience, it is better to not use it.
But for me, the fixed learning cost clearly paid off. Kubernetes is lot better than alternatives if you want to assign fine grained securities, namespaces, complex scaling rules, using multiple clouds, multiple node types etc. Even if I don't need any of these now, I am not giving any extra time investment for new projects and it guarantees I don't need to do any migration to some other form of deployment if my project becomes big.
Ok so next time I'm laid off I will have some tech to choose learning. May I will spend that month.
Even when I wasn't a parent I don't think my life was that open that I could say well it only takes a month, I guess I will learn that. Lots of things take a month to learn. people pick and choose.
The systems evolution that led us to Kubernetes does have it merits of course: I just know I can trust the simple foundations of control loop meets immutability in order to maintain complex distributed systems, including expectations for self-healing and resilience. Though IMHO the "freedom" aspect you hinted on above is the more important one these days - yet usually cloud platform dependence unfortunately creeps in, in other ways still.
Personally though I bet that for an easy 80% of "need to deploy software" cases, Kubernetes is indeed overkill - as long as it isn't "managed" / abstracted away at least to the level of a Heroku. And of course there are many other ways and platforms to benefit from "containers" and their promise (?) of independence these days.
All power to anyone though if they have the money/time/energy to put work into that layer, in addition to whatever else they are trying to achieve. Disclaimer being that learning can definitely be its own merit (that's how I got started myself) but it's important to know when it's "just" that, when it's more and when it's simply overkill.
What's your cost breakdown like?
And how are you finding it so far?
Excellent writeup. This line seems to track with what other people have told me about maintaining Kubernetes deployments. Almost no-one really understands the YAML (beyond the basics), so you end up with services defined via tens of thousands of lines of unmaintainable copy pasta YAML. This problem probably isn't going to surface so quickly (if ever) for a smaller project.
his output is crazy. that's why he gets the big bucks
So why fix something that ain’t broken?
That’s Capistrano v2 implemented in Ansible. Honestly been tempted just to try it on a side project for fun.
Once MSRK has gotten "good enough" for whatever Basecamp needs and DHH has moved on, how many other folks with the dedication and skills will help take over when maintenance and weird edge cases surface?
So far the rails community has been strong enough where this _usually_ isn't an issue, but it definitely is a real concern. You can see the impact of this in some of the recent asset bundling solutions DHH spiked out and then mostly left to the community, where things like js-bundling and css-bundling are close but not completely solid for a bunch of use cases.
I don't get why HN is going crazy about this. Please tell me if I am missing something obvious.
Not here? To me, general sentiment on news.yc has been anti cloud / k8s for at least 5 years, if not more.
Personally I use a ~200 line shell script + docker to deploy self hosted apps on a cheap office computer that lives under my desk.
It’s like comparing AWS with Chef. Maybe there’s some overlap in philosophy, but that’s about it. Maybe fly.io is just using it as a way to do a bit of marketing.
MRSK is a direct competitor to Fly's own orchestrator, and is meant for use when hosting on your own hardware or on a VM rented from an infrastructure provider. So it's still a "use either A or B" situation.
I guess a better title would be – Hosting on the Fly.io platform vs. on your own servers/VMs with MRSK.
I'd love it if all 'vs' and competitive marketing was this good.
Also, it could be implemented using Ansible, but it has been implemented in MRSK. A serious difference if you plan to count on someone to keep it updated and supported.
I've been using Ansible with Docker Compose since 2015. I can set up a full server with all of the bells and whistles in about 20 lines of inventory configuration. Behind the scenes there's a bunch of portable and generic roles that work for baseline "server level" components. I've done this now for dozens of companies, it's a tried and true model with an easy way to customize things if needed.
Ansible sets up the server, Docker Compose runs the apps (stand alone `docker compose` commands, not the Ansible module) and git is used to deploy everything.
All of those example apps are deployable to production (and can be used in development) once your server has Docker installed.
[0]: https://github.com/nickjj?tab=repositories&q=docker-*-exampl...
Those examples are for running your app, not deploying it.
Zero downtime is also a loaded term.
MRSK isn't zero downtime in a way that's commonly described as zero downtime. MRSK uses Traefik's built in functionality to queue requests so if your back-end is down it won't respond with a 502. Instead the user will see a busy mouse cursor until your back-end is up.
A user will be able to automatically "resume" their request after your app reloaded which is nice, it saves them from having to see a 502 and manually reload the browser but if you have a Rails app that takes 27 seconds to boot up then the user will be waiting for 27 seconds with no feedback. You could make a case the UX there is worse than a user explicitly seeing a custom 502 page that mentions to reload in a bit.
That's much different than a true zero downtime rolling update that something like Kubernetes will give you, or if you rolled your own solution to do it without Kubernetes. I don't know if MRSK will handle that, given the direction it's going it's quite possible DHH will implement that behavior in which case that would be very useful.
Rollback is also tricky once you start introducing database migrations. I personally always roll forward.
While these are used to deploy a Scala backend and a Scala.js frontend, I have adapted them many times to work with other backends and js frontends, the logic is the same, the build scripts is what needs to be updated.
For application deployment, I use simple scripts that restart docker compose via shh that we can run from Github actions. Ansible is overkill for that.
Thanks. I made a video and blog post on why I stopped recording it here: https://nickjanetakis.com/blog/taking-an-indefinite-break-fr...
The state of the show hasn't changed since then, it's RIP'd for now. I still have the urge to do the recording bits tho, it's a cruel world!
I very strongly believe:
- Ansible is a great tool for setting up a server to get it to the point where you can run / deploy applications
- Docker Compose is a great tool for running your app
- Git is a pretty good tool for initiating deployments
In my opinion these are 3 distinct things. Trying to make Ansible do everything gets messy, you also lose feedback. For example if you git push, a post receive hook will stream the response back to your terminal but Ansible will only show output when a task is complete.
Using the raw Docker Compose commands also keeps things more consistent between dev, CI and prod. You run basically the same thing in all environments.
I actually had a quick look at the code - and at this point it doesn't strike me as a tempting building block.
There's no developer overview i could find (birds eye guide to the MRSK code structure and concepts). There doesn't appear to be much in the way of tests?
Specially - i was wondering how hard it would be to replace traefic with caddy or haproxy, do letsencrypt via MRSK-managed parts of the deployment or drop in varnish.
As best I can tell - there's no clean or sensible way to do any of those things.
Despite that, as a simple Docker glue stick it seems quite useful.
I must say i was disappointed with one thing in the demo - hosting the app on the internet in plaintext covering it with a TLS figleaf via cloudflare. That's terrible guidance in a demo/tutorial.
I have mixed feelings towards NIH people. One bunch is the RIIR crew (ripgrep, ruff, etc.) that produces high quality re-writes with amazing performance and safety, the other bunch is who re-writing tools without any added value.
I am not sure which category MRSK fall into, there has not been enough details from them to justify this particular re-write.
You might think of it as NIH, but I didn't at the time that I wrote it. Because I saw a gap in the ecosystem that wasn't being filled. grep didn't offer the UX of a tool like ag, and ag didn't degrade gracefully to behave like a grep. (And also wasn't as fast on single file search.) That gap is what motivated me to turn ripgrep from what it was (a test bed for Rust's regex crate) into what it is now.
There's a reason why you basically never see me criticizing people about "NIH." Because it's actually quite difficult to accurately cast that criticism. Saying things are NIH is like saying things are broken. It's terribly overused.
I want my text no wider than 80 characters, but I like the images to be potentially much wider than that. What is your solution?
Along with all the other answers here, given that this is a Rails adjacent I would say Convention Over Configuration.
I like a basic text oriented page myself but sensible defaults would be useful. If the user wants to override the CSS in their browser they can do.
Like, provision Hetzner -> pass the IP address to the tool + a Dockerfile -> your image Just Works on the server and restarts if it crashes.
From the MRSK readme[1]:
Connect to the servers over SSH (using root by default, authenticated by your ssh key)
Install Docker on any server that might be missing it (using apt-get)
Log into the registry both locally and remotely
Build the image using the standard Dockerfile in the root of the application.
Push the image to the registry.
Pull the image from the registry onto the servers.
Ensure Traefik is running and accepting traffic on port 80.
Ensure your app responds with 200 OK to GET /up.
Start a new container with the version of the app that matches the current git version hash.
Stop the old container running the previous version of the app.
Prune unused images and stopped containers to ensure servers don't fill up.
https://gitlab.com/stavros/harbormaster
SSH into the server, run the Docker container, give it a config file with the images you want to run, and it handles everything else automatically.
mrsk does require a ruby install on your machine, tho.
If you have an image that can run multiple things, like a rails app that can run the app process for web traffic by default, but it can also run job workers with the right command, you can provide the cmd in the mrsk config. You can see this in the jobs role in the example: https://github.com/mrsked/mrsk#using-different-roles-for-ser....
This is MRSK. It's just that rails now comes with a dockerfile. And your service need a health check.
MRSK won't handle your load balancer, SSL, etc.
Btw if you are interested in Rails config for Fly.io, I have one: https://businessclasskit.com/docs/how-to-deploy-rails-sideki...
They stated 2 reasons.
Deployments to K8s are slow.
K8s is hard to setup on bare metal.
¯\_(ツ)_/¯
I was just watching the video DHH put online about this new project and it was quite good seeing copy pasting IP addresses to YAML files still excites some people or think that this is somehow on par with a serverless offering to any cloud company.
If you're building a simple app, using Functions as a Service along with serverless document based storage solutions will enable you to have a very cost effective cloud deployment which includes so many things out of the box like the ability to scale globally without having to change your application architecture. For my small hobby projects, this amounts to a few dollars a month per application as the only cost which isn't purely consumption based is disk space for the database. I built an application for a small company (around 10k MAU) which costs them around $40 / month using these tools. Yeah, they could be on a small VPS somewhere with the same performance characteristics for the current user load. But then I would have had to build in all of the things I get from managed solutions myself. I haven't had to think about patching my server OS in years.
They didn't say that they're replacing their k8s setup for Hey explicitly, but they go on to say the reason they wrote MRSK is that they like the advantages of containers while getting lower complexity and being able to use their bare metal hardware. It seems like they want to go in that direction.
Toss https://aws.amazon.com/ecs/anywhere on your own hardware and let someone else wake up at odd hours to worry about the container images making it on to the host.
This tool covers our use cases so far, and is easy to reason about. Ergonomically, it's very similar to Capistrano, which we're all familiar with.
It wasn't too bad to setup GitHub actions -> AWS ECR -> AWS CodeDeploy -> AWS ECS -> On-prem hosts via ECS Anywhere and that's worked well.
We'll see how many years it will take them to come back to Kubernetes or equivalent.
And the argument k8s is too complicated so build our own looks really bad from an engineering perspective.
When you look at the issues: https://github.com/mrsked/mrsk/issues/44 you can some see trivial shortcoming that has been solved for years by other solutions.
There’s nothing wrong with multiple projects in a space. It’s a big market with different niches.
1. Moving our stuff out of the cloud without going back to static hosts. 2. Giving new rails devs a tool where they can deploy their application easily, in a modern fashion.
Both of these are not so large or complex that you must use hostnames in configs instead of IP addresses. I will note, that most of our internal configs, do in fact, use hostnames rather than IP's. But judging a tool because an example used an IP address seems shortsighted.
There are plenty of things in mrsk to discuss without fixating on that.
I move companies between clouds and on-prem to cloud, cloud to on-prem (even though a bit bigger one than 37singnals) and I could use tools like Ansible and Terraform. In my experience when a smaller company starts to writes tools that solve the imaginary problems of the CTO they are not focusing on solving customer problems and this usually ends badly.
>> 2. Giving new rails devs a tool where they can deploy their application easily, in a modern fashion
Could you share the comparison chart of tools that you considered? What thought process led you to believe that this is an unsolved problem and requires a new tool? Genuinely interested.
>> But judging a tool because an example used an IP address seems shortsighted.
How should I judge it by than? Reading the code? Figuring out how to hold it right myself?
Like writing their own web framework in obscure programming language?
Which caused the biggest spike in CO2 production for recorded human history while making operation teams rage-quit their job over undefined method [] for nil messages?
Disclaimer I love Ruby and Rails to death.
Where did anyone say we don't use other tools? Chef + Terraform are wonderful and still in use for us.
> Could you share the comparison chart of tools that you considered? What thought process led you to believe that this is an unsolved problem and requires a new tool? Genuinely interested.
Your phrasing here and in your previous quoted reply leads me to believe you're not genuinely interested. Our environment is not your environment, our experiences are not your experiences. No, I can't share a chart of other tools that were considered, but off the top of my head, Capistrano, various CI integrations, Github Actions, etc.
We're a rails shop. We're going to look at tools in that area. We're not going to go dig into dagger or garden.io or something that causes us to have to conform our dev/deploy environment to a mental model that adds more friction for our developers.
> How should I judge it by than? Reading the code? Figuring out how to hold it right myself?
It's not that complicated of code, only about 2k, total, IIRC? I mean, you could read it. I'm baffled that you're choosing to compare a structural design flaw like the iPhone antennae issue to the fact that a configuration example in a new tool used an IP address. Go off, king.
Wow. That’s uh, that’s certainly a decision.
> They're the same service, because they're running the same image
If something purports to “run my containers” and “be an alternative to K8s”, I’d kind of expect it to run whatever it’s told. Whether running identical containers on the same host is a good thing or not really depends on your architectures choices and trade offs. Blindly being like “you can’t have this because reasons” makes this whole tool a giant non-starter IMO.
On the other hand, React is too complicated if you just need a bit of JavaScript on your mostly HTML/CSS page... So now you can use Hotwire if you like.
I don't see what's so bad about that (same things as MRSK and k8s)
in a world where backend is a single binary and frontend is a single html file…
who needs containers or complicated devops?
If so, what do you perceive as the gaps between Dokku and Platform.sh? I ask as an employee of a PaaS (not Platform.sh). So I'm curious for discovery and learning.
Other contenders are CapRover and Coolify that I just recently got to know here.
I guess it would make more sense to contribute to one of these projects than start a new project.
What PaaS are you working at?
If you want I can hook you up with an extended trial, beyond what's offered on the website. I'd love your feedback. We aren't open source, though we have considered it. Maybe your feedback can help push us over the edge. My email is henry AT aptible.com.
Same offer goes to anyone here. Feedback would be really valuable to me personally.
For what it's worth, "lots of new interested customers" is nowhere near the top of the list of Bad Problems To Have In Business, I think, so they're probably fine medium term. Just the growing pains and possibly unwanted pivoting -- and monetizing the freebies -- might be an issue for a bit.