How Fly.io and Tailscale Saved Notado
notado.substack.com
notado.substack.com
Or why should I target a bunch of different cloud APIs to make my tool consumable when I can target one?
Why target any cloud APIs at all if your product have a small amount of users? You don't need to scale up/down fast at if you're just at 3000 users. Throw it on a Linux box, write a systemd service and be done with it.
Most companies have small amount of users and don't have rapidly shrinking/growing user bases, meaning their infrastructure need is not rapidly shrinking/growing neither, in most cases.
Hence, Kubernetes being accidental complexity most people can do without, rather than necessary complexity.
I get why people say k8s is complex, but there really isn't another great option for running a containerized service based architecture, maybe nomad
But again, what specifically are you able to do in Kubernetes that you cannot do in Nomad? You mentioned it's not as "functional" but seems you're unable to actually specify what you mean by this.
> Nomad is built by a for profit company while k8s is purely OSS
Not sure how you can claim Kubernetes somehow is "more OSS" than Nomad? Both are under FOSS licenses (Nomad: MPL, Kubernetes: Apache) but both are mainly maintained by two big for-profit companies (Nomad: HashiCorp, Kubernetes: Google). They are both the same in this regard, both are the "same amount of OSS", whatever that means.
This is deeply untrue of Kube, it's maintained by lots of companies, and is in a totally separate organization than Google with distinctly different governance. Kube goes out of its way to ensure its not dominated by one organization and has stats on this. I haven't looked at the Nomad stats but I imagine its nowhere near that and still under the Hashicorp umbrella and pretty much only works with Hashicorp stuff.
> But again, what specifically are you able to do in Kubernetes that you cannot do in Nomad?
The big thing for me is API extensibility. The Kube CRD is one of the more powerful abstractions to hit infrastructure in awhile. The ability to create your own controllers and offer consistent APIs to an end user is imperfect but incredibly useful.
Other thoughts:
* Kube is supported by every major cloud provider.
* I'm not a fan of HCL, some people like it but I prefer Kube's approach of decoupling the templating from the core app.
What cloud APIs are involved in running Linux on a VPS? Backups for your VPS are often one click to set up, and the only vendor-specific service you need.
If k8s is ____, fly is ____.
I cant even compare fly to heroku, almost the same but different underneath.
I one day aspire to have the cajones of ilrwbwrkhv.
For the reference of other potential solo devs, if done right, Fly.io will still be cheaper than the Vultr 5 dollar micro instance (or equivalent) with an incomparably better DX.
You might be able to save a little, you might pay a lot more. Hedge your bet and use vultr.
My gut feeling is that if someone is going to struggle with optimizing for a relatively fixed target like Fly (the billing is extremely predictable compared to some other offerings out there, if not completely fixed), the chances are that they won't be cut out to do double time taking on what is effectively a Linux sysadmin role.[1]
[1] I may be completely wrong
Actual billing caps are very hard for cloud providers to implement with the HA distributed systems clouds use for billing, and would carry a significant performance cost. Alerts are the best you can do if you take the variable costs.
This means that I can't misconfigure my way to a three-figure-plus bill, which is hard to find in cloud providers.
(Even if we all acknowledge that it is one motivation for many tech fads).
Honestly, if this is on a resume the next question should be — tell me why you needed such a system…
Just as Magpies like their shiny trinkets, I like my shiny frameworks.
Like in sqlite or something like json files?
I remember hearing that Oscar (a health insurance startup whose main differentiation was having a decent app) ran their own mesos cluster for some reason creating tons of cost and complexity with little to no reason (also operated tons of microservices with startup headcount).
Oscar (no longer a startup; IPO'd last year) ran their own mesos cluster for the same reasons anyone uses k8s today: to help teams deliver in parallel. "Startup headcount" can mean different things to different people, but the company employs hundreds of software engineers, and has been around for over a decade. Kubernetes was not an option when they hit those scaling constraints—but now that k8s has established itself so firmly, Oscar's moving to a managed k8s setup.
My experience before Oscar was mainly in a Spring monolith—which, yes, made some microservice overreaches pretty clear in my team's corner of the codebase—but even so, it'd be madness to coordinate all the org's development through e.g. a Jenkins instance and a bunch of ansible.
It's odd because this blog post also announces the end of the free beta for the service. It goes in depth about how they jumped through a bunch of hoops and introduced more complexity to the service because they were annoyed that DigitalOcean raised their prices. And at the end they're asking you to pay $5.99/mo for it?
The other point that I didn't really cover in this article was all the monitoring goodies you get for free with Fly; definitely not something to be overlooked as a solo dev.
The medium-term plan is to move the database to Fly as well, but I wanted to share a way for people to do a partial migration to Fly if they are not quite ready for a full database migration.
As for using a Procfile manager being a "hack", once we are freed from the constraints of Docker containers and back in the realm of VMs, I think running multiple processes again becomes the norm.
I don't quite agree with the "one machine (or machine-like abstraction) => one process" thinking in this regard.
However, people do seem to love the fly.io developer experience. It's a testament to fly.io that the auther seemed to enjoy spinning up on fly.io.
It's also a testament to DigitalOcean that it wasn't painful or expensive to partially migrate away from their services.
This is an excellent summary!
> ... which of course could have been done on DigitalOcean.
> However, people do seem to love the fly.io developer experience. It's a testament to fly.io that the auther seemed to enjoy spinning up on fly.io.
It could have been done, but it would have required me to introduce more moving parts and complexity to handle deployments of updates (do I keep the same VM and SSH in to update? Do I build a fresh VM with the latest version and then cut over the traffic? etc.) It's pretty hard to beat "fly deploy" in my opinion.
Similarly, the vertical scaling story is much more streamlined on Fly.io - it's also pretty hard to beat "fly scale vm".
I've mentioned in another comment on this post about how great the monitoring goodies that Fly gives you for free are.
Another point that I didn't highlight in the article itself is that Fly.io is currently not collecting bills less than $5/month (this might change in the future, I don't know), and if you have a resource-efficient service (or services), with the numbers that most solo devs working on side projects are talking about, it's not difficult to stay under $5/month and have the excellent developer experience to boot.
> It's also a testament to DigitalOcean that it wasn't painful or expensive to partially migrate away from their services.
No complaints on this point; I'm also still very happy with Digital Ocean's managed database services.
I would recommend that you go with something like Hugo[1], throw it on an S3(-compatible) bucket and be done with it.[2]
[2] This is what I have been doing for my personal blog and for my wife's professional website for many years and I am very content with it
Instead, I would now use Astro[1] for any static site or blog.
YMMV, but if you don't already love golang and their weird template language, Astro is just HTML and TypeScript and is overall simpler and less time-consuming to understand.
(I mean it will get hella complicated if you want it to, and let you integrate React or Svelte or blah blah, but out of the box it works great for generating static sites like blogs, or just regular-ass websites.)
[1]: https://astro.build
As might be expected, Notado saw a huge surge in registrations, saved content and large batches of comment imports today; I'm happy to report that this "hug of death" was handled without breaking a sweat.
Thanks to all of you who provided your feedback, thoughts and engaged in good-spirited critical discussion with me here today. I don't take your time or your words for granted.
Unfortunately I'm starting to reach that "had enough internet for today" point[1] so I'm going to be signing off now.
I have already received a number of emails from people interested in specific parts of the Notado stack and how things work. If you would like to email me to ask about something specific, I'd be very happy to receive a mail from you: hello at notado dot app. I hope I can respond to your messages either bilaterally where possible or in further technical articles where appropriate.
De khuday p'aman!
The article currently links to a deprecated Angular.js project with the same name (https://github.com/geddski/overmind)
it was very confusing at first cause I relate "vpn" to something like nordvpn or express vpn when I hear it. but no tailscale is a lot more than that.
The Okta groups sync is a beautiful cherry on top.
Like PulseSecure garbage that gets mad because you don’t have the right antivirus installed, when in reality your version is newer than the pulse client recognizes.
As you can tell, not a network guy here! I could be totally wrong.
As those static IPs are in a reserved range, they will never clash with a public service like a website and Tailscale does not mess with your regular traffic.
You could make things more complicated by setting up exit nodes and exposing subnets, but that’s the general idea. So simple, it’s brilliant for their standard use-case.
many people don't understand the implications of vpn'ing into their home devices. being able to access their network shares "over the internet" in a safe way. being able to access all of their random self hosted stuff without needing to be at home without needing a reverse proxy. all the while being easier to secure and safer than internet facing alternatives. these are things that people will actually understand that aren't just arbitrary abstract ideas.
You have a crowd of people (like one of those massive crowds you see in Tokyo etc). And you are both at either sides of the crowd, and you each have a can with a string connecting you to each other – and I explain that the crowd is the public internet, and tailscale allows you to create that string across the whole crowd in a way that nobody in the crowd can ever listen to you guys talk, even though the string literally passes through/over tens of people in that crowd.
If someone has a photo like that, I'd gladly use it :)
I know enough about all the nuts and bolts under the hood with Tailscale and that makes it feel MORE magical to me, that with all those things to deal with, it still just works. And it's no compromise (uses wireguard, etc)
The last time I saw that level of magic was in a product called Hamachi. It Just Worked™ more than a decade ago. LogMeIn bought them and ruined it.
But with Hamachi I didn't know how it worked under the hood.
Tailscale is very much more like a real VPN than those other things that are pretending to be VPNs.
VPN's are usually used to create Virtual private networks. Extending your LAN across several sites using software, or granting you access to your LAN when you're not physically located close to it.
It's hard to imagine why the meaning has changed as much as it has, because even though a consequence of this connection can be that you exit to the internet via your destination networks internet connection; I feel like this isn't the original use case at all, similar to how you can use a chair as a ladder.
> encrypted proxies
I suggest "encrypted tunnels"
Because the supporting infrastructure was historically called that, and it stuck. Just like how pocket computers are called "phones" because they evolved from telephones.
Yeah, that's a really good point.
I guess at least we have the modifier "mobile" and "smart" to differentiate the different types of phone (up to and including the pocket computer).
I don't think there is a written equivalent of a VPN which is only acting as an exit node.
In fact, TOR describes an overlay network, and that's how projects like Nebula also describe their functionality. Granted, Nebula is more like a VPN than "VPN providers" also.
"VPN providers" provide a service for accessing the public internet as proxies, tunnels, etc, for evading region blocks, getting around privacy destroying things, and all that stuff. I won't knock their value. I've use them when needing internet over an open WIFI access point. I'm generally not involved in activist activities nor otherwise in harms way, so I don't have much else use of them.
But I do have use of a virtual private network. As in the core idea of it. I have a private network at home. It's not publicly available. I use Tailscale, currently, to access it when not connected directly to my home network (as in, not at home). My portable device becomes a member of a private network that has access to my private home network. This is a VPN.
I really think VPN was (incorrectly) chosen for secured proxy services because it's a simple and familiar name. The marketing using the terms "virtual private network" came far later to describe a way to secure your privacy. But it's diluted the actual technical term.
I work on open source projects that solve many of the same problems, but it's a chicken/egg problem, and Fly.io and Tailscale are building the infrastructure that will show people what's possible.
I hope that one thing that potential solo devs take away from this post is that you can get very far without even needing to think about touching k8s (especially if you are already very comfortable with k8s).
After all, HN and Pinboard comfortably serve many more users than our side projects likely ever will without bringing in any kind of heavyweight container orchestration platforms.
Besides this, I'm not sure I'm in the best position to compare the latency and performance of a service running 100% on a Digital Ocean VPS vs. a Fly.io micro VM. Hopefully someone else can chime in with their experiences on this thread.
Having worked on firecracker and with firecracker, it does a good job in the networking area.
Note that there's no multiqueue implementation available, so depending on how many connections and data you're passing through, it may start struggling if the workload is high.
I don't know the specifics of how DO sets up their VMs, but I assume that they're not doing any fancy networking optimizations, so you may end up with similar issues on both setups.
> Users will always have access to all of their previously saved content regardless of subscription status.
> There will always be a quick and easy way to export all of your content
I like the respect you're showing your users. I would feel more at ease using a service that I know I could host myself, but this is probably the next best thing.
I managed my own SQL DB once upon a time and I can tell you I would rather have it done for me. 10 years on from when I set it up, my provider was going to get rid of the hardware and couldn't VM migrate me, so I took it down. No urge to manage this stuff myself.
Back in early 2020, Meilisearch did not have a Rust client library, but it did have a Go client library[1], and there was nothing comparable to lib/pq[1] in the Rust ecosystem which would allow me to create a listener on a table.[3]
Go is generally my "fallback language" when something would not be practical to do in Rust; it has a very nice, mature ecosystem, and as long as you aren't condemning yourself to interface{} hell, it remains in my eyes a perfectly capable and reasonably ergonomic alternative for well-defined use cases.
[1]: https://notado.substack.com/p/how-notado-syncs-data-from-pos...