The Tech Stack of a One-Man SaaS
panelbear.com
panelbear.com
The second thing I’ve done is avoid containers. VMs work fine for many types of applications. That’s especially true if provisioning and configuration is mostly automated. A product like Google Cloud Run looks interesting for a few low-volume backend services that I operate. Containers are a good fit there because it simplifies operations rather than increases complexity. The core infrastructure will remain VMs for the foreseeable future.
I began doing Ruby and Ruby on Rails (RoR) development in 2006. Ruby had been around for awhile but the growing popularity of RoR was driving a lot of changes in the ecosystem in those years. It was a fun time but I was working for a large company that allowed me the time to keep up with everything. I see today’s frontend and container ecosystems in the light of that experience. I’m now more content to wait it out and let the dust settle as much as I enjoy learning new technology.
Best done as an employee is spot on.
Sometimes that might be your employee, and sometimes that might be you.
I guess it depends on where your passion lies, and what your objectives are?
I imagine that achieving one's financial ambitions in a relaxed and enjoyable work environment is great - irrespective of which part of the stack one enjoys tinkering with the most.
I usually assume a role like engineering manager and learn from the teams feedback, challenges and commits
Making money, getting tax deductions and sometimes research credits
Just build in what you know.
Yup, I agree.
The apparent difference in our situations is that you know React and everything that comes with it. It seems like that'd be a good choice for you.
I'm not proficient in any of the currently popular frontend architectures. Gaining that proficiency and moving a 10+ year old product to a SPA type architecture would be a massive investment. I enjoy learning new things and pushing things forward but I can't justify that investment when it comes to the frontend. Instead, I invest in other areas to move forward in.
My old and boring stack works just fine on the VMs that it runs on, and and continues to support me and my family financially with little to no downtime and decent work/life balance.
Of course I'm aware of the cons of programming in PHP versus other languages, but after 20 years of coding in it, I like to think that my code isn't all that bad.
Still, I'm reading some comments here and feel like I'm waaay behind the tech curve with my use of such an old stack, and am somewhat scared to admit it.
Your comment brought me relief :)
Tell us about your SaaS. What were your challenges early on? What were your hardships? And what were your lessons learned? What would you do differently?
I'm a private guy. But I will say that it's a Greek startup, birthed in my English dorm room (Comp Sci grad; taught myself programming at age 11 by reverse engineering GORILLA.BAS; shout-out to all you DOS 6.22 babies #486dx2 #duneiichani #zerocool #snowcrash #bbs #qbasic #mode13h #nehe #0x5f3759df #lamp #vi).
> What were your challenges early on?
From a life standpoint, it would have to be student loans. But in crisis lies opportunity, because by working to pay these off, I programmed database integration layers between a bunch of local companies that ultimately led to a sweet SaaS system.
From a technical standpoint, there were many fun challenges, such as 2,000+ days of uptime with multi-master replication. But the biggest challenge was probably getting an early version of Quixote and a cranky SOAP protocol to talk to a bunch of SAP EDRs; that's when I learnt people skills.
> What were your hardships?
The usual stuff... paying the rent, feeding myself, no free time. It was all systems go back in the day, and dating wasn't really an option; but I'm thankful for lots of genuine friends, who welcomed my intermittent visits.
From a business standpoint, the biggest hardship was my constantly wrestling with the idea that I could do anything but I couldn't do everything. Once I concluded that exercise I found peace, and then everything else just clicked into place.
> And what were your lessons learned?
1. There are multiple SaaS strategies. One such strategy is to go super niche. Emphasis on super. 2. Going super niche doesn't scale, but it will allow you to prioritise free time - whilst living a steady, balanced lifestyle. 3. To go super niche, build your SaaS in such a way that you can run it from your mobile phone. 4. Always keep an eye on your phone, so that when the need for customer comms arises, it flows with elegant efficiency. 5. The quality of your communication with your customers is your competitive edge. Excel at this to justify your pricing. 6. Optimise all your business processes around minimising your phone's notifications - the goal is to maximise your free time. 7. At your leisure, routinely check in on your customers for a friendly catch-up. This enhances your competitive edge and bargaining power. 8. Always add value to your customers, and if you can't add value, walk away. You'd be surprised how much you can earn and still add value. 9. Enjoy time. Discover who you are. Explore life with your family and friends. Exercise your body and mind outdoors. 0. Help others. Volunteer your skills and resources. I advise several entities and it brings me great joy.
> What would you do differently?
Perhaps I'd have picked a name that was easier to recall, and easier to spell.
But when you go super niche, it doesn't really matter what name you pick.
Yeah there was a second's regret as I saw my submission's resulting format.
An interesting trick by the devs, this temporarily available edit window.
But fair is fair. I like it.
Good job HN devs.
#486dx2 #zerocool #gorilla.bas #mode13h #0x5f3759df #lamp #vi
PHP has come a long way. With v7 (and soon v8) and with Laravel programming in PHP is not only a delight, but also extremely robust and fast.
For the front-end, jQuery to Vuejs is not a big leap. You can sprinkle it around to test waters and gain confidence. Such a pleasure to work with.
The SaaS I run isn't mine. Back when you were starting your SaaS, I was probably spending all of my time chasing the latest shinny tech :) Some of that was worthwhile but some not. The experience taught me to slow down a bit. Sometimes I wonder if I've slowed down too much. We'll see.
If it pays your bills and gives you the work life balance you want, why do you care?
Running on the perpetual treadmill of cutting edge tech (especially with JS and containers where everything new is old every other month) is nothing to strive for. I know plenty of engineers who are very up to date on the latest trends but have no personal lives or families and are basically married to their job. I don’t envy them, I envy you.
I fell into the shiny object trap so many times before. This time, I stuck to what I know inside and out and jazzed things up where I could. Long live the monolith.
VMs (even self-updating ones) can be built easily using templates with tools like Packer.
And without containers, a lot of things can be done very simply, often more easily, using modern versions of systemd. You get ad-hoc namespacing, ro storage, resource (CPU/network/memory/I/O) limiting, CPU pinning, private networking, logging and much more by the power of a single line more in the .service file. Throw a tight selinux configuration on top of it and you got a pretty good per-service, secure "container" on your system. Additionally, you get smart .service dependency handling, service auto-restarts, notified restarts if you care to implement it, scheduled tasks, and so on in a multiprocess environment that makes you just sad thinking back to having to resort to stuff like supervisord.
The amount of complexity and interdependencies these "here's my stack" posts describe are always a huge cognitive overhead for running a one-person SaaS.
Heroku costs more, but there's a reason for it. If you're even remotely making money from your product, and are alone in running the show, it's a no brainer to drop a couple hundred dollars a month to eliminate 99% of your ops headaches: the documentation, the add-on's, the fantastic UI, decent reliability, and ease of operations are totally worth it. Literally the worst thing one can say about Heroku is the cost, and possibly, the inability to build certain complex architectures (which most single-person SaaS services aren't).
Is that what you're doing? "Standard" dynos and just a few of them are working for you, I assume, for that budget? In my tests, with a Rails app, "standard" dynos are looking surprisingly slow, possibly unacceptably so. Very curious to hear about other people's heroku formations if anyone wants to share. What platform you're on (Rails or something else), and what your response times look like, would also be interesting.
These both are k8s under the hood but with everything extracted away to a heroku level, but much cheaper. Additionally, they both provide static site hosting like netlify.
I'll take a look, it definitely looks interesting, thanks!
For user-facing work, I use performance dynos.
Heroku _is_ expensive, but in exchange, you don't have to worry much at all about the ops side of things.
YMMV, it may not be suitable if you need to be profitable each month and can't afford to spend the baseline cost of, say, $1,000/month.
But once you are using a single performance dyno, you are unlikely looking at a bill that's only "couple hundred dollars a month".
Thanks for confirming you use performance dynos for user-facing stuff -- I thought I was going insane discovering that it didn't look like standard dynos were suitable for that for me -- is "everyone else" using them though? The heroku docs imply they are indeed... standard (that's the name), and the performance dynos are for unusual performance needs.
Which is not what it was looking like to me.
Based on my current investigations, I agree being prepared to spend $1000/month is a better back of the napkin to-start-with estimate.
Which, sure, is quite possibly still a value compared to the number of hours you'd be spending setting up and maintaning something else. Quite possibly! But it's not "a couple hundred dollars a month".
For many projects this is more than enough. However, in my case I would be paying 2-3x more if I was using Heroku.
With Kubernetes, in case I wanted to deploy new projects or even spin up a "testing" env, I can use the same stack/cluster, and not have my costs increase.
I do realize that this is not for everyone. But for me it just made sense, and enabled me to work on new features faster than before.
The vendor whom we replaced = somewhere above $5000/mo (just infra)
Our final, most costly, GCP spend, including support contract = about $2000/mo (this involves a bunch of inefficiencies caused by developers unable to live with container - or dynos)
Expected Heroku cost, based on their online calculator= starts at $3000/mo, probably more
Expected lower time needed developing on Heroku = negligible
So it wouldn't make sense for me to use Heroku (a tool I don't have much experience with), and pay more.
In terms of use of my time, to be honest I currently spend way more time sending emails, updating docs, and on "marketing" than on devops :)
I would never recommend Heroku for my employer or for any company that has real production loads and a budget for administrators that know how to set it all up: Heroku does seem to get prohibitively expensive at larger scale. But for hobby projects that get couple of hundred requests per day this is more than enough, and if one of them gets too successfull, I'll just pay someone for a day or two of work for migrating my code to other hosted platform - and since I never use vendor-specific tech, this shouldn't be too hard to do.
We’re an Aussie company, and a large percentage of our customers are in Australia. Having servers only in the US and Europe mean that Australian customers speeds are slower than they should be, and private spaces are far too expensive to warrant it.
It has almost all the same features and runs at almost any GCP location.
My SaaS has grown from $50/month costs to over $2k/month costs on Heroku over the last few years and I'm always eyeing AWS and others as a cheaper alternative, but honestly the ease of mind Heroku gives me is more than enough to cover the costs. I've had a few "hugs of death" over the years when posts went viral and scaling up both in quality and number of servers was literally just a few mouse clicks and then I was back online. Other times, when e.g. Postgres fell over from heavy use, their support brought it back online for me free of charge (and quickly), and offered a list of helpful suggestions that I could do to keep it from happening again.
I see paying extra for Heroku like paying extra for a devops guy. If you're comfortable enough in AWS to handle things in the heat of downtime, it's probably worth moving because it's significantly cheaper. If you aren't, then Heroku might be a pretty invaluable resource even when growing into the upper echelons of cost.
Docker containers avoid the lock-in of a particular platform or CLI, and they can also be used as cheap CI/CD with multi-stage builds within the container.
If you're on Heroku or Google App Engine, you deploy your application code. That's it.
You can choose to build on a scratch/blank image just as easily, which means nothing except your app itself is in the image.
Or you can build an image from scratch without any distro and then there's nothing to maintain.
My one complaint is that their support seems to have gotten worse over the years - they used to have great support engineers across the board, but have hired poorly recently, specifically in off-shore markets, so depending on what time you send in a ticket request, you may get someone who doesn't fully read your ticket or comprehend the situation, and just fires back a canned reply.
This usually gets solved if there is enough back-and-forth as they then seem to escalate it properly, but I have noticed that the initial reply to support tickets are not as good as they used to be.
Use Heroku.
It's super easy to get started. And when you reach $2000+ in monthly bills, hire me to move things over onto dedicated or EC2 so that you'll get 10x that performance for the same price.
As for the actual language, I think PostgreSQL + Ruby Backend + JS Frontend is still the easiest way to get started.
All those great architectural ideas don't matter much until you have your first $5k monthly in reliable revenue. And by then, you'll have a completely different perspective than what you think you need now.
With JS, do you mean React or jquery or literally JavaScripting the DOM?
I use this stack and I'm pretty sure I could pump out any SaaS business in about two weeks with it. Host on Heroku, use whatever analytics system you want.
I agree that GraphQL + React will be pretty and work well. But for one-man start-ups, the focus is usually on "looks good enough and kinda works".
So I believe I'd go with Bootstrap & Glyphicons, just to get results faster.
My site is a firebase powered SPA, so the only thing Heroku has to handle is the initial pageload....I would think that the standard 1x dyno would be sufficient for that....which is not very expensive...
could also rope your friends/family into it if you want your app to be "distributed", might require a lot of work though especially if they are not very technology oriented.
Say, you’re a business, and want to set up your own e-commerce site. Whether it makes money immediately or not, is not a concern, as the business will pay for it, and absorb the initial loss. And the development costs to build the site are already in the $50,000+ range.
So with all this upfront expense, why don’t the business just go with EC2?
Unless this is for some silly hobby website of yours, that you have no clue will make any money or not. Then in this case, maybe your suggestion is valid.
Just the discussion on this post alone, is a goldmine for anyone trying to figure out what worked, and what not for others. This is why I love HN.
To clarify a point about my article, I think everyone should do what works for them. Just because my unconventional stack for a solo founder works great for me, doesn't mean it will do the same for you.
Your mileage may vary, and it all comes down to tradeoffs.
Also, I'm sorry if I haven't been able to reply to each question, this post got way more attention than what I expected.
But I'm happy to answer specific questions, feel free to ping me on Twitter: https://twitter.com/anthonynsimon
Cheers!
We're also hiring :)
I really like your product. It's crazy fast, might solve one of my issue, is super simple to use, and the UI is gorgeous.
I've a few questions though:
- how do you compare to other privacy focused analytics tools, such as https://usefathom.com/ or https://simpleanalytics.com/?
- it seems like you just launched your SaaS a couple of weeks ago. Why didn't you launch it earlier by removing some not strictly necessary/MVP features, such as the security login emails, the ability to manage multiple websites, a less scalable but simpler architecture ... ?
- how much time have you spent developing the product? I've been working on my personal SaaS for more than one year, planning to launch it in the next few weeks. I'm still working at my 4/5 full time job, and I really wonder how you managed to deliver this beautiful product while still working fulltime. That would be a wonderful opportunity for blog article BTW.
Great work!
About your questions:
1) Fathom, Simple Analytics, and Plausible are great. They have done an incredible work too. Panelbear has already most (if not all) of the features you might find in their offerings. However, I do intend to dive into more advanced functionality, while keeping it simple to use. I will be talking more about it when I feel confident about what I can deliver.
2) I launched in September, but only with a few posts here and here. It was also a lot more barebones back then. I try to release a new feature each week, and traffic has been slowly ramping up. I guess I just haven't had much time to promote it, but I plan to keep writing blog posts if there's something interesting.
3) I try to limit work on Panelbear to 2 hours per day. If I want to be in this for the long-term, it's gotta be sustainable, and fun. I try to leverage most of the tools I already know, even if unconventional like Kubernetes, to help me focus on shipping rather than learning a new tool.
Glad you like the design :) I took courses in design when I was studying, but I actually struggle a lot with it. It has helped me appreciate just how much work goes into a good design.
Thanks!
There are too many dependencies and too much complexity here. Kubernetes is overkill for 95% of applications, especially single founder SaaS businesses. Clickhouse may make sense for an analytics product but caring and feeding it is non-trivial (see: ZooKeeper dependency and all the problems you get once you move to a distributed database).
Most people would be better off with a single beefy machine running Linux, Postgres and whatever web app framework they know. The whole "cattle not pets" thing is fine once you obtain product market fit and your product needs to scale up. At that point you'll have time and money to do it - before then you're just wasting cycles.
I'm working on a side project and it's literally a bunch of .py files in a folder. Libraries include fastapi and sqlite3. I cannot think of a reason to use container orchestration over a "single beefy machine" in almost any use case.
However, it just made sense for me due to the simplified operations, and re-using most of the learnings from my full-time job. It has enabled me to move faster in shipping customer-facing features.
If I wanted to start a new project, I can leverage the same stack and be up and running in minutes. Which means I can take advantage of my cumulative efforts for any future project.
It's definitely not for everyone as you said, but it might be a good solution for many :)
As someone who is mostly in the same boat as you, you have got a strong point in terms of favouring tricks you know best.
The amount of code I have had to throwaway made me not want to invest in things that require lots of resources to run and maintain (in other words, things I'd not be comfortable with managing all by myself).
I almost always default to managed and serverless products: This won't necessarily be a winning strategy for everyone to say the least as it brings its own complexity to the table. It is pragmatic; however (like you call out), to trade one set of complexities (one that I am comfortable with) for another.
Using a tool "just because" it worked for others, seems to be what often drives many decisions in this industry. For example, would you go for the 2,200+ microservice architecture from Uber[1] just because they say it solved all their problems? They probably have their reasons, and they are unlikely to be the same as everyone else.
I know my stack as a solo founder is unconventional, but it's what has worked best for me. And I'm happy with my choice.
If serverless does the trick for you, then focus on that, and keep building the features that your customers actually care about.
- Much cheaper, reducing bill from >100$/month to ~40$. Important for early stage startups - More performance, easier scaling. Found that my application was much better suited to run in K8S, but this is definitely specific to my use-case - Consolidation of resources. Except Postgres/Redis, everything is in my cluster, simplifying management and maintenance
For my business, this was a great move, but as others have said, I wouldn't recommend it to everyone. K8S is a great and powerful tool, but also is very complex.
The downtime issues with DOKS are directly related to the resources allocated to the control plane, which is not set up in a HA capacity. The resources assigned are directly related to the size/number of nodes you use. API-heavy applications can very easy knock out the control plane, and in that situation is takes it a (relatively) long time to recover on it's own (I would typically see ~4hr before it became responsive again).
Their support team is able to modify the master resources for a given cluster (to assist with recovery), but the turn-around time on that shouldn't be considered "production ready".
At this point my advice for DOKS would be:
- Are you using very basic, out-of-the-box Kubernetes to host "apps"? You will probably be fine, but be sure to have a back-up.
- Are you planning to use Operators, or anything that heavily interacts with the kube-api? I would recommend not using it, or over-provisioning your cluster (which would very quickly offset the advantage of the "free" managed masters).
I know that they are working on fixing some of these reliability issues and I have hopes that it will be more stable shortly.
At this time I have a "stable" cluster which (unless their support lied to me) had master resources manually increased by their support team after the 3rd incident of it dying. I haven't had an issue since then.
Does k8s auto scaling count?
That was my case too. It was mostly when I installed the prometheus-operator that things would go south, even if the nodes themselves were perfectly healthy and underutilized.
I like the autoscaling and the infrastructure as code, and price for DOKS is good.
In the beginning I was even running everything on a one $10/mo “cluster” (cheapest DO offers). Even Postgres and Redis, all inside the cluster.
As I got more customers, I needed something less risky, so added more nodes, and moved the database out.
But to be honest even the 1 instance $10 cluster was fine.
Cloud Run, as you said, is pretty great. In my other start-up, we're using Cloud Run and Cloud SQL, and again, Cloud SQL is the biggest cost (~95-99% of the bill)
- PM for DO Kubernetes.
If you need more than that, the privacy@ email is the right way. https://www.digitalocean.com/legal/pipeda-faq/
Is this true? At my last company we wasted a bunch of time every week dealing with the accumulation of ad-hoc changes in different environments creating different behavior. When we pivoted to a continuous deployment model (we also moved from EC2 to ECS/Fargate), all of this time went away as did all of the guess-work and fear of deploying to production (because production very rarely deviated from lower environments with respect to behavior, because all environments were reproducible). It also enabled us to stand up whole environments with the press of a button, which made it a lot easier for developers who were interating on infrastructure (we had a lot of async ECS/Lambda workloads). I'm sure if we were smarter or had the right kind of discipline we could have made pet EC2 instances work, but we benefited a lot from moving to cattle/containers.
NOTE: I also suspect that cattle/EC2 is a lot harder than cattle/containers, but that may also just my inexperience with the former. I would really like to know how to do reproducible EC2 instances properly, but from the research I've done, it seems like one has to reinvent a good chunk of Kubernetes.
Maybe I'm misunderstanding the whole "pets vs cattle" thing but you don't need to lose repeatability or give on process altogether; it's a false dichotomy. Just don't dive headfirst into the massive complexity that is "scalable" devops - you can still use terraform or docker compose or whatever to deploy your application and provide repeatable development/staging/test infrastructure.
Getting it off the ground is one thing, keeping things reproducible so you can stand it back up reliably if the instance goes down is another thing. You need to know what are the relevant changes that people have made to the instance over its lifetime in order to reproduce them on the new machine. Maybe this is not what people mean by "pets vs cattle", but they seem pretty closely related if nothing else. I guess the thing I care about is "reproducibility" and for all of its complexity, Kubernetes makes the happy path pretty clear in this regard whereas with EC2 you need Ansible and a whole bunch of playbooks to reproduce things like SSH, logging, monitoring, process management, etc that Kubernetes or ECS or whatever give you out of the box. And if you find that you need to scale beyond one or two instances (not everything is a CRUD webapp, after all) especially autoscaling, then it seems like the EC2 story becomes even more complicated (you now need something to automatically invoke Ansible with the right playbooks for each node). I'm probably biased by my experience (aren't we all?) but it seems like Kubernetes or ECS/Fargate are simpler in these regards.
Have you ever been working a solo project that needed autoscaling? If so what was it doing that was so resource intensive per $ earned that it was still a solo project?
I very nearly launched an event analytics app at the beginning of this year but ended up just rolling one for my own internal use.
Are we talking about pet servers or kubernetes here? Jokes aside, I think you're underselling how complex Kubernetes is and overselling how complex logging, monitoring, process management, et al are. You don't get any of those things with kubernetes "out of the box" unless you've got an ops team or someone's got a year plus experience running one cloud provider's version. It seems really easy cause we've got one click clusters and a bunch of great UIs but all the time people spend chasing down problems and wrangling the extra complexity of k8s in production gets papered over because it's the hot new thing.
The bash history for the setup above fits on a single page and since the advent of Nix and now Terraform, repeatability hasn't been a problem. The harder part is discipline among coworkers but Linux user/group permissions go a long way towards making it readonly for debugging purposes while still giving fine grained control over servers.
I'm pretty sure cloud providers give you logging and monitoring out of the box (as well as a bunch of other stuff) and Kubernetes itself is a process manager with its own "SSH" (kubectl exec) and so on. I'm not wed to Kubernetes either--you could use ECS/Fargate or whatever. But to do something comparable in an EC2 environment requires a fair amount of experience just to get to get the basic things that Kubernetes (or ECS) gives you out of the box. Of course, if you're well-versed in VMs already then I don't think you'll get much value out of a container solution for simple workloads, but if you're not familiar then I think container solutions are easier. But again, I might be biased by my experience.
That is an issue a disciplined one-man operation won't have since (generally) they have a Dev and a Prod environment, the changes would be made in Dev first then duplicated in Prod.
If you have 2+ developers, you have environment drift like you experienced because (generally) Dev is the local environment on the laptop or otherwise a distinct environment for each developer. So now you have 3+ environments you need to sync and then discipline starts to fall apart as you have to communicate every change.
And the disciplined single dev documents their changes. That is why it works in a 1 man shop but not at scale, there is always someone who is unreliable once the group gets large enough.
K8s is such a huge mental overhead that if it's not already naturally in your head because you've had extensive experience (or apparently born geniuses) no one with limited tech budget should ever do this.
Running a business on a single node is not advisable. If your scale is that small, or uptime doesn't matter, then sure go with that. If you're running on multiple nodes, you're going to need a way to schedule containers to your cluster one way or another. Scheduling and orchestrating is where complexity get introduced - Kubernetes wrangles this complexity.
You can run a lot on a $5 server. I would probably run 3 $10 servers for a small business at least, which feels like still within the realm of a cheap budget for business infrastructure costs.
As far as Kubernetes complexity goes, I think this is a self-perpetuating idea on internet tech communities. I don't think it compares to the complexity of actually programming the web apps themselves. There's like 6 resource types you have to be familiar with to work with Kubernetes, and most of that will just be Deployments (where containers actually get scheduled and run, health checks, etc). The others are for configuration and routing.
Configuration getting messed up sounds more like a k8s problem than anything else. In any case, if you aren't using git and don't know how to at least back things up you really should be asking yourself if a tech company is right for you, as a one man/woman operation. There are countless other ways to make money online. Etsy, Craigslist, Shopify, etc.
> until a node dies and you need to recreate it
The way to solve both of these problems on a single node operation is to create a snapshot/clone of your machine. At that point you would really just need to worry about the IP address changing and/or updating DNS when restoring the image. The best part is you can take this "master" copy and deploy it as a staging server. Or even horizontally scale by throwing a load balancer on top. You can even get your VM exactly how you want it locally and then deploy it to DO, Linode, Vultr or whatever.
I don't know if this was to intended to directed at me, but all of the tools mentioned imply using them with git or other version control, hence Github actions / CI. Kubernetes does not spontaneously mess up your configuration, well intentioned developers do. Willingness to learn new-ish technology puts you in a much better standing for running a tech business.
The snapshot/restore method is fine if you don't plan on updating very often, but this isn't a good alternative to infrastructure as code. I would make a base image that bootstraps your node to a cluster in Packer and then hand off all the deployment of nodes/DNS/Loadbalancer wiring in Terraform. Then just snapshot your database volumes as normal and deploy with your orchestration.
Of course, you can't most of the time run production apps on a single machine, but my general approach has been to add the least amount of infra overhead - in our case it has been elastic beanstalk. It has been reliabile and zero touch for us for years now. I have heard of some people saying EB is good till it works and when it doesn't you're screwed - my backup plan for that case is to continue keeping that EC2 instance running, and generally make sure you can run your app without too much hassle in a fundamentally different stack at any given time.
In the end, by depending on RDS and EB we have never had to worry about infra or scaling in our team. This is in stark contrast to every other team I've seen or worked with, where upgrading your eks version, or certificates expiring or etcd getting unresponsive or environment variables leaking between multiple prod and stage or someone accidentally deploying to prod because they didn't specify the correct name space in their deploy script or a hundred other reasons made their k8s deployment a nightmare. You can argue that this is a sign of bad training or practice, but I'm working with the same engineers on this stack that's so much more foolproof so when can we start blaming the tool a bit as well?
The problem is back ups. You've got to figure them out yourself. And there's a decent chance that you do it wrong leading to potentially catastrophic data loss.
IMHO, managed DB + S3 equivalent and then doing your own server is the sweet spot. You don't have to worry about data loss and it's still pretty cheap and flexible.
If you think a one man band and his ruby app can probably go pretty far with self-discipline, but if I'm choosing to do any sort of IaC I'd probably recommend looking at k8s as a) a faster solution, b) a more versatile solution and c) largely vendor agnostic solution.
Point being, I'd probably recommend folks learn to use k8s before they invest time learning lambda, EC2, or ECS, but if you're a small team and you already have lambda, EC2, or ECS, and it's working for you, then it's hard to argue for change.
Agree. Blindly recommend this to anyone is not a good idea! It comes down to author's skill sets, and if the author is truly interested in tech, or just some guy want to implement an idea. Personally I am full stack and know how to make server scale without using Kubernetes. Since I've being using commonJS since it's inception, it is pretty easy to write web apps that scales using a light weight server layer like v8.
Did you like it?
Point being that it takes very large scale for the principles people refer to when they say "cattle, not pets" to kick in -- even when you're actually raising cattle. But by the same token, automated reproducible configuration is a huge win whether you have 1 server or 10,000.
This phrase assumes a unique product, while many have probably been done before.
If you're a professional engineer and you're starting a tech side project with the goal of shipping - use what you know.
I do a lot of work with node, cloudfront, s3, and React for my day job. All of my side projects are built on node, cloudfront, s3, and React. This works well for me because my goal with side projects right now is to be productive, not to learn new tech.
An important clarification is that I use the tech that I use in my day-to-day, not what my company uses. Just because I have coworkers who know kubernetes back-to-front doesn't mean I do!
Also, there's nothing wrong with doing side projects with totally crazy/new tech stacks as a way to explore and learn new tech. I just think it's good to be really intentional with whether the goals of a project are educational/process-oriented or results-oriented.
I would agree with others who have opined that this is overkill. You couldn't pay me to use Terraform or Kubernetes for my SaaS unless it was a much larger team.
My stack usually consists of this: A framework I am comfortable with that removes a lot of boilerplate (happens to be the one I know best in the language I know best), the database I know best, the memory store I know best, Turbolinks for load times, a theme from ThemeForest (with the commercial license), self-hosted analytics, Amplitude, and Heroku.
The author mentions "lessons learned" regarding Clickhouse and Kubernetes. I'm not sure what their original goal was when starting their SaaS. Usually mine is financially motivated more than learning things.
What I'm really saying is that just go with what you know best. Having to scale is a good problem to have. Pre-optimization is the root of all evil — asking "is x the right stack for y?" will always be yes when you ask in the x subreddit.
Stop thinking and build.
More broadly, can you help me with an opinion on my present struggle? Here goes:
I have a lot of ideas, a couple of which solve problems for me and I think are even _good_ ideas. However, none have shipped because I'm spending an inordinate amount of time building out what is essentially my own SaaS template. Is it worth investing the time in doing that, or are some of the commercial SaaS templates out there (sjabloon, bullettrain, etc)
I prefer Python/Flask, but I'm trying to learn Rails now, since a lot of the small saas ecosystem seems to exist there.
There is a meaningful amount of boilerplate that remains for any stack if you're using it for a SaaS. It may not hurt to invest in a commercially-available solution that takes care of billing.
For some products (super-hyper-MVP), I've gotten away with having a `companies` table with a `state` column which was either `active`, `inactive`, or `past_due`. I would update this manually from Stripe every day. When it became a chore (aka "I am spending an hour on this every day and my time is better spent elsewhere") I only then wrote code to handle it.
There are solutions like Recurly or Chargebee that take a way a lot of pain on a platform basis. Those may be helpful in managing some of the complexities, but they come at a cost: financial, vendor-based, flexibility.
It's important to keep in mind that your ultimate goal is to get customers validating your idea. While it's important that you conduct a meaningful transaction in exchange for solving their problem, that does not mean you need to handle every single last billing case.
Accept their money, let your solution solve their problem, and repeat.
If you want to talk more my email is in my profile.
To clarify, this has already helped me to ship features to my customers quickly. I actually spend little time working on the stack. Last time I touched the infrastructure components was probably a few weeks ago.
Panelbear was originally built and launched in a weekend ($5/mo DO instance with SQLite), and I've been working on it for the past 3 months. I have been releasing a new feature every week. That is, in addition to my full-time job, so most of the time we're talking about 2 hours per day tops.
I'd also like to clarify that I am not suggesting this stack should be used by other solo founder SaaS. I actually advice against it in my post :)
I just wanted to share my stack, as it has worked really well in my case, and I thought it'd be interesting material for discussion.
I didn't mean to refute what you shared. In fact I was reinforcing some of your points. Doing things the cheap way is arguably the best way. Learning new tech while building is good, but I wouldn't recommend new tech to build a SaaS.
My concern was copycats. Some starters reading your post may have this feeling that "because Panelbear did it with Clickhouse and Kubernetes and Django, that's the only way to do it." What it really should read is "because Panelbear used what he knew best, he was able to focus on getting customers; that's the only way to do it."
That's what I was going for :) Now I realize that I should have highlighted that part more on the blog post.
The last startup I was at hired a DevOps engineer to come in to completely containerize our simple Rails monolith at a time where they were just simply way too small to warrant it.
Development velocity plummeted. Deploys started to take half an hour. Features weren’t being released. The professionals our app was servicing lost trust in the engineering team.
All in the name of checking off a box on the CTO’s list of “things that Google does”.
With docker you have to worry about how containers talk to each other. Which at a bare minimum means exposing ports and adding a network layer. Suddenly you find yourself in the weeds with docker-compose and debating whether or not it's actually suitable for a production environment, when they recommend using swarm instead. Then you find out swarm is on shaky ground and now you're hearing about nomad and ten different other solutions just to avoid k8s.
Infrastructure-as-code is a wonderful idea. But the reality is it's a full time job. There will always be orphaned containers and things that need cleaned up and automated. And if you don't, then your environment will run out of space or RAM and all devs will be sitting there doing nothing.
Building docker images also takes time, especially if you don't optimize the image. The simplest deploy workflow I've seen was a git hook (post-update, I think) that ran a few commands when the repo is updated. It'd really be hard to make up for the efficiency of that workflow after messing about with docker for days.
I know it seems like a lot of tools, but most of them are set up once and forget. Except for the occasional round of dependency updates. But even for that I have automated dependency/container scanning to detect vulnerabilities, and get notified if updating an image is recommended.
Most of the stack consists of "platform" tools, such as the ingress-nginx, external-dns, or cert-manager. So it's stuff I take with me for every project. It's like a personal Heroku I've been building, but way cheaper and fully customizable.
I've been iterating on this stack over the years with various other projects, and also learned a lot from running Kubernetes in production at my full-time job, so it's not something I'd advise for someone who just wants to deploy their MVP for the first time.
Kubernetes/Terraform and friends are really useful as a solo-dev if, and only if you have the required experience with them.
"I know it seems like a lot of tools, but most of them are set up once and forget. Except for the occasional round of dependency updates. But even for that I have automated dependency/container scanning to detect vulnerabilities, and get notified if updating an image is recommended."
The docs make it look really complicated, but it comes down to enabling the service on the Terraform config.
Also for app dependencies, I use bots to open PRs on my repo whenever an important security fix is published for one of my dependencies.
The ops environment still looks overwhelming to me, if there's a way to do it sustainably without a really heavy-weight ops infrastructure, I haven't figured it out. Like, something that to me requires like a full-time job just to do ops, although maybe not if you've spent some years learning the ops tools, but then same same.
You can get really far with the managed services from your cloud provider.
If you're already familiar with Docker containers, just that already gives you plenty of options. I think every platform now provides a way to just run containers behind a load balancer with a few simple steps.
Yes, maybe it's a form of vendor lock-in, but if your worry is acquiring customers, and growing a company, Kubernetes is probably not helping you in this case.
This is an insightful article. A learning for me was how the author managed to switch multiple cloud providers even while being a one man SAAS.
Kubernetes is overwhelming because of 2 things: Overload of terminology, and Distributed systems day 2 operations.
Author succeeds so well because he has a starter stack (install, CI pipeline, secure, monitor, scale) maintained as code, and it works every time.
For someone starting new, the learning curve is high. Bootstrapping is easy, but day 2 on kubernetes is hard. Benefits outweigh is learning curve imho.
Specific to DigitalOcean kubernetes API server issues, we have rolled out some resource provisioning improvements for newly created clusters very recently. Also making ongoing improvements to backend monitoring.
I have also had a fair share of issues with AWS services in the past (Athena, they took care of it a few months later), and GCP too. And I understand that no product is ever perfect.
Django has just so much stability, it was an obvious decision for me to go with it. And that's speaking as someone who has mostly worked with NodeJS/React for the past 5 years.
I love the pythonic principle of "There should be one obvious way to do it". This helps me cut down on decision making which is something quite annoying to deal with as sole-tech-founder. Now I get to focus on the business, and technology is just a matter of execution.
Another rule of thumb is to have a setup so basic that articles like this are no fun to read or write. The limiting factor in a one-bro setup is complexity, and this guy maxed out about one paragraph in to his description.
It's pretty cool that there's so much going on in this space, but is it really wise going into the market with seemingly the exact same features that Plausible, Simple Analytics, Fathom, etc. offer? Even Cloudflare recently entered the privacy-friendly analytics market with a free product, so it seems that market is getting really crowded. Then again, every (commercial) website probably needs such a solution, so maybe the market is big enough.
I wonder though if these cookie-less solutions can give people the insights they need? You can do only very simple overall statistics and funnel analyses with this approach. Maybe that's enough for most websites though?
Yes, I understand I'm entering a crowded market, but Panelbear is still in the early stages, and I'm trying to understand what problems my customers have that I could solve / how much value can I bring to them.
A crowded market also means more options for you as a customer. Specially when it comes to privacy-friendly tools, not only analytics, I'd argue that there's not enough options yet.
I initially built it for myself, as I was not fully satisfied with the existing solutions. And I do intend to offer more features than what you currently see, but unfortunately my time is limited and it's going to take a while to bring Panelbear to the feature-level I'd like it to be :)
- GKE kubernetes - Managed Postgres (CloudSQL) - GCS buckets for file storage - Cloudflare for DNS - Countour for ingress - Keel for automating deployment updates - Mailgun - Sentry (errors) - Node-RED various little automations such as healthchecks
Planning to introduce Elasticsearch too, so far I have been testing the operator and it seems to be pretty good quality.
From maintenance perspective GKE is great, as long as the bills are paid, no need to login there pretty much ever. As a one man company this means a lot, I try to never spend any time on ops.
One piece we've struggled with is just mypy doesn't seem to play too nicely with Django. It definitely feels like a world of difference between the type support we get in our React app with Typescript and mypy with Django. Would be curious what OP's experience has been with that.
They're all server-rendered HTML with a little ajax sprinkled around the place where required for interactivity. I use Materialize CSS to give me a leg-up on the design side but the next thing I release will be on Tailwind UI.
They're all written in Go using a simple MVC-ish pattern. I write a lot of integration tests and a few unit tests.
I use Google's hosted PostgreSQL database. It's not cheap but it's not expensive either and it's been rock solid. The frontends are all on Cloud Run as of a few months ago which has been wonderful. Super reliable and fuss-free. Takes care of all the application logging, etc.
Task queue stuff is handled by a library on top of Postgres. It won't scale to a billion jobs per second or anything but it handles load just fine. If I ever start pushing it's boundaries it's all pluggable so I can switch it back to SQS or PubSub or whatever else.
The task workers run on a single-node k8s cluster hosted on GKE. A lot of people grump about Kubernetes being "overkill" but it doesn't have to be complicated. A single YAML manifest defines your service and your environment variables, you tell GKE to make it happen, you go play in the sunshine. It takes care of restarts, logging, full disks, etc etc etc all for you.
I use Amazon SES for emails. I had some real problems with deliverability until I ponied up for a dedicated IP address and now it's fine.
I use Freshdesk for ticketing. I don't love it at all but I don't yet hate it enough to replace it with something else.
* AWS EC2
* .NET Core Backend (Business logic + Kestrel Web server + Let's Encrypt + JSON Web API)
* NOSQL Database: The filesystem
* SPA InfernoJS front-end (similar to React)
--
The application is performance intensive and must deliver responses sub 100ms at at all times. Each 'request' from the user does a lot of work. I optimized to get all this to run efficiently on a $20 / mo EC2 instance, with no other costs
At various points, I've looked into opening a single-owner LLC to hold some small side projects I want to put on the internet for fun, but the number of decisions and amount of overhead has always seemed pretty daunting.
It's a position I couldn't agree with more. It might be one of those mistakes you can only truly learn from by making it yourself, but honestly, while DIY is awesome for learning its only place in a business is at the core of what your business value is.
One way I think about it is anything DIY is surface area for either innovation, or mistakes. To mitigate the risk of making mistakes doing things DIY you'd better do it only in areas where innovation could actually benefit your core business value and give you magnified upside. If you innovate outside these areas, I guess it's nice, but you've taken the risk of mistakes for no significant upside. Not to mention a whole load of work invested.
I personally just use one ore more VPSs on DigitalOcean, which scale pretty well at first and if you get enough traction to need "infinite" horizontal scaling, it might be better to only focus on that once you have the money to hire some sys architect/admins.
I haven't read comments yet, but I bet some people would criticize the tech stack as "too complex" or "not perfect" or "not even mention my favorite tech"...
1. Typically tech stack starts very simple, then evolve into something a bit more complex. It's normal.
2. If this tech stack is the author's most familiar tech stack and it gets jobs done, then it's a perfect tech stack for the author & the business. There's not a tech stack that works well for every developer, every company, every business...
3. Treat this tech stack as a data point. Find other data points on the internet. And you'll come up with your own "perfect tech stack", just for you, for your specific business.
fsKube provides as Desktop Web UI to setup entire AWS EKS Kubernetes cluster, with all the components and Django framework template with a full DevSecOps enabled, along with monitoring and operations.
The whole thing within 45 minutes. Support for Spot Instances, Graviton Processor and AMD AMI coming soon, will reduce already cheap EKS prices further.
Watch the full demo here https://youtu.be/Nt5yowdwm5Q
But as a techie, the magic to me no longer lies in the product building, but in the ‘finding the first 10-100’ customers part.
Have there been any posts on that lately?
I've read several of these SaaS walkthroughs and it invariably seems to follow this pattern:
1. Author has a blog with a large following on some topic.
2. Finds niche in that topic.
3. Makes a product and pushes it via their blog.
4. "Incredibly" their product gets tons of customers.
Getting noticed is one of the hardest parts. Just having a great product is no longer enough.
For every Basecamp, there are probably tens, or even hundreds of similar competitors.
Can I name a few? Maybe.
What about the top 10? Probably not, and that's exactly the problem.
In the linked post, they mention both Django for the backend and Redis as a cache. Django has caching built-in though, so until you need more complex caching logic, you probably don't need to reach for Redis.
For your front to back integration: I suggest you look into django-graphene and reacts apollo-client. That small addition to my stack has, at least, doubled if not tripled my churn speed. Throw in something like Django-channels and you get a modern full async websocket capable reactive framework :)
This is one-man who made > $1million a year.
If you don't mind me asking, roughly how much do you spend (time) on this every week? How much of that is split between development, operations and product or business development?
Since I have a full-time job too, my time on this project is pretty limited, usually no more than 2 hours per day.
That's why I tried to automate and save time as much as I could using Kubernetes operators, well supported open source projects, and most importantly, tools I already feel comfortable with.
I don't have hard rules in terms of how I distribute my time, but recently it may have looked like this:
- 60% of the time I'm trying to understand the problem I'm solving (talking to customers, researching, evaluating features).
- 20% working on features, and issues.
- 20% promoting: writing blog posts, documentation, reaching out on Twitter, HN, Indie Hackers. Helping out customers integrate their website / understand how something works.
Of course, there's tons of other smaller things here and there, but those are the big "buckets" of my time.
Once I know what to work on next, I tend to prioritize as follows:
1) At the start of each week I pick 1-3 topics to work on next (for example writing a new blog post, working on a new feature, or even just writing documentation which I am going to next).
2) Each day when I'm working on my project I pick the most impactful thing I could work on, and start with that. I allow myself to explore a topic in depth, sometimes without a clear goal or deadline. After all it's supposed to be fun if I want to stick to it for the long term.
3) I try to respond to customer emails as soon as I can, and understand why they're giving my product a try. That's my gauge for what to work on the following week.
It's interesting to see how you split your time and how much thought goes into all this. A followup question, if you could;
> and most importantly, tools I already feel comfortable with
What do you do in situations where X tool would be 'better' than Y tool, but you're comfortable with Y tool? Do you do a comparison of the time/effort it takes to learn AND use X tool vs. Y?
GAE's offering of free (and auto-renewing) certificates also makes things cheaper.
I want the minimal devops/sys admin work possible and have all my effort go towards adding features.
If I’m successful I could cut costs later and switch to something cheaper.
Is that a good strategy?
Scala (akka http, slick, Twirl) Postgres AWS EC2 Jquery/Javascript
https://www.formgadget.com/ for those curious.
I always use API-generation code for my products. The real value i bring to customers is my database, not API.
About pricing:
- Do you have any tool for cloud pricing tuning/estimations ?
- Are you using specific tool to track all your expenses in general ?
I do track my expenses on a sheet manually, using a ballpark figure. At the end of the month I compare it with my AWS bill. If something seems off, I investigate it further. But I don't try to predict exact figures, more like expected ranges.
I do miss the fixed-cost nature of DigitalOcean/Linode. I didn't have to worry about sudden charges, you pay $20/mo per node and that's it. Lots of bandwidth included.
But for the most part it's fine, as long no bot starts bombarding your services with useless requests, egress costs tend to scale with revenue in my case, as my product's pricing is based on usage.
I also have Cloudflare in front of my AWS stack, it has blocked a few small attacks already, but nothing serious yet.
I still have a lot to learn, but IMO nothing beats struggling with something for a long time, and pushing along until you're no longer terrible at it.
I feel that most of what I know I learned by starting projects on topics I initially new nothing about, and sticking to it.
Regarding on the job learning, I have been working with Python/Scala for my full-time job over the past 3.5 years, and that helped too. Specially in terms of running multi-regional services, and the challenges that come with that.
We run several applications on Kubernetes using AWS (combined >2k req/s at peak time), and that was a big challenge for me from which I learned a lot. Specially when you're on-call, and one faulty node brings down the entire cluster, that fails over to another region and repeat :)
I mean, K8s? Come on!
Scala was great, but unfortunately it made it difficult to onboard new team members into the project, so development suffered. Also the rest of our stack is mostly Python, so we couldn't use a lot of the common tooling, and libraries we had built for other projects.
We migrated it within a month, and it's been running for almost a year without issues.
At some point the service even recorded 3k+ req/s at ~200ms 99p latency. Yeah, maybe not Google scale, but more than enough for 99% of the businesses out there.
This service handles about 2.5 billion requests per month from one AWS region. We're a small team, and that's only one of the services we support. Which is great because we don't need to spend all of our time optimizing it.
There absolutely are performance pitfalls, and Python is less power efficient than other alternatives running some kinds of workloads at scale, but in a ton of environments the perf difference doesn't matter.
But while I too had in my head "for web dev most of the real work happens in a database of some flavor", I recently realized that was not true for my app, probably hadn't been true for some time, and probably isn't true of most Rails apps (possibly not the same for non-Rails web apps?). If you've properly eliminated n+1 queries and other inefficient querying, I find that my apps are spending only 20 or 30ms waiting on DB results, and a couple hundred on CPU tasks to render HTML.
I know people are going to reply with "That's because Rails is slow," but I'm not sure that's true for what we're talking about compared to similar Python platforms, I'd be interested in seeing numbers for other real apps. In the Rails community too, people still repeat the assumption "most of your response time is spent waiting on the DB" -- but I dont' think it's actually true anymore (it may have been once).
Python, Ruby and friends are bad for page speed metrics. REST response time of ~200ms vs ~10ms for our java backends
That said, hundreds of milliseconds sounds slow by an order of magnitude or more for html rendering, even if all the work is done in a batteries-included framework for a dynamic language, and it doesn't jive with my Python experience at all. Do you mind me asking what kinds of tasks are taking that much time?
But rather than get into the details of my app, and parts that need optimization (whether in my local code or in Rails or in ruby), I'm more curious about the overall concept.
Are you sure that most of your web app's time is spent waiting on the DB? My suspicion has become that this is conventional wisdom that is not actually true of most apps anymore. But I could be wrong. Or I could be right only for Rails and not python because Rails is slower than typical python, or something. I am curious to find out.
Positive. It's my first tech job though, and we use C# at work to the extent that matters.
The kind of Python I write off the clock probably isn't a good example of waiting on the DB though. It's usually a thin wrapper around C or assembly (or calling into such a library via numpy, networkx, etc) to do something horrendously expensive that I absolutely would not want to run in vanilla cpython. That said, when I do ordinaryish web-related stuff in Python I'm looking at well under 20ms total elapsed time per call, which is why your ~200ms html rendering time stuck out to me.