Scalability is overrated
waseem.substack.com
waseem.substack.com
These days I often advise would be founders to start with doing their idea manually with just the minimum amount of tech.
Maybe just a google sheet and answering emails and a handful of “customers.” If you can’t give people a good experience with a totally custom personal experience, they’re not going to like it any more when it’s automated and scalable.
SW is weird in that way, we can basically build houses that nobody wants to live in. Thing is you probably were building the right thing, structurally speaking.
My rule over time has become "structure follows complexity" i.e. Anticipate complexity where possible but don't apply the engineering structure until the complexity really exists. If you built an MVP house, don't add the second bathroom until you've got customers complaining about deadlocks on the first.
It's tough because that basically runs counter to the Spirit of an Engineer, which is just to imagine the grandest well-thought-out + organized thing and march toward creating it.
The bonus of having two decades+ in building SW though is you start to see the patterns and realize that the same fundamental needs drive everything, so the intersection of building something great with customer needs becomes less of a mystery and more of a "how fast can we juice this thing". At that point I think scalability prep is like compound interest.
Very recently I’ve come to think of my architecture style as Jazz. Like the old trope about listening to the notes he isn’t playing. But also the improvisational nature and the flexibility to adapt to new information.
This is such a wonderfully concise way of describing something that I try to teach people.
I have come to use the phrase "no regrets". Leave things out, but do it in a way that you won't regret it, if you need to add it later.
Another phrase that I read somewhere else was, "Weeks of programming can save you hours of planning" (pretty sure I read it on HN, and I wish I had kept the link). The point being that when you decide to leave something out, or when you decide to intentionally do a half-assed job of something, you should still think about how someone in the future would go about efficiently fixing it.
If you do it right, there's missing functionality, but little or no technical debt. If you do it wrong, you miss out on functionality and also take on technical debt. And I think that's the subtlety that you speak of.
There’s a list of things I won’t try to add on later. It’s smaller than it used to be, but the shiv I brandish at you for suggesting we cut those corners is also much scarier.
Some things you can cut corners and save time.
Some things, when you cut the corners off you also cut your fingers off.
We have all promised that we will “fix it later”. But whatever we do now is what’s done. Fixing it later is a promise whose resolution is often not in our control.
Or as Feynman said, the trick is not to fool yourself, and you’re the easiest person to fool.
Or a personal favorite, unattributed: I know the guy who made this plan [me] and he’s full of shit.
Jim Highsmith has the best theory I’ve heard in the subject. His opinion is that the reason we keeps having the same unwinnable arguments is because we think in terms of solving problems but the real issue in front of us is resolving paradoxes. There are no neat answers that fit in a short essay box on the test everything depends, and everything depends on everything, and something is always changing.
What the PTSD reactions come from is other people making decisions on behalf of the rest of the team without consent. But like most things in software it’s a 2 dimensional problem not a 1 dimensional one. The trick is to delay irreversible decisions as long as possible and get all responsible parties involved.
The flip side of the coin is that reversible decisions should be made as cheaply as possible. For instance my “Kobayashi Maru” solution to the bike shedding problem is that the team should have budgeted three colors of paint into the bike shed budget, picked green for the first coat, and moved on immediately. That $200 is far cheaper than the time and importantly energy that would go into discussing this at a meeting with senior staff members present. If you start thinking about all of the $5000 conversations you have to solve $2000 problems it’ll drive you mad. I almost recommend you don’t look at it, for your own safety. Almost.
This is exactly right, but the irreversible decisions are often horizontal slices of software design that underlie other decisions - thing like authz, IPC, API design etc. If these questions can be identified and resolved up front, the cost of building vertical slices - features - goes way down. You're not tempted to reinvent the wheel, you can focus on the business logic.
What I'm saying is: it's worth spending time planning the foundations of a product. That includes the technology choices, but also the high level logical layout of the whole product. If you approach this foundational stuff like any other project, then if you're doing it right, you'll "get all responsible parties involved", like you should.
But defer the detailed planning of domain logic until you're actually ready to write it, and remain flexible with the plans you do make. "Plans are nothing; planning is everything".
Because like the GP said,
> the later you design something the more info at hand you have to make the design fit better
I find it remarkable how often engineers just leap in and start coding, and worry about the foundations later. Or never. I suspect this is one reason why rewrites are so common. If you start with bad foundations, it's often easier to blow up the whole building and start again.
To be fair, leaping in and starting coding allows you to discover the problem better. The problems start where you get stuck with your initial design and don't adjust/remake it once you explored some ideas, maybe found some new along the way and found out what doesn't fit.
That is of course problematic if you have deadline driven development because rewriting last 2 weeks of code might seem like total waste even if then-better design gonna save you much more even in near feature.
Not rewriting code when design flaws appear will lead to rewriting it eventually later and more expensive.
Of course, all of that depends on how much you know about domain and the problem. Exploring existing solutions and their flaws or having really detailed requirements might be much better approach that doesn't require starting to write code.
https://news.harvard.edu/gazette/story/2007/04/frank-h-westh...
Not sure what the actual source is.
I tried to bottle it in this document: https://pcblues.com/assets/pdf/approaching_software_projects...
Don’t put a hyperkinetic person in charge of something thinking that you’ll call their bluff.
The potential need to scale and add features will inform my design decisions, but they will not direct them.
That's responsible, future-oriented development.
I'm not sure if Spirit of an Engineer is a book, essay, or some prior set of principles I'm unaware of but if it's not (and even if it is), I tend to disagree here.
The spirit of of an engineer, in my opinion, is to solve problems and problems for people--that's what I and most engineers I've spoken with have been drawn to (ignoring those purely seeking a high paying profession here since were talking about "spirit"). When I was youthful I sought out technical complexity and grandiose solutions (I admittedly still admire it when it's done correctly). None the less, to me, most of it is wasteful and at the end of the day I want to build the most useful solution for people to make their lives easier.
Some problems require high complexity and glass cathedrals in terms of technical solutions, most really don't, at least most of the problems I've been exposed to.
And yes, engineers really do err on the side of designing for future problems and future users that often will never exist. This effect is worst on a rewrite of a system - look up the "second system effect". We tend to only truly get pragmatic tradeoffs the third and later times.
Make a bare bones version and see if it sticks, and go from there. Both the Progressive Enhancement philosophy and the Lean Startup methodology demand the minimal before building further. Ex: Will anyone click this survey link? Who knows but you'll learn real quick if you offer them $50. If they still don't, then it probably ain't gonna get clicked ever. If they do, then you found at least _something_ viable.
Good idea, you are welcome to help me hone my later chapters. For me the engineering spirit - especially in software - is being a creator/builder. At the core, someone who takes the Legos in front of them and creates with those pieces.
A couple people took my "grandest" comment as a nod toward scale/size/narcissism, which is reasonable, and I meant it more in terms of total architectural completeness - i.e. a system perfectly fulfilling all the requirements that necessitated its creation. A blade of grass is not "grand" in the primary sense, but it still is grand + magnificent by its own perfection in design.
I’d vote for SWE if we are taking up acronym namespace
i think that's just delusions of grandeur. Not everybody would be (or should be) building the sistine chapel.
There is however a pitfall that this approach seems to lead to unless you actively prevent it. Roughly the process is 1. You hire someone to take care of things manually. 2. As you grow, your devs are worrying about other things and so instead of automating you hire more and more people to handle the toil that should have been automated along the way. 3. Because these jobs are handled by one group of people and developers are working on customer facing features, the communication and prioritization necessary to fix the issues rarely come up.
Ignoring history is a guaranteed path towards repeating it.
For purer technology / product businesses, how do you do this, fundamentally? How would Google have manually mocked up their early product? How would Facebook? Github? Tesla for that matter?
Sometimes you really do just unavoidably have to build the product out before testing the market, and if it doesn't work, just accept the sunk cost and throw it away - and sometimes fail completely as a result.
I don't see this as a fundamentally solvable inefficiency, just part of how tech product startups work, and the very tradeoff that must be made to allow for innovation to happen.
there are still smaller pieces you can MVP to a smaller audience before launching it to the world.
> Google have manually mocked up their early product? How would
Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
> Facebook?
They had Myspace as an example so the idea wasn't exactly unproven.
> Github?
Kernel developers were already using the software, successfully, all (for large values of "all") they did was bring it to a wider market with a better UI.
> Tesla for that matter?
People don't get this, but Tesla's biggest achievement to date, isn't the cars themselves, but the factory that they're built in. There's no way to MVP an entire factory, but building a car in a bespoke, pre-assembly fashion is totally possible and totally doesn't scale.
If you're asking if electric cars were known to work, the first one came out in 1832. If you're asking about product-market fit, they keep selling cars they haven't made yet, just to gauge demand. Aka where's my Cybertruck!?
The hundreds of millions of USD as an interest free loan seemed more important than anything else.
> Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
But that isn't a mock up, it's the real thing but on a smaller dataset. If you're going to do the real thing anyway, why not run it on all the data you can?
After all, the throttling factor to release is in the engine, not in the dataset. If you're going to write the full engine anyway, there's nothing to be saved by limiting it to a subset of the data.
Because more data requires more cleaning and standardization (with more edge cases). It also requires a bigger scale to obtain and process.
You are right that some businesses scale better doing high touch customer service like this. In the case of Pilot, you have the lead sales guy and accounting domain expert (CEO) asking SaaS customers to schedule enterprise service sales calls with him.
Which makes total sense. What he’s not automating is setting up Marketo or Hubspot and Drip or Constant Contact and committing to some CRM system that is both impersonal and adds friction when you’re going for quality over quantity.
He could hire BDRs and CSMs or outsource and spam and set up AI and knowledge bases, and possibly scale up faster.
But not only would that take away the personal touch and competitive advantage, he’d lose the opportunity to educate himself on what real customers really need.
Not to mention all the time spent evaluating tools and setting up automation and negotiating contracts that lock you in to a specific process that might not be what you want 2 years down the line when your business changes.
Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
You should make architectural choices that CAN be scaled within reason, on demand and ideally incrementally. That doesn’t mean you need to build your app on Kubernetes and automate welcome emails, surveys, metrics, A/B testing etc. But you probably shouldn’t build on top of fundamentally non-scalable systems either.
This is provided you’re in an environment that can reasonably scale to 1M users or whatever, which isn’t true for all domains. Say B2B where that last B is only 10000 people world wide.
Unless you built something absolutely atrocious, scaling it vs getting it ‘right’ or ‘right enough’, is trivial. You couldn’t sign up to Facebook if you weren’t a college student.
Chatgpt overloads daily so what? When your app becomes necessary to enough people, bring in your k8s evangelists and shove into x cloud provider. But by god it doesn’t take 30 people to write a marketable mvp.
Chatgpt doesn’t have competitors. Customers can’t go elsewhere. Twitter had the same protection when they launched, there wasn’t a short form social network so they could have full day outages. Friendster would be the opposite, when they closed signups, customers just went to MySpace.
Scale goes beyond the ability to support traffic. It also means efficiently handle data/traffic. Heap is a good example, great product but they haven’t figured out to provide that level of analytics without significant costs which are passed on to the customer.
Everywhere? Sounds like you're cargo culting.
There's a use-case for it, and that use-case is when your load is so high you need the orchestration.
Right now I'm using cluster of 2 nodes (master, worker). No reliability, but convenience is good and we have very simple path to more nodes when we would need it.
But ... no one claimed it didn't, did they?
The claim was that the unnecessary complexity (of which Kubernetes was just one) from day 1 consumes valuable time that could be spent on making the product better.
Kubernetes might start consume valuable time if you would start to bring niceties. Like GitOps, CI, and so on. Those things are good but not necessarily justify time spent from the start. So you can start with simple deployment yaml which you can write with 10 minutes and apply it manually whenever you need to update it.
Kubernetes almost out of the box brings to you:
ingress. You don't need to set up your own nginx reverse router. This task alone could consume days of setting up and maintaining. I work with organization which spends 2-3 hours to update their nginx if they need new route. It must be done in off-hours because they break it every single time they touch it and then they need to fix it. Kubernetes ingress is a breeze of fresh air after this nightmare.
cert-manager. Again, I work with organization which failed at setting up certbot. Their certs break all the time. cert-manager just works.
zero-downtime rollout restart. This is absolutely nice thing which works from the box and hard to achieve manually.
kubernetes-dashboard. There're other dashboards as well. I think that lens is very popular although I didn't use it. It allows for people who know very little about kubernetes to interact with it productively. For example in my team we have developers who use kube-dashboard to view logs, to exec into pod and check things, to update deployment, change envs. They just push buttons in GUI, they have no idea about any yamls or kubectl. And it works.
And those things in my experience so far do not bring any unnecessary complexities. Quite the opposite. They solve lots of inevitable complexities for very little cost.
I'm not sure why you are getting downvoted. It would be great to hear of some of the bad experiences of those that disagree. Were they running their own cluster instead of a managed one? Or doing k8s with a team where nobody had any experience? Or run into some helm-chart spaghetti from hell?
Depends. No one at a unicorn says that, but I've seen plenty of startups that. The unicorn is the outlier, not the rule.
> I worked at one company that we couldn’t rack servers fast enough to keep up with our growth. Our competitor who was the market leader had to turn off the ability for new customers to signup to prevent their site from crashing under load. We ended up surpassing them.
Were they still in business at the end of all this? Because it turned out you made a bet, and it came through. The odds are against this, though.
> Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
A million in what time period, and what are the usage patterns? Because from what I've seen, as long as the backend isn't in Python, Ruby, PHP or similar, a single hosted server easily vertically scaled to handle more than 2m million users, because it a) didn't need to handle 2m concurrent connections, and b) usage patterns were infrequent enough (users interacted only every few minutes at most).
as an engineer this part I entirely agree with. Far too often CEO driven development leads to pivoting 1Month into 9 month projects, 9 months in a row. such that you have nothing deployed when you at least could have had 1 thing, but instead you have none.
This can be solved as much by discipline to stay on track as by choosing to reduce scope to 1 month ideas only.
Edit: (The load was easily handled by a single Heroku dyno. I turned it up to two dynos overnight, just in case, but it turned out to be unnecessary.)
Not that all spikes are low quality, or even that one on HN ones are. I reckon for some products they are.
We spent a fair amount of time building something scalable, rented some co-lo space, and in a couple of weeks had something we were pretty sure would scale to tens of thousands of users. We got . . . like a few dozen. Oops.
Never even fazed our sales guy. His pitch was something like: "We built this proof of concept system to scale to ten, twenty thousand users. Do you know how many we got?"
Customer: <non-committal nod>
Sales guy: "We got thirty."
I could never do sales, I just don't have that mindset.
(We had some pretty nifty technology for doing real-time messaging, it actually worked and wasn't a technical failure, but it was sure hard to get market share with just a bunch of APIs and some demo apps. After I left the place it was bought by a company that was in turn bought, and that's how I turned a ton of cash into 17 shares of Oracle, -spit-).
It's easy to say "scalability is underrated" if you've dealt with businesses built on hundreds of standalone PHP/Perl/Python/JS scripts with zero unifying architecture, where migrating a data schema is virtually impossible because of the state of technical anarchy.
Scaling is hard.
There was a brief moment after Oracle purchased Sun where I thought Oracle might try to engineer am Excel competitor on Open Office that had an easier migration to a real database (Oracle), but that dream died early.
Are there scaling issues? Version control issues? Absolutely! But again, that doesn’t mean that it’s not the best tool for the job.
It’s easy to mount the highest horse from a technical perspective, but as engineers it’s also our responsibility to be curious about what people are using voluntarily, and not just what we think they should be using.
A git integration in the editor, and a decent test runner. Some API hooks do Jetbrains can do something with it maybe.
The whole structure is called Open Packaging Conventions and it is implemented in `System.IO.Packaging`.
I think I was hit by everything. Easy stuff at first: Crlf issues, XML Apis, weird rpc apis.
Then, halfway through the project, the results had to change. Not only the order, datatype and headers (I actually overengineered the first version so those were configurable), but the format, duplicate on multiple columns (and empty fields counted as duplicate...). Worst job I've ever done. I'm also disappointed in myself tbh.
But now I'm a bit of an expert on excel format issue and limitations, and that already helped me.
IIRC he just optimized their sheets and set up a CRM for the things that had no business being stored in an Excel file and that was already very helpful.
Attempts at writing actual software to deal with the problem would have failed and everyone was acutely aware of that.
They eventually replaced me with a handful of their relatives, or at least it seemed. It was a lot of fun watching how many LOC one could "write" to request some trivial data from an endpoint. Luckily I was only golfing the latter half of that tenure so I have no regrets.
That's why you use something like a Rails monolith....but oh wait, Ruby doesn't scale well!
Look at Github, for instance.
If you're doing any computation or highly concurrent workloads then you will discover the performance issues with Ruby well before you outgrow your persistence layer.
It took more CPU than if we'd rewritten the core of it in something else, but it let us iterate the rendering engine itself much faster, and most of the expensive computations were done by extensions in C anyway (e.g. GDAL and the like).
Of course you can find areas where it's still not viable, but my experience is that if you start by prototyping in whichever high level language - no matter how slow - that is most productive for you, you'll inevitably find you need to rewrite far less than you might think to get it fast enough. But more importantly: The parts you end up rewriting will very often be entirely different parts than what you expected, because being able to iterate your architecture quickly tends to make you end up in a very different place.
With a kafka queue, and worker nodes, compute-heavy jobs are easily distributed across many worker nodes.
For many parallel requests, you load-balance.
If the PostgreSQL or mysql table is the bottleneck (persistence layer), well this is actually a design decision thats orthogonal to the programming language (PostgreSQL won't scale better with a JDBC ORM).
But the question is, are you trying to make your life miserable by scaling before exhausting other options?
Most applications can be optimised for performance by orders of magnitude, much easier than trying to scale them by orders of magnitude. Any problem is much easier to solve and faster to deliver when you can fit it on a single server.
Usually it is simple thinking that goes wrong. System is slow - add more hardware. But then it turns out developers did bad job and you could still run it from single small server only that someone would have to write code understanding big O notation.
Main point of big O notation is that there are things that implemented incorrectly will be slow regardless of how much hardware you throw at it.
The Silicon Valley world would do well to learn some things from finance. You’d never write a horizontally scaled order matching engine.
It wasn't successful....
Exactly. The author forgot the put the word "premature" before the word scalability.
I have a Java codebase that calls more constructors inside their constructor. So you have this massive god object that instantiates the whole project in side the constructor. If you want to run parts of it in separate threads you can't just take it apart. You first have to rewrite all the constructors.
“… Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. “ —Joe Armstrong, creator of Erlang progamming language
I don't think this is a problem with object oriented languages. You could certainly do the same thing in any programming language. All you have to remember is to keep your constructors as simple as possible. Passing in dependencies through the constructor is often an easy solution so I don't get the hate for dependency injection frameworks. The original XML driven iteration of Spring was pointlessly overcomplicating it but nowadays you can just define your "beans" in code which basically means the DI Framework just helps you set up a dependency graph and nothing more.
The bit that stuck with me (re-reading for the first time since 2008, I forgot how aggressive it was): "Scalability is not your problem, getting people to give a shit is."
And, funny enough, the people that didn't think about scalability, if they're smart, have typically moved on to something else.
Being overly speculative and ambitious early on is a great way to fail. Building on top of something past its limits and ignoring the (developer, customer, velocity) pain is yet another.
If your service opex breaks even at 1k users because you designed for capacity up to 100k users, and you actually end up with 100 dedicated super-sticky users, you'll resent the users you have as they bleed you to death.
If, instead, you build a system that less efficiently, but still reasonably, scales to 100k users by ramping up cost to support 100k users by 4-8x... but has an opex break even at 50 users... Congrats! you've planned for success and planned for failure.
Go get those 100k customers, then optimize. But don't wait to optimize an unscalable system until you already have customers. That's a recipe for lost customer trust and "ahead of their time" epitaphs.
OTOH, the smart people who thought about scalability and designed a highly complex system to handle the load of 5 requests per minute have also moved on.
Pilot's customer base probably looks very different than yours. The author says in the article that they are a "startup-focused accounting firm", so for their customer base which is likely startup founders or similar, the experience of talking to a fellow CEO probably "helps to 10x the customer experience" as the article puts it.
On the other hand if you are selling me some software for a boring Enterprise integration, please don't send me calendar invites for a conversation with your CEO, I'd rather get great API docs and integration guides.
It's often more beneficial to focus on solving a problem effectively and efficiently, rather than trying to scale it before it's even proven to be successful. By starting small and iterating based on customer feedback, you can create a better product that solves a real problem, which will naturally attract more users and drive growth.
That being said, scalability should definitely not be ignored altogether. As a project grows, it's important to plan for future scalability, but this should only be done once the product has proven its worth and a solid foundation has been established.
Another way of looking at it -- scalability is pretty much the entire value proposition of software. It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something. When you're first starting out, demonstrating even being able to _do_ the work is the first hurdle, but past that you're definitely going to need to show you can scale that out efficiently.
That sounds like a good way to look. People are Turing-complete, although that doesn't mean that they are very good at scale without instructions, and "programming" clusters of people has its own set of quirks. :)
> It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something.
But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency. That's, for example, most of Internet as we know it, from the hypertext browsing to videoconferencing, and most software we would call "embedded", from watches to spaceships.
Yes, that's a good point!
I find this blend of comment unproductive and intellectually dishonest. "Often" and "too much" are weasel words that remove any objective meaning from the verbs they qualify, to the point where stating the exact opposite is also valid.
The truth of the matter is that scalability is only not important when you don't need it, but it's already disastrously too late if you failed to scale when you need to.
Isn't it also survivorship bias to claim that scalability is overrated because a non-scalable system didn't broke in a particular case of a local deployment with residual demand?
Meanwhile hikes in demand hitting an unscalable system can and do break businesses, and when it happens is already too late to rearchitect things.
Doesn’t have the same ring to it.
That being said, scalability should definitely not be ignored altogether.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.
Even Facebook didn’t start trying to figure that out until they had to. Famously you needed an invite for a very long time. I’d imagine infrastructure limitations factored into this.
You’d be surprised how many successful companies push off dealing with scaling problems until they absolutely have to.
And then it opened to needing a college/university email address. Then it opened to everyone.
I got in with my college email address, and at the time it was considered a feature, not a limitation: a combination of higher quality since people couldn't just make new spam accounts whenever, plus a bit of a status symbol compared to just being on MySpace.
Thinking this way actually leads to one of the most common failure modes. Specifically the day-dreamer death where the principals become so enamored with a fantasy of what the end-state could look like that they lose focus on the next step to improve their odds of success. This is under-reported because day-dreamers are so common that when they try a startup and fail, it's not really an interesting pattern to report on. Such founders will tend to cite specific reasons that sound better in context (eg. wrong market, product flaws, sales funnel flaws, etc). But as someone who's been in the web startup space 25 years, I've seen it over and over again where founders fail to fully engage with the hard problems right in front of them in favor of working on what's fun, easy or tied to some dopamine hit of imagined future success.
In practice, you should not spend one second thinking about the requirements of 1B users until you have at least 10M. You will need to completely rewrite everything multiple times along the way to that scale anyway, and more importantly, you won't know the product requirements to actually get there until you achieve the earlier milestones and get the necessary user feedback about what's working at each size of user base.
I think people often underestimate the strata of learning, and the essential feedback that gets you to the next level. Or at least that's how it works for mere mortals like myself :)
If the answer truly is "yes", that's something you'll need to figure out early or at least plan for.
For the other 99.99% of situations where the answer is "no" or "almost certainly not" then absolutely, you should not be wasting time building an F22 or even thinking about it.
I'm advocating for understanding your business, not for prematurely building things.
If you're in the extremely rare class of businesses trying to build something on the scale of FB or YouTube where that sort of massive scaling is intrinsic to your mission then yes, you need to start thinking about those problems early.
For everybody else, yeah, absolutely focus on product and getting to 100 or 1000 or 100K or whatever users first.
What I see also is not just scalability over-engineering, but the same in the solution domain. When you don't really understand your customers needs sometimes there's a desire to create something with ultimate flexibility.
In the days of Xp we used to say YAGNI when people would be overly general. I'm trying to bring that back into fashion.
It will hinder growth and creativity because you are solving the scaling problem, which is a hard problem. Especially if you're early on and don't really have a product or customers.
I've seen multiple projects tank because people working on it were solving problems they didn't have and wouldn't have for the next 2-3 years at least. But they only had a few months of funding.
Could you expand on this? What are some challenges Fortune 100 companies face due to data quality?
Even if there are standard interfaces and APIs to supply data to, each and every engineer doing the integration will fill the fields differently.
Nowadays, software engineers barely finish learning basics of their first programming language and they jump in on scaling their first applications they developed while following a tutorial.
It is always better to first exhaust other options like improving basic efficiency and performance of your applications. A single server can do hundreds of thousands or even millions of transactions per second. I have many times seen vast farms of servers doing each at most hundreds or thousands of very simple transactions per second.
A problem is almost always easier to solve and faster to deliver when it is only expected to run on a single server.
And don't start arguing on the need of having multiple servers for redundancy. Properly automated, new service node can spawn within seconds -- maybe couple of your users will see a slight hiccup if you handle it correctly.
The goal here is not to say those practices are bad. What I am trying to say is that engineering is about understanding tradeoffs and making informed decisions.
--
I worked for a large bank which had internal automation platform. It consisted of about 140 "microservices" (apostrophes appropriate here...), 20 servers running those services and about 40 developers.
When I left 3 years later we had 3 servers (bank required hot backups, one within same primary DC and one in secondary DC), just one monolithic app and only 5 developers.
Our job was easier and we were happier than ever before. Reliability improved vastly. And our internal clients started getting the functionality they needed.
Previously, any service call from a client had to jump through multiple layers of "microservices". Vast majority of the application was devoted to communication between nodes, error handling, recovery from strange situations, etc. Once everything rolled into a single service those vanished because there were no longer network calls but method invacations.
And we no longer had to release 140 components and maintain versioning and dependencies between all of them. We only had one component to take core of.
I made a "new component checklist". Whenever somebody wanted to spawn a new component they had to first go through team review and prove the requirement would benefit from being separate component rather than part of existing monolith. No new component was ever created after this was instituted.
Yep, they def did not need microservices...
You can stop doing CEO calls if customer volume surges. You can't instantly stop using non-scalable infra choices.
The dynamics on that choice are fundamentally different when you don't have that get out of jail free card. You're forced to make a judgement call & walk the tightrope between not overengineering scalability vs ensuring enough scalability to handle surges.
I am sure there are many situations this isn't true, but I have not seen them in my experience.
At a previous position our devops guy had provisioned a cluster of god knows how many AWS Redis instances. I pointed out this was a lot more than we needed. He swore up and down we needed them because scalability.
He had no idea how much data was in those things. LESS THAN A MEGABYTE of data. In the whole cluster. We had no plans to grow it, either. And we were only accessing it a few times per day. We didn't even need Redis. We could have just stored it in the database directly.
He didn't know any of that, and he didn't care. Because it didn't matter. Resume-driven development.
He wasn't stupid. He was very smart. He knew what he was doing. "Managed a Redis cluster" looks good on a resume. "Just used the database instead" does not.
Did your company fire him and replace him with a DBA instead? If not, then the resume driven development is working!
I hate it when people look down on resume-driven development while simultaneously also doing things that promote it by letting HR people unrelated to technology screen resumes for recruiting.
Here's a hint -- if your company is screening, even subconsciously for buzzword (and believe me, most of them do) then you are enabling resume-driven development whether you like it or not.
Our CTO actually thought he was doing a good job.
The company already had a bunch of poorly written legacy systems which were used by about 50 clerks and written in PHP, Node.js and some .Net. They had problems with poor code and data quality, but none with performance.
So they recruited a new CTO and a new Head of DevOps which planned the new infrastructure together. Of course now everything has to be split up in Microservices (or Nanoservices, because each one should basically have the least functionality possible) and thrown in the cloud.
The only thing where a separate service actually makes sense are the machine learning processes that should extract information out of received documents. The rest could basically run in a basic PHP application on a Raspberry Pi without performance issues.
When I quit that circus, development had basically stopped altogether and they were already paying five figure numbers each months just for hosting.
I moved to a large corporation with the expectation that things should be better there. Turns out, that basically the same is happening on a larger scale. There they have a team of four guys having to manage more than a dozen microservices which were cut in strange ways.
I am only a senior developer and no architect myself, but the ones I know are everywhere between crippling depressions and just laughing their asses of because of the stupidity that's going on in the tech world right now. There are companies where it's done properly, but either I have extremely bad luck in my surroundings, or those are an absolute minority.
Why not just lie that you did that?
How employer would check that, lol.
(I know... that's a big "if")
So it is beneficial to have actual experience instead of just lying about it.
Sometimes.
then he'll probably have harder time anyway if the interviewer knows their stuff
There are many things that scale pretty far just by doing them right, without a lot more work.
For a process like calling new customers, I bet they have a halfway-decent CRM and email system to send out invites, track acceptances, and schedule calls. You could manage it with a spreadsheet, but it'll be easier and more reliable and contribute to other CRM tasks to manage it the right way from the beginning. And when the CEO can't do it anymore the whole process seamless shifts to letting VPs or account managers pick up the task (until that doesn't scale).
On the technical side, for an example: it's not very difficult to start with a scalable-by-default architecture, like with one of the "serverless" Docker-image based managed platforms like GCP Cloud Run. If your service fits there, you get scale-to-zero and scale up for basically free.
The trick, as always, is deciding what to do when you hit some feature that would be easier on a single stateful server. Do you bail, and go single-server or push through? If you stay with serverless, you're better positioned for future growth. But, you might hit that next feature that makes you go single server, and have wasted the work to stay serverless. This is what makes this stuff hard.
At this phase of the startup, we aren’t missing a beat.
Did you compare between DO, Lightsail, Vultr, Linode, etc. or was the difference between that tier of offering so little that you just picked one and ran with it?
I consult with companies that are looking to move in the other direction, from Digital Ocean to AWS.
Their biggest pain points as they grew are:
1. The lack of granular permissions scoping and access control.
2. The networking primitives in Digital Ocean ended up being too restrictive. For example, they wanted to have static egress IPs and it was much harder to do that in Digital Ocean than AWS.
3. The need for more of the managed services AWS offers.
If you are starting the next Vercel then presumably you need to think about scale from the beginning. There will be no forgiveness for not doing so.
Are you actually a tech company or a company using the internet as a channel, should be asked.
Obviously there's nuance to each individual situation but I think there's some good points here.
But for them, scale is at the core of their business, which is really the argument being made - don't let concerns about scaling problems you don't have distract from getting your business going. Scalability can't be a distraction if that's what you're selling.
Obviously with some caveats. That you don't fall over at a small amount of users and you are at an early stage.
This is more general than scalability: The more of any work you can afford to defer without hampering growth, the more of your initial capital you can spend on growing, and the more you manage to grow the cheaper it will be to raise money to build out the rest when it's time for your next investment.
The hard part is to determine which aspects you can safely defer.
And as the article implies, not just systems but also your processes. I recently saw a moderately big startup strangling themselves on processes that they won’t need for several more years. Mid level leadership (in itself a problem) agonized over scale, and didn’t realize that the products were being built at a snails pace and money was being burned with abandon. Late 2022 was a harsh wake up call.
I built it to scale: the frontend could be hosted at any ISP and the backend could be run pretty much anywhere which could accept a TCP connection. The 727K data layer hits were served off of a DSL connection to a server running in my house.
But there were politics (literally) around the data. Whatever hosted the data had to be under control of identifiable meat.
In the end though, the data owners wouldn't clear me to distribute the data to other operators until it got to the point that the only way I could do so would have been to drive around with CDs (because there was no spare bandwidth on the DSL link).
You can call that a success or a failure, it's politics after all. But basically they didn't care, which is what they were never going to say.
every system I've ever worked on has needed work to get to a certain scale. and that carries you for a while and if you need to get past the next plateau you need to rearchitect again. rinse and repeat.
so its not a binary thing.
so unless you have some specific goal in mind, invest to get a little past where you need to be today.
Using the author's example perhaps it's like they only way they actually convert customers is if the CEO calls them.
Or sometimes it's more to do with engineering. Each customer will only stay if you implement a feature that costs you more than their CLV.
More than once I've worked at a startup, or a company that has acquired a startup where this is the case. And when you suggest to fix the problem usually people can't quite comprehend it, so instead it becomes "you're the problem"
Whenever this is the case it eventually catches up with the. Sometimes through burnout and burn and churn employees, sometimes because they finally learn you can't sell a dollar for 80cents forever.
Slack still answers every customer feedback personally which is endearing and encourages more ideas.
I want it to work well, with that many, so it works awesome for less.
There are a number of things that would need to be done in the dashboard/backend, to afford much more scaling. I want the frontend to work well, at a fairly humble scale, though. That’s what most end-users will experience, anyway.
And I’m aware that 10K users is a rounding error, for a lot of folks, around here, but it is 10X what I expect, and it will buy me time, to do a proper dashboard and backend.
You don't have to build to scale right now. But you should have a plan of when you're going to need to scale, and how you'll do it. Or don't, and wait until your pants are on fire and then run around screaming.
“How did you go bankrupt?” Bill asked. “Two ways,” Mike said. “Gradually and then suddenly.” ― Ernest Hemingway, The Sun Also Rises
This single sentence communicates more than the author intended. I see the phenomena all the time: poorly architected systems that are brittle in face of user requirement changes. In the extreme case, a new widget or input on the UI echoes all the way to the database, touching and mutating every layer in between.
So, it is a perfectly valid matter to weigh the benefits of optimistic and forward looking engineering vs gaining deeper understanding of a market faster. This is a valid consideration. But if anyone tells you that going down any other path other than tacking on bits and pieces on the way "as we learn" is an error because of "user experience" (other than, ahem, performance) then be certain, certain, that the speaker lacks the necessary technical understanding to be offering strategic technical advice.
Devs hate writing this code, every fiber of your being recoils at the thought of just copy/pasting everywhere, ignoring anything that seems like a pattern, and having several identical API endpoints just to keep everything wet. But you will never be so productive in the face of changing requirements.
I couldn't disagree more. Failing to represent each piece of knowledge from the requirements in a single authoritative place means that simple requirements changes often have massive costs, as instead of one change to the place where the changed element is unambiguously represented, you need to change all the distributed locations where it is (often opaquely/implicitly) embedded in logic.
The problem is when DRY gets misinterpreted as “never write generally similar-looking code” rather than “every piece of knowledge must have a single, unambiguous, authoritative representation within a system.”
Doing things that don't scale makes sense when it gives users a white-glove, personal experience, or when it saves you from building a system before you know what that system should look like. Manually typing the email and clicking send does neither in this case. He's already sure that sending an email is the right thing to do, and if he's unsure of the wording or timing, he could be A/B testing or at least only occasionally changing these rather than manually doing the steps every time
There is an appropriate amount of architecture and design that should go into systems. One aspect of that is how much effort you put into scalable design. Building and maintaining scalable systems isn't free, so the "right amount" is pretty much somewhere between "none" and "we are AWS."
Knowing when adding complexity is worth doing takes experience, humility, and maturity, the exact qualities that are sorely lacking in many egomaniac software developers.
It's why we get ridiculous fads and pendulum swings where people get fed up with bad design and then go do the opposite thing, having learned nothing.
It’s much more difficult to do things that don’t scale when you’re competing with physical products on established markets who’ve spent years building a huge chain of efficient processes.
I don’t think this piece of advice will remain relevant for many more years. Starting a digital product today is already much more difficult than it used to be because the bar is much higher (and regulation has grown too).
For example, there's a "right way" to not do scalability and a wrong way. The right way would be avoiding patterns like microservices early on, while still minimizing human intervention and brittle integrations. The wrong way is to just have some kludgy solution that causes headaches for your customers.
With our budgets tightening we axed startup vendor we were paying $90K/yr to because their shit was slow and kept breaking, pissing off our customers. We were able to spin up replacement functionality for their product in 6 weeks.
The number of people I see building product before they talk to a single customer astounds me. I'd really like to understand the psychology of it. Building for scale is doubling down on this behavior. Not only do you not know if your customers will like it, you scaling something people don't potentially want.
Does it work well enough in the geo you want? Yes? You're good to go .
> I still personally email every new Pilot customer and offer to get on a call with them.
How is this a thing that needs to scale and how is it time consuming to write some automation? Just export the contacts, and write a small script that loops over it and sends an email using some transactional email script if you've got something like ses. It shouldn't take more than an hour to do from scratch, and like 5 minutes if you look it up using chatgpt3 or even Stackoverflow.
Email is almost always one-way spray-n-pray. A call on the phone is two-ways. The other party on the phone always tells you more than you asked, whereas the other party in an email reads it, files it away (if you're lucky) and moves on to the next email.
The other extreme is to play golf with your business partners; you'll learn their pain points more effectively than by sending an email, but it will take 4 hours.
OTOH, they'll have golfing buddies who are potentially your next client.
In huge text: "We partner with the best financial tools in the business" and a lot of logos from famous companies! I think, oh wow I haven't heard of this company before. But then see the text below the large text in small, gray text: "We're fluent in your finance tech stack and seamlessly integrate with the tools you use."
oh...
In addition to having deep integrations with these tools, we actually do have real partnerships with almost all of them.
"What's a real partnership?" is a great question, but I'd mostly think of it principally as "We know them, they know us, we have a red phone we can pick up to get things solved for each other" (which is good for the customer), and then also "And we occasionally do some comarketing together."
Works fine until markets use-case suddenly shift, and a platform is hammered out of existence.
Really depends on whether a business plan includes rapid growth (i.e. throw more money at the scaling issue), and an apology tour in the media. =)
I assume they are complaining about front end frameworks, libraries and approaches developed at big tech.
They did not actually try to sell you this stuff, it’s free. Stop complaining. The fact that you failed to properly evaluate your situation and see it was very different from theirs in your fault.
Other kinds of scalability (financial, development) are underrated.
The other side of that is, of course, that I don't want to design systems that will last a long time. I always have a "EOL" threshold for when a system we've built needs to be rethought; 100 customers (we're a b2b SaaS so 100 would be a gigantic ARR), 10k users, 500 reports a day, etc.
Do rebuilds actually work once there's that much code though. The system gains so much inertia by the time you can hire a bunch of folks it's probably too late.
That said if you know where boundaries are in your design you should keep those obvious and clear so that scaling will not be hard if you are lucky enough to need to scale.