Hacker News Reader

Autoscaling is a very particular solution to a very particular problem. Being a startup is like trying to make a hit record: everyone wants to become an overnight success. Nobody wants to be an overnight success and then have to turn customers away, or give out a bad impression because their site can't cope.

So a lot of effort goes into producing a site that could be "Facebook scale" if it became necessary. Although since this is difficult (and expensive!) to test, it's likely that some panicked manual intervention will be required if it does take off.

It's really worth asking "what is the cost of scaling" next to "what is the cost of NOT scaling, and what are the chances of it happening?". Is your organisation even able to benefit from rapid scaling - would it be profitable? Or are you going to end up like that luggage company screaming at its customer service reps to work dozens of hours a day?

If your cloud hosting cost goes from free-tier to $100,000/month in one month, and it's real traffic, is your business actually going to be able to pay that invoice?

Is the static overhead of scalability better for your use case than what a normal PC is capable of, which these days can be quite a lot: https://adamdrake.com/command-line-tools-can-be-235x-faster-...

(Been thinking about this since I mentioned the SRCF earlier, which was a single-server host for years: https://news.ycombinator.com/item?id=21741789 )

Cost is a very important factor... But you focused on the high cost end..

The cost of running a production server is something small startup businesses need to budget for... And you don't want to run your production website on a 5-10/month shared server. If you design for the cloud/serverless, you can get your costs down to near-$0 for light usage... But yes, using things like lambda mean a possible bill shock if your traffic grows faster than your income can recover it.. but if you have an advertisement on the page, it probably will cover the execution cost if it's lean enough.

In another example, I inherited a CICD pipeline that cost $300/month and could run 2 of our larger test suites simultaneously.

I converted this to AWS Spot instances and now support thousands in parallel and I pay under $170/month ( https://ldoughty.com/2018/01/gitlab-multi-runner-w/docker-ma... )... But to your point about "what if" costs of this, yes, there's a limit on place for this setup so it can't go over 20 or so... And it is important to consider the impact of auto scaling with regards to the bill... For a small business, AWS effectively doesn't have a way for you to really stop "Bill attacks" or legitimate high traffic on API Gateway/Lambda short of you taking down your site.

My recommendation is: ideally, don't expose something that costs you much too run unless you can recoup the cost from that action (paying customer, advertising).. your homepage should be static or cached in a CDN until you're ready for that large bill.

Dedicated servers are actually quite cheap. I pay $60/month to run one of my products that services thousands of paying customers across North America (High performance native TCP and UDP based game like servers) and the performance is quite predictable

Indeed the servers can be cheap! The inherited configuration used a redhat image with support that added a premium cost on top of being a slightly beefier server to run a few dozen containers simultaneously for testing. With the configuration change we optimized the choices and dropped support.

That said, it really wasn't much work to drop the 6pm-6am costs to effectively 0 unless a developer decided to work late (it automatically scales up)

Please research opportunity cost. The total time you spent to optimize the service was probably in tens of hours.

Multiply that by the amount of money you cost your employer per hour and you'll realize that your "optimization" actually resulted in a net loss.

I'm fully aware of opportunity costs, the investment was made because an investment needed to be made regardless to stand up 2-3 more servers. Doing so would have raised costs to perhaps $400/month if we simply copied existing setups (plus 2-3 days of work, probably, since we had not revisited CICD in over a year)

So by that, I've saved $5,000 in pure server costs. It probably cost me 10 extra man hours to do this way... But I also resolved a bug that was not terribly worth fixing before but took up 30 minutes of time every few months.. and improved the security by taking the servers off the public web.. however you want to quantify security improvements.

I always look at opportunity costs and the automation cost/benefit! The article was researched and written on my own time for my own enjoyment (so this was actually "free" from a business perspective, but my responses factored it in as if I did it as paid work. Just clarifying in case someone wants to claim I'm publishing "work"

For me a huge benefit of serverless is the ease of development speed.

Google's Firebase Functions are ideal for REST endpoints, they bill by the millisecond instead of AWS's 100ms (I think?) pricing.

If you buy into Google's ecosystem entirely they provide you a lot of functionality to get off the ground running. E.g. their (free!) auth service integrates with security around functions. In less time than it takes to look up the YAML for Nginx config you can have a proper authenticated set of REST endpoints up.

Now that said all I have my CPU intensive stuff running on a fixed price VM.

Exactly!! I still think that people do not think enough about the cost of NOT scaling nowadays. And especially, what is it going to cost when I can't scale and I have to change course. I recently heard a podcast where they said: "choosing a distributed databases is probably premature optimization right?", I was pretty surprised. A startup often sells a better user experience. Although less important for US startups since US is a big market, international start-ups won't get many clients further away from their country if the latencies increase signficantly so distributed databases is something you might desire anyway. Secondly they might indeed end up losing clients if they break down due to scaling issues. And at that point, if they want to keep their clients, the options are limited, 1. they'll have to go through a risky deploy to change the database, 2. try to scale it yourself by creating multiple replicas with all problems and technical depth that might bring, finally maybe still losing your clients if you fail and maybe ending up to pay more due to overprovisioning. 3. Accept that clients complain or leave. I would choose FaunaDB or Spanner for my startup personally.

Let's not forget about AWS Aurora Serverless! I keep ranting about this to anyone who will listen, but hands down one of the biggest innovations in cloud infra for exactly the reasons you describe. As the API layer is generally a stateless shim around the database, serverless has limited upside if you can't scale your database sufficiently. But, Aurora Serverless unbottlenecks that concern and makes some very simple and powerful architectures possible. I wonder if there's any open source equivalent to it?

My very obvious question for this is... what's the latency of "spinning up" the DB? It's not discussed on the Amazon page. Their example is someone running a news site, so I imagine the wait time is something under 1 second of overhead to spin-up the DB, but what does it feel like in practice? Are we talking 50ms of wait time for the DB spin-up, or 500ms?

A news site seems like a weird example for an auto-scaling database use case, since, asked to design a news site from scratch, I would aim for 0% of public requests to touch the DB.

You can run a minimum number of vCPUs and add the condition to scale from there. That way you have a minimum threshold for requests.

Ah, so there is always one vCPU running. This makes more sense and was not clear on the product page, thanks.

The company I work at is considering this as part of a pretty big shift in our infrastructure. Beyond the obvious upsides, what cost do they come at?

There's a few. My company has been using serverless aurora pretty much since preview and will keep doing so for some parts of infra, but are moving some services back to provisioned.

- Data is replicated cross region, but your compute all lives in the same AZ, so if (when) an AZ goes down you have to wait for a new instance to be created elsewhere. AWS says the time for this is undefined, IIRC we've seen it be around 15-20 minutes before the DB is back online.

- If you have lots of long running transactions or queries, serverless can have trouble finding a scaling point and won't scale up/down. You can set it to force scaling after 5 attempts, but this results in dropped connections and 1-2 minutes of downtime every time.

- Scaling up actually takes 45s-1:30 for new capacity to be available. If your load is spiky enough that that's too slow, you're stuck with overprovisioning anyways.

- Tools like VividCortex don't work for serverless if you rely on those. Teams here that use serverless have shifted to DataDog APM for this purpose.

- Loading data from S3 doesn't work. This wasn't really a usecase we had but it's something to be aware of.

That said, for gradual load or DB downtime tolerant services it's great! Also very nice for dev environments as the scaling down to 0 can result in some very real cost savings.

Just to make sure: You're talking about the same Aurora Serverless which consists of a single database server instance in a single availability zone without any guarantees regarding availability in case the availability zone goes down? To quote the AWS documentation [1]:

> The Aurora Serverless failover time is currently undefined […]

Don't get me wrong: I really like the idea of Aurora Serverless, but with the current feature set it isn't very compelling.

[1]: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

How many small companies would have any failover if their colocated database server went down? Not only would they not have a warm or hot standby, they probably wouldn’t have up to date data.

Because Aurora separates computation capacity and storage, the storage volume for the cluster is spread across multiple AZs. Your data remains available even if outages affect the DB instance or the associated AZ.

Part of the design consideration here should be limiting SPOFs like this and creating bounded isolated components which interact with their own persistence layer.

If you're in a traditional three tier architecture you might run into your database being overloaded by the autoscaling behavior but this is fundamentally a provisioning problem. These problems are better served in models that are scalable.

Your suggestion to by a (necessarily overprovisioned) on prem database server is cost inefficient, degrades in value over time, and is effectively a lock in.

Embracing eventual consistency is not that scary!

Even eventually consistent databases have severe scalability limitations compared to how far out stateless functions can scale.

And nobody said anything about on-prem databases. You can hit these issues just as easily with e.g. Aurora or Dynamo.

> So in reality I think the only really scalable part here is the database. And scalability in databases has no universal solution and the ones that exists come with the load of caveats.

I do wonder what limitations and caveats you are talking about. That is true for many databases who are only consistent on one row/collection/region/snapshot, have strange limitations such as no joins, require your data to be preformatted in a certain way etc. DynamoDB also has a few limitations in those aspects. I have both love and hate memories from my experience with DynamoDB. But nowadays you get to choose between databases like FaunaDb, Spanner, FoundationDB, CockroachDB... So what are the Caveats you still worry about?

If you would combine such distributed databases with edge serverless frameworks like Cloudflare workers I could see amazing possibilities tbh. It's strange how front-end programmers seem to be picking up distributed databases at great speeds due to the benefits of low latency for their users, zero configuration and autoscaling. But it seems that we backenders are reluctant and still prefer our hardware for some reason :). Is it pure pricing?

Everything doesn’t have to be a “FB scale” to need autoscaling. We are a small company, we process messages that come from a queue. During the day, messages trickle in to the point where one small instance can handle the load at a decent speed. At night, to get the same throughput, we scale up to 10 instances. Scaling is based on the length of the queue. It’s a Windows service so we can’t use Lambda, so we have the smallest instance size that we can for it. Some of our newer processes do run on Lambda. The database is not a constraint.

We have another process that is read intensive. The database is a constraint. We autoscale read replicas. With AWS, all of your read replicas read from the same storage arrays (grossly simplifying)

I tend to agree with you, but I don't have the data to back it up. Do you have any success stories to share of modern products that operate at scale on simple, dedicated hardware configurations?

It seems to me, like most things in software these days, we sacrifice performance for convenience at every turn; and in the cloud case specifically we also pick up a good portion of complexity which rarely pays dividends.

I'd wager that for most non-global scale products you can spend a few thousand dollars at the local PC shop and assemble some hardware that will more than support your needs. Although, I'd still advocate backing up necessary data in the cloud for resiliency.

If your software is super resource intensive then you could have people install and run it locally gasp.

> I think the only really scalable part here is the database

That sums it up pretty well.

The great irony here is that those thin, stateless web servers that are the only things to easily autoscale are pretty much guaranteed to never be your performance bottleneck, unless you're doing something really strange.

>unless you're doing something really strange.

all you have to do is run Wordpress with Woocommerce, multiple additional plugins and an overloaded theme with visual builder/editor that needs 512 megs of RAM and 500ms to display a contact form[0]. Some themes can be amazingly CPU intensive.

[0]and 900+ queries, but since most of them are simple selects any cache like redis or memcached removes their performance impact.

RE scalability of databases, it is a problem which happens when some of the autoscaling instances are paused in the background and the DB runs out of connections. Amazon's solution is just spend more money on databases

>and scalability in databases has no universal solution and the ones that exists come with the load of caveats.

How is this true? Cassandra/MongoDB/Scylla are all very very popular NoSQL DBs that scale very well and work for a good number of common use cases. Sure you have to be a little more clever when writing certain types of applications with them, but seems like an appropriate trade off. If you don't like that trade off, you can spend a bit more money and get Spanner/FaunaDB and you don't have really any tradeoffs (other than $$$).

Cloud auto-scaling has made quite a few things in the public sector cheaper to operate. There are the obvious use cases like tax-returns or student-registration, which are annual loads where you could technically scale without the auto part. As more and more things become web-based applications, however, auto-scaling proves it’s worth on things like student/teacher/parent work tools that see a high load during the hours from 08:00-15:00 and again from 18:00-21:00 and almost no loads outside of those times as well as during holidays or weekends.

I've worked places that had racks of servers that were only really used for a few hours a month to support massive financial consolidation runs - that would have been a great workload for a cloud service.

Can't argue about this particular use case. For this cloud is good. You can rent super-duper hardware for few hours and be done with it.

I think one of the key benefits of autoscaling isn't the ability to scale out when necessary, but to scale in.

Say you have an idea for a web app that needs a server-side component, but you only ever expect to have a few users. A VPS will cost you at least $5 a month. If you can make your backend serverless, you'll probably only pay a tiny sliver of that, if you even end up exceeding your provider's free tier.

It's not $5 per service unless you only have one service. The scaling is the fact that you run more than one thing on it.

I run a bunch of stuff on a $5 VPS and more still on small machines like raspberry pi's and so on behind it.

On my distro at least, spinning up a real paravirt container (not docker, a full OS with systemd, SSH, etc) within a VPS is a matter of unzipping a bootstrap to /var/lib/machines and running 'machinectl start whatever-you-called-it'.

It's just a different set of abstractions to learn, and at least IMO one that's more valuable.

What you are describing is a very typical "business application" something that is a step above a spreadsheet and a word doc on how to use it.

In those scenarios, the _real_ cost isn't the $5/mo VPS but the managing upgrades, security patches, etc - which would be mostly handled for you in this scenario.

So, if you really are this price sensitive, here's a flip-side.

I built a simple email verification service for A company I used to work for. Used API gateway, lambda, s3, and sqs. Few thousand users a month. I don't need to maintain it, check for security updates etc.

The monthly bill for all that was ~$2 NZD.

Not unless you also make sure all downstream services are similarly auto-scaling using something like DyanmoDB etc so that capacity can be managed further down. This is the idea of Serverless. Not just taking one aspect of your stack and abstracting it out to a managed service but using managed services across teh baord to help maintain that same "autoscaling" feature.

Sure, but if you have relational data you probably don't want to be giving up your RDBMS for Dynamo. Pain (and inconsistent and hard-to-reason-about data) lies in that direction.

I reckon the "Embarrassingly Parallel" vs "Inherently Serial" problem, I think what you are describing is just this all over again

Well, if you really need the autoscale feature to handle sudden and ludicrous spikes of traffic, you will most likely also use a database that autoscales too. i.e. DynamoDB.

For most applications, it's really not needed though.

But like they said, it comes with a ton of caveats. DynamoDB is one of the most confusing technologies I have ever used.

Absolutely agree, I tried it out for a short personal project and was disappointed with the bad documentation and even worse libraries that I found. I'm assuming large companies have their own internal libraries/ORMs for this, and that's how it is intended to be used.

In terms of the actual technology itself, it's very interestingly built, and as the poster above me mentioned, with a ton of caveats to the promises it makes.

It’s because it is just really expensive (both in time and money) to do basic things like a table scan.

That's actually one of the caveats that comes with unlimited scalability: You must design your application to not require full table scans.

It is one of very few technologies that I need to get a pen and paper and some quiet time to decide on a table schema. Most of the time I need to redo it after coming up with new query requirements.

I hate and love it at the same time.

"DynamoDB uses synchronous replication across multiple data centers for high durability and availability"

This does not seem scalable for OLTP type load on some busy store. Again I think you'll be way better of money wise hosting your own database on real hardware either on prem or colocated

DynamoDB is used by Amazon.com.

"Across the 48 hours of Prime Day, these sources made 7.11 trillion calls to the DynamoDB API, peaking at 45.4 million requests per second." [1]

[1] https://aws.amazon.com/blogs/aws/amazon-prime-day-2019-power...

You're forgetting specifics here and the amount of hardware resources thrown into it. I've already told that I am not discussing here FB/Google/Amazon scale stuff. It is their problem and not shared by more regular deployments

TIL: A day has 48 hours at Amazon.

The same day across multiple time zones could take 48 hours

Of course Amazon would call that an Elastic Sundial.

is there any transnational storage solution that actually handles well a traffic spike?

my understanding is that most storage solution scales well when planned, but scaling during a spike only causes more stress onto the current replicas/partitions for all the time needed to boot and shift data toward new replica/partitions, and the new instances don't contribute in moving the load until the process is complete.

so they work if you can predict a spike, but they can't handle a sudden spike, especially if the data to replicate/partition is sizeable.