Hacker News Reader

Comment by sudhirj | Hacker News Reader

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours. Even then the platforms might be cheaper because you're only paying for a fraction of the dev-ops team's salaries, but you get their full benefit.

View on HN

I work for a PaaS.

- The ideal fit for any hosting PaaS is a company who has a large hosting and infra footprint but for whom the technology is _not_ the core competency of the business. Slack is very much better off running their own systems with their own people.

- As someone who deals with customers every day I can tell you that yes - we know our platform specifically and how the internet works generally better than almost 100% of our customers, but we do not know _your application_ at all.

- Many problems in The Cloud are the result of application developers not understanding that there are performance differences between localhost and The Cloud, specifically around IO, or that there are performance differences between the Cloud at 1x and The Cloud at 10x with everything. Systems can run smoothly and applications can still kill themselves because of the interaction between the two.

So this is where 12factor comes in - unless the application is operating at lower than layer 7 (and with some of the newer offerings even layer 3) there’s not much technology centric stuff going on. Everything Slack is doing is happening at the HTTP / gRPC / Websocket level, and it’s hard to make the case for self managed hosting.

The idea of requests going to an application server on a TCP or HTTP connection with the application server being able to access a database or datastore is now common enough for all the PaaS providers to have abstractions for it, with the load balancing and database/data stores being managed. If customers aren’t happy with the auto scaling logic that can be overridden, but it seems like everything else is pretty rock solid.

Slack’s USP isn’t to reinvent load balancing, distributed configuration or database management, so if I was running it and had a clean slate a PaaS seems like a better bet.

If I had To guess, I’d say that Slack’s infrastructure footprint is probably as big if not bigger than any popular PaaS.

That seems... Questionable.

Question on how they measure - I'm in 7 slack workspaces. Do I count 7 times?

Well, I have to create a new account every time I join a new group, so I'd say yes.

Two if my workspaces are tied to the same enterprise account, they should be the same, but who knows how they measure. My base assumption is "in a way giving nicer numbers"

Slack had 12m DAU at the end of last year.

Snapchat runs on App Engine (and other stuff runs on App Engine too) so App Engine is almost certainly bigger than Slack just based on that.

>>"but we do not know _your application_ at all."

You're missing the point. A PaaS, or serverless service doesn't need to know your application. That's the whole point. They're just API calls, and they need to succeed with consistently low latency.

You're telling someone who works for a PaaS company what a PaaS does and doesn't do. I think maybe they know.

More broadly, almost any application these days is "just API calls". The issue is which API calls, how many, with what frequency and where. That essentially is the application, and a PaaS employee doesn't magically know that stuff instinctively.

> consistently low latency

That's (a) just one objective and (b) too vaguely stated. What's an acceptable p95 latency for a FaaS? For a database write? For a message queue? The answer is a very big "it depends". And why are we only talking about latency?

There are tradeoffs literally everywhere you look in this space, and not knowing enough about the application's performance, reliability, and efficiency goals can sometimes be a real hindrance to being good at running that application.

> That's the whole point.

Until something falls over and you haven't deployed any new code and all the vendor's systems are green. Part of the reason you went with a PaaS in the first place is that you didn't want to manage the infra yourself, you just want to ship application code.

This has worked great so far and so you've stopped thinking about the infrastructure at all. Disk and network I/O still exist though, you've just been incentivized to stop thinking about them and so all you know is that the vendor sucks, when in reality it's the application's fault.

Somebody's gotta roll up their sleeves and see what's going on in there though, so hopefully your PaaS vendor is as cool as we are :)

I/O is especially problematic. I've seen people very, very confused about the totally predictable failures of their system components due to burst exhaustion on EBS.

You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it.

I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular serverless solution. Once you give the green light to adopt a specific cloud solution, you tie yourself to that cloud, and that can turn out to be a bad idea in the long run.

Lambda is a great tool when used right. It can take big workloads without costing too much. But in itself it does very little. It still needs to intergrate with something to be triggered, and if you go with AWS API Gateway that can cost you a pretty penny. Load balancer also incurs costs that are difficult to predict and is not so flexible as other load balancing tools, so sometimes you might still need to provision you own service discovery and load distribution layers, just like they had to do at slack.

Serverless is nice, solves a lot of issues and give smaller teams a shot they otherwise wouldn't have, having to manage everything by themselves. But cloud costs can be opaque, cloud implementation can be very complex, cloud solutions can be too rigid sometimes and tying your product to one vendor can be detrimental in the long run.

Not just costs! AWS services are opaque performance minefields, too.

If you need reliable low latency -- which is probably core to the slack experience, or any GUI in a competitive space, really -- lambda is not a good option. They just don't share your priorities. AWS support will happily waste a lot of your time chasing Just One More Trick to mitigate the problem, though.

Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better.

It doesn't matter that you're locked-in, with slightly higher than wanted costs when you've failed due to poor priorities.

Become successful, then worry about removing your lock-in if you actually need it (and you probably won't).

I'd disagree pretty strongly. You may evaluate the tradeoffs and decide to go all-in on a single cloud (and you should do that analysis!), but it's far from a given. You're not wasting engineering time if the costs get out of hand and bankrupt you. Managed services may accelerate your time to market, but they do so at the expense of lock-in and your bill.

Context: I work at a startup that benefits enormously by avoiding AWS/GCP (for most cases) and renting cheap dedicated servers. It is context-dependent; our exact business doesn't benefit much from managed services and really needs big servers.

Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability.

Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more reliable if you use SQS rather than your own message bus (and let's be honest, it'll probably be cheaper too).

Yes, you should always evaluate the costs, but reliability and the amount of time you're going to spend maintaining something is something that somehow always gets left out of these evaluations.

(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us)

An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month.

Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month.

That's a difference of 965 $/mo = 11,580 $/yr. If you have 10 of these, they'll pay for a full time sysadmin. Now, that's leaving out a lot of details (bandwith costs and application hosting come to mind), and assumes truly massive databases. On the other hand, as that sysadmin, I promise our databases don't take anything like my full attention, and you really should have some sort of sysadmin/ops team anyways (please do not make devs run your AWS infrastructure; it will end in tears for everyone). Every time this argument comes up, people do mention reliability and time spent on maintenance, but... it's really not bad. Hardware doesn't actually fail that much, postgres isn't that complicated to configure, OS patches aren't that hard to apply. Your mileage will vary, but sometimes it's just not worth using AWS. (And sometimes, it really is; if we didn't need to run oversized databases, I'd push us to use AWS in a heartbeat)

100% this. Cloud vendor lock-in is absolutely a relevant factor for consideration -- especially when the non-cloud alternatives are ~10x cheaper.

At my company we made the decision to stick primarily with managed dedicated servers over AWS in our very early days. Now we're a decent size a few years later (25 employees) and the cost savings we're realizing are tremendous. We did the math and found that if we had gone with AWS in the early days then we would now conservatively be paying an extra $165,000 on our hosting bill annually.

We still use AWS for some specialized services (e.g., Lex) but the bulk of our stack runs on gear that we now colocate for a fraction of the cost.

Yeah but why pay someone to look after databases when I could get them do something else that'll have way more value. Using managed services just reduces how much money I have to spend upfront.

It depends™ on your (human and computer) workloads, but in my experience you need ops people to keep everything working smoothly, and once you have them it's easy to have them do this stuff without taking away from other areas. YMMV.

This matches my experiences entirely. By running our own hardware (mix of Hetzner and colocation) we go from 20% of our revenue spent on server costs to less than 5%. Well worth any potential headaches, which are almost always overstated.

You are forgetting to price in some minor features that Aurora provides: - Aurora's storage is spread across three availability zones. - Backups. - Automatic failover. - No need to configure anything, it just works.

If your time is free, and you don't actually need anything resembling high availability for the data in the database, then that's a good price comparison. I'm not arguing that managed databases makes sense for everybody, but if you're doing a price comparison then at least factor in multi-site redundancy for the data?

> You are forgetting to price in some minor features that Aurora provides

That's true and fair, although in both directions; skimming the docs it looks like aurora prices include 2 replicas? But backups aren't free (to store), bandwidth isn't free, and iops aren't free. Also, my difficulty in figuring out a fair pricing comparison highlights another point: a dedicated server has a fixed price. Other than more servers for more instances/replicas, you're never going to pay more, and even then it's a simple "adding another replica will increase our costs to X*(N+1) per month", not a "scaling out will add X to our costs, but if we use more I/O than expected we'll add Y to our costs, and exporting data will cost Z in bandwidth".

Getting to market faster is fine for an early startup, less so for a public company which needs to justify costs to its shareholders.

Not every company is a startup dude.

Not in the scenario I described. You implemented a solution using lambda and api gateway. Hundreds of microservices later you may realise you could save considerable money going unmanaged. By that time your lock in is costing you an unhealthy amount of money and there is no easy roadmap to mitigate it.

Again, everything must be planned beforehand. Savings could also be marginal, but it could also be significant. In big enterprises, where billing is north of a couple of millions of dollars, every percentile you can save is justifiable.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours

This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you.

My rational mind has trouble accepting things based on faith, which is also why I don't trust RDS: I don't know of any way to run a distributed SQL database without data loss (neither does Jepsen), so why would I expect RDS to do this correctly?

Using those services does provide a warm and fuzzy feeling, though.

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

AWS is pretty good whenever we've needed them. Google? Probably not.

Google swears gcloud support is good, of course, but I've never actually used it -- but then again, I've never actually needed it.

Meanwhile, I need AWS support constantly because their entire platform is a gigantic social experiment in minimum viable products. How crusty are people willing to tolerate? Evidently: very, very crusty.

Vice versa for us. Google always has troubles and outages and AWS worked fine. Not sure why.

We don't use most of the platform but the stuff we do use is rock solid (VPC, EC2, RDS, Route53, S3 etc). I do sometimes wonder what the point is for half the stuff they release.

What I do know about GCP is they had a production bug in their tooling that was breaking everything for literally 100s of customers and they never even bothered replying to the bug report on their support forums. That experience along with the general modus operandi of shutting things down that don't further their surveillance capitalism business model means I won't be trying them again.

> I do sometimes wonder what the point is for half the stuff they release.

I'm convinced they mvp every possible idea because it makes their platform more sticky. The more services you use, the harder it is to leave them for something better. The problem with that is you get 100 half dead zombie services and it's really hard to know which services are actually supported and which aren't.

It's the Amazon equivalent of Google creating 10 different messenger applications only Amazon never kills an old service, they just let it rot forever.

It almost seems to me (as an outsider) that Amazon has a policy that any tool they build internally or for a specific customer also has to be offered to the public on AWS. Like their managed satellite service or any number of things that make you say “I’ll bet some critical AWS thing was the first customer of this hyper-specific product”.

AFAIK this is pretty spot on. I don't know if it's "policy" or what, but based on talking with AWS account managers, AWS does not often (or ever) create custom AWS services/products for individual customers. They do however let customers submit feature requests, and then let customers lean heavily on AWS to implement those feature requests in the public AWS services.

I think this is what leads to a lot of MVP-type services. A large customer clamors for some individual niche feature, AWS implements a MVP version of it, and then the team moves on to the next feature request (which might be for a completely different service, leaving the MVP in a perpetual MVP state).

I launched a large AAA game backed by a hybrid (GCE+Physical) infrastructure and I would swear by their support.

Yes, we paid for enterprise support, but it's especially good. They even contributed code to third-party open source projects to solve one of our bugs.

I hear often about google's support being terrible, but the enterprise support on google cloud is definitely an exception.

F.D: Aside from giving a talk at google stockholm once, I am not affiliated in any way.

This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

Slack's problem though had nothing to do with racks going down.

I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.

> I'm not convinced Amazon's team is immune from the sort of complex failure mode

You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable.

This varies by service, but they always project an image of their infrastructure being rather smart, and in the cases where I've been able to make an informed guess about what's actually going on, it's usually wildly inconsistent with the marketing. They don't warn you about the stinkers and even on services with good autoscaling and no true incompatibility between AWS's hidden choices and your needs, your scaling journey will involve periodic downtime as you trip over hidden built in limits and have to beg support to raise them. Sometimes you get curiously high resistance to this, leading to the impression that these aren't so much "safety rails" as hardcoded choices.

Oh, and just last week we managed to completely wedge a service. The combination of a low limit on in-flight processes, two hung processes, immutability on running processes, and delete functionality being predicated on proper termination led to a situation where an AWS service became completely unusable for days while we begged support to log in and clear the hung process. Naturally, this isn't going to count as downtime on any reliability charts, even though it's a known problem and definitely looked a lot like downtime on our end.

We're a small (<10) team with modest needs. AWS lets us do some crazy awesome things, but it really bugs me how reliably they over-promise and under-deliver.

> You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive.

Yep. Very much so. Mostly because I don't have enough personal Lambda-specific warstories to feel confident badmouthing it in the context of this discussion thread. But the bits of AWS I do use are certainly not all rainbows and roses...

I have one app/platform I run that basically sits at a few requests an hour for 11 months of the year, then ramps up to well over 100,000 requests a minute between 8am and 11pm for 14 days. Classic ELB (back in the day) needed quite a lot of preemptive poking and fake generated traffic to be able to ramp up capacity fast enough for the beginning of each day (aELB is somewhat better but still needs juggling). We never even got close to getting autoscaling working nicely on the web and app server plane to let it loose in prod with real credit card billing at risk, we just add significantly over provisioned spot instances for our best estimates of yearly growth (and app behaviour changes) for the two weeks instead, and cautiously babysit things for the duration.

It's nice we can do that. It'd be nicer if I didn't have to keep explaining to suits and C*Os why they can't boast own the golf course that they have an autoscaling backend...

Hey, let's not be hasty, it sounds like you've built a bespoke autoscaling backend that intelligently predicts future usage and dynamically allocates compute resources to match customer needs.

In my experience the ops team will be constantly changing things and upgrading to keep all those customers happy, and they don't really care about your personal risk tolerance. They probably screw up a smaller proportion of the changes but they make an order of magnitude more and you can't predict what impact it will eventually have

What? You can totally run disturbed SQL without data loss. Did you mean to imply something in particular?

CAP theorem strongly suggests if you optimise against data loss you have to trade off something else which is probably important too...

'Disturbed' SQL confused me, had to look up the thread to see Distributed SQL. And yes, I agree, you certainly can run distributed without data loss, depending on the platform design and your 'transaction level'.

Could you describe what you mean by a distributed SQL database?

I don’t think RDS would generally fit that concept as I understand it. Aurora’s data store possibly but you choose to use that specifically.

Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services.

Obviously, there are good aspects of outsourcing devops/admin work. It's a tradeoff, as most things. If you are a struggling startup it's difficult to justify the cost of owning a bunch of hardware and hiring expensive infra people to manage it. However, facebook is probably better off owning their own infraestructure.

FWIW, that time AWS had a massive massive failure it was actually really convenient being on AWS as so much of the Internet was down at the same time that pretty much no one was upset at me for my one service also being offline.

I think it depends on your competitors and what your software does. If your competitors are still running their services but not you, you're bound to lose some customers. Also, regardless of that, you can easily be losing money while the service is down. For instance, if you have a food delivery app and it's down, while your competitor is up, people will just use the other app and you're going to lose money while AWS is down. Even worse, they may actually end up liking the competitor's app better, and you can even lose customers that way.

However, if you're looking from the point of view of an employee trying to justify to your boss why the service is down. I'm sure they'll be more understanding that's not your fault in this case. It's a risk they decided to take.

This is an underrated point. The problem with AWS and those massive services is that they are "alive" and continuously evolving. In most cases it's fine and you don't notice, but when it goes wrong it can affect you and you have no control over when that is likely to happen.

In contrast with in-house infrastructure, you can make your stack as simple or as complex as you'd like depending on your needs (a lot of projects can get away with a handful of physical machines all configured manually, no Terraform/Kubernetes/etc) and you control when you make drastic changes that risk breaking things so you can plan them during a time when downtime would be the least damaging to your business.

You think you can run something like Slack without intimate knowledge of all the moving parts?

The only thing that 24/7 crack devops team ensures in a situation like this is that you continue to generate billable workload, even if it's just millions of little serverless pods spinning in a circle from some configuration mishap.

> you get their full benefit

and their full downside.

what, prey tell, downside?

they operate at a scale (since you mentioned it) much larger than yours. you don't benefit from scale beyond what you need, you "only" benefit from the SLA.

you don't get to tell them what to do, to set their priorities on features vs bugs vs performance vs meaningless metric of the day.

you don't get to interact consistently with the same person or same set of staff, to understand their foibles and to nudge effectively.

you don't get to decide what features are critical to you and cannot, ever, ever be cut no matter how otherwise impactful they are on the environment.

you don't get to set the timetable for "events".

it's not the absolute no-brainer you are making it out to be.

that said, i agree that the value is solidly there for those in the fat part of the bell curve.

CloudSQL just took down our DB for the hell of it once. Must've cycled the node or something.

Replace "serverless" with "managed service" and I agree. But serverless doesn't give you any magical reliability improvement over anything else, because your NFRs dictate the implementation that will produce the most reliable product.

The design of a wooden table will inform what tools will produce the best version of that table. It may not be the newest power tool; it may end up being chisels, hand planes, winding sticks, a kerf saw and a mallet. If somebody maintains your tools for you they'll stay reliably sharp, but that doesn't lead to a good table unless you pick the right tools and use them the right way.

I prefer to refer to NFRs as Operational Requirements. Cloud vendors are entirely in the job of satisfying operational requirements.

I’ve anecdotally experienced success doing this; the only downside is cost. In many cases you end up throwing money/hardware at a problem, but having used PaaS for high scale apps where money was no issue was like life on easy mode.

Reply on news.ycombinator.com