Serverless Horrors | Hacker News Reader

Serverless Horrors

serverlesshorrors.com

292 pointsfranciscop169 comments

> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud. But still this doesn’t invalidate the point of having protection against such attacks, and limit the spending.

This is extremely sad. It's like we are taking steps backward. A 3.44MB should not be an issue and if it is, the answer should not be to host it elsewhere. If there is no other lesson learned here, it's that there is no such thing as 'free'. As others have said, there should be more done to prevent such large losses without some kind of limit.

I would also like to point out that VPS's are extremely cheap [1] and extremely easy to manage. They have automatic limits.

[1] https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/

VPSs being “easy to manage” is a strong option full of assumptions.

I remember a comment from this site. Something like: we must study how Amazon and other big cloud's marketing work so great to make this generation developers think that vps or selfhosting are hard.

IMO vps was easy before and even easier now to manage.

Please explain how you rotate ssh keys, store audit logs, backup (and test backup procedures), configure secure network between vpses such that your neighbor cannot eavesdrop.

I think on a $5 vps that you're using to host a small website you simply just don't do those things.

For my personal stuff, when it comes to SSH key rotation and secure networks, I like to let Tailscale deal with both. It handles SSH authentication for you via your SSO provider, so there's no need to rotate keys.

(Aside: you shouldn't really be using SSH keys to begin with at anything but a small scale. SSH certificates are much more flexible)

1. You didn't do that on a VPS

2. Often there would be a cPanel plugin/extension/app/config value (if the hoster enabled it for you) that would just do for you what you needed.

You don't need to do any of that for hosting a content website.

The content & config are pushed by rsync/ssh from a git repo, so there's no need for backups. I can recreate a server in half an hour. I guess I lose the webserver logs, but I rarely look at them so I don't care.

A single server has plenty of bandwidth for a personal site, so there's only one EC2 instance and no secure network is needed. If I need more bandwidth, I'll use a load balancer but there's no need for secure connections between the load balancer & web servers because what's the eavesdropping threat model for a public content site?

Let's Encrypt seems to deal with https key rotation without manual intervention.

The cloud servers just have the usual ~cloud/.ssh/authorized_keys login setup, and I guess I rotate them every time a stronger crypto is recommended, which is 4ish times in 30 years.

If it is just a content website then maybe yes. Cloud complexity probably won’t worth it.

Still I can think of a corporate blog, and you have employees come and go then it became a problem even for a small website. Otherwise an angry admin can deface your website and damage your reputation.

All other things like secure net won’t apply for a small website, of course.

For a simple use case this is not an issue.

- you add your ssh public key to the hosting provider, so any new VM will have it automatically

- you use the snapshot service of your hosting provider for backups. If you have a database, run a cronjob that dumps it so it's in the snapshots as well. Alternatively use any backup tool to backup files to somewhere else

- you do not need a separate network for simple use cases. Just encrypt traffic if you have multiple servers, odds are you only have one here anyway.

  * You don't have to rotate what doesn't get out.  Limit ingress to relevant IPs reduces this surface area a lot.
  * SCP to a system built for storage.  Not really essential for many systems - system logs are fine.
  * Every VPS provider comes with a backup check box.
  * Tailscale is really simple.

This is how you protect one small server, you don't need cloud for this. But if you do use it for a small server you'll overpay a lot in relative terms and small amount in absolute terms.

Cloud is just merchant of complexity

> VPSs being “easy to manage” is a strong option full of assumptions.

There are definitely many footguns with managing a VPS but I think the threshold to get vaguely competent with a VPS is not really that far off with getting familiar with the average cloud platform - which comes with its own dangers, like the near-total inability to put an upward cap on fees that that person found out with Netlify recently.

Having a $5 VPS and knowing it's never going to cost your more than $5 might balance out a lot of things on the other side for a lot of people.

(And, as a bonus, it comes with the benefit of having a better idea of what is going on on the actual computer which is running your code.)

Platforms like https://coolify.io/ (which I have not tried, but looks interesting) seem to give you some of the abstractions that you get in cloud platforms to save you having to mess with too much low level stuff and become an expert in a billion separate systems.

If you have Debian with automatic updates that does most of the heavy lifting for you. The hardest problem I have is resisting the temptation to just install everything, because the cost to do it is capped at my VPS monthly fee.

So yep, it comes with a lot of assumptions. But so does everything!

a basic bitch static website takes three commands in the terminal and a very basic nginx config file to setup on a completely fresh ubuntu VPS. you could read a guide from 2012 and it would work perfectly fine.

is this harder than dealing with cloud 'platforms' and their ever-changing UIs, APIs, SDKs?

maybe? You go on Cloudflare's console, click on "pages", click on "new app", name it, drag your folder of website files in and you are done. automatic deploys are also available by linking a github account.

It's probably one less click to get a droplet turned up on Digital Ocean, and then you have to configure nginx or caddy (which is like five minutes if you've done it before, 30 if you haven't). Probably worth doing if it's your livelihood and you're afraid of Cloudflare disappearing, but if it's just a mess around... eh.

what would be the price-tag here?

Yes, this is harder. I don’t want to mess with maintaining or configuring nginx.

then install PLESK or something, it's dead simple.

+1 vote for virtualmin

`sudo apt-get install nginx` `vi /etc/nginx/sites-enabled/default`

add your config, use chatgpt if you want, save `nginx -s reload` and bam! you're good to go, practically forever.

This is why I will always be employed.

I won't run a VPS. I'm entirely capable of doing it.

Badly.

I'm a mediocre Linux admin, at best. The current environment is so dangerous, these days, that it's worth your life, to have your essential services run by a mediocre admin; even if I can convince myself that I'm the best (spoiler: I'm not).

I'll generally run shared hosting, or managed servers.

I can't speak for you, but it's really not too difficult to host some static content on a VPS. At the very worst, the VPS is compromised somehow, and you refresh it from the admin panel.

I'm not talking about static content.

I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.

It's easy, sure. It's easy to create an insecure server, that can be pwned. I know of which I speak. I have done just that.

"A man who holds a cat by the tail, learns a lesson he can learn in no other way."

- Mark Twain

It's easy to create a secure server.

Use private keys. Use a firewall (ufw is really simple) and only expose your reverse proxy (e.g nginx or haproxy). Use docker to run your crap.

Any software engineer should be well capable of setting up a secure server. It really is simple.

None of that stuff is “simple”. Git pushing and having Render.com auto deploy is, however.

Using private keys is not simple? sudo apt-get install ufw and ufw allow https is not simple?

The whole process takes maybe 30-60 minutes to setup for someone completely new and following guides.

Render's is simple, true, you just pay $300 for 1TB of bandwidth.

It's crazy how much Merchants of Complexity fooled devs into thinking that running your own server is complicated and you need to pay 1000x to save few minutes of your time.

Merchants of Complexity didn't fool me, trying to run my own severs convinced me.

What did you find difficult in running your own servers?

Even trying to figure out which OS, for a start.

Backups, recovery on VM host failure

Blows my mind that tech people don’t see that as simple. The comments on this post did wonders for my imposter syndrome

> Any software engineer should be ... It really is simple.

I get that, a lot. It's the "No True Scotsman" of tech. This is also used as a way of validating college-style leetcode.

Let me introduce you to my GH Activity Graph[0]. See all that green? That's pretty much all coding in Swift; mostly in shipping apps and whatnot. There's a bit of PHP, for server-side stuff, but I like to spend a lot of time, coding frontend app stuff.

Every minute I spend, being a Linux admin, is a minute that I don't spend on executable code. I know that there's a number of folks, hereabout, that can code circles, around me, in Swift, and a few more, that can code circles around me, in PHP, but pretty much all of you, can run circles around me, in Linux admin. I'm not especially interested in competing, there; especially since a number of you are likely the folks that would Do Bad Things to my server, given an opening.

[0] https://github.com/ChrisMarshallNY#github-stuff

Every minute you spend on here, is a minute you don't spend on executable code as well, yet you're on here. Perhaps because spending every productive minute on coding is practically impossible.

You don't have to be a linux admin to be able to setup a properly secure server, nowadays it's quite trivial. I bet you, that if you tried, you would be able to do so in less than an hour (at a relaxed pace).

We get lied to about the complexity of hosting by the cloud. I was able to host forums and game servers on VPSes when i was a young teenager, and I'm not technically gifted, it just wasn't complicated at all.

Well, we choose where to spend what free time we have. I like hanging here. There's really no pressure to perform. I'm too old to play insecurity-sop games.

I tend to like shipping stuff, which means taking Responsibility for its operation, maintenance and security.

That often means a lot of "not fun." My servers are working servers. They have data and capabilities that are important to a lot of "not-Chris" people. It's my job to make sure that they get what they are [not] paying for (I write free stuff). That can be a bit stressful, at times; especially when some bad actor is making life miserable for me. I'd much rather that be someone else's problem, where I can write an email that says "Make it so, Numbah One!", instead of spending two days, wrestling with config files, and CLIs.

Right up until someone runs a container with 8080:8080, which unfortunately bypasses ufw and the container is suddenly exposed to the entire internet .. :|

Reading the GitHub issue about this is somewhat entertaining: https://github.com/docker/for-linux/issues/690

People are getting hacked a lot because of this, and docker doesn't seem to care all that much.

Oh man, that really sucks, to the point I would consider against using Docker for anything deployed. 5 years and such a basic security issue goes unfixed.

The minute you implement a firewall, you failed

Firewalls are made for two things:

  - packets alteration (iptable table mangle)
  - applying filtering on behalf of a badly configured OS

So, if your case and if you want to prevent remote access to your database, you have a bad way: create a firewall rule to drop connections to tcp/3306

And you have a good way: configure your sql to bind to ::1

The firewall way requires two configuration (hence: complexity) and hide your intent : the mysql say : "I accept connections from everybody", and then the firewall say "I deny all connections".

While the good way is clear and sane : one component who say : "I only accept connections from localhost"

would a cloud be securer though?

I guess you gain some security, when you don't have to worry about some things. But you also lose some security, because of the added complexity.

good thing there are easypanel, flightcontrol or portainer etc that can make it easier to self host app these days

sure it still needs work but much much less everyday

> A man who holds a cat by the tail, learns a lesson he can learn in no other way.

What a silly quote. “No other way” except all the other ones. Everyone watching the man and the cat will learn the same lesson. Everyone who hears the story will learn the same lesson.

I never held a cat by the tail, nor have i ever seen anyone foolish enough to attempt it, yet I am certain I know what happens next.

I think part of the point of the quote is that a man who would try to hold a cat by the tail is an example of someone who can ignore the experiences of other people.

People FAFO when they should know better all the time.

> I'm not talking about static content.

I think the person in the original article was, at least for the sound file.

> I need the whole enchilada: DB, Web Server, Dynamic Languages, etc. Also, for a shipping, production application, with hundreds of users; where privacy and security are of paramount importance.

It seems quite simple to me, you either learn security, or pay for the expertise of someone that has. You need to decide whether it's worth your time.

I would suggest one thing, though - even with the likes of <big cloud>, they will only provide security in limited cases, i.e. DDoS. Nobody at <big cloud> is going to make sure your application logic works correctly - they clearly won't even make sure you use their resources within sensible bounds.

Yes, it’s actually a pain in the ass.

>> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud.

And people wonder why there's only half a dozen websites any more and the "long tail" has vanished. You can post 3.44MB of audio to Facebook or Twitter and know that you'll never be billed for it.

I don't think the Netlify case has happend due to serverless architecture. Yes, there are many technical issues with serverless architectures, but getting a huge bill for incoming traffic is independent from a serverless architecture. You could also put your own hardware in a colo and pay for traffic. If your data center provider does not protect you from DDoS and simply bills by the TB, then the same situation would occur. Of course, one could argue that the prices per TB are much cheaper in a colo situation compared to Netlify and therefore a 100k bill is very unlikely to occur, but then I would make this the main point: absurdly high traffic costs and no DDoS mitigation, and wouldn't put it behind a "serverless horrors" title.

If your stuff scales to infinity your money burn rate also scales to infinity. A single server in a colo can only do so much damage to your wallet before it’s saturated.

Sure, but according to the article, he received the 100k bill because of peak traffic of 60TB of bandwidth per day over several days. If you have a dedicated server in a colo, then a 10GbE connection is very common these days. With a 10GbE connection, you can serve 100TB of traffic in a single day. Hence, if the colo had the same prices as Netlify for traffic, then a single server with a 10GbE connection would be sufficient for such a high bill, no serverless and infinity scaling required, just absurdly high traffic costs.

No Colo charges the same insane prices for Traffic as Cloud Providers. Hetzner charges 1€/TB over 20TB for their 10GbE Servers

Yup, this is exactly my point. It's not the serverless architecture, something you set up on Hetzner yourself with OpenFaaS (https://github.com/openfaas/faas), that is to blame here but the insane pricing scheme for traffic.

... which you generally get with serverless offers.

Correlation does not imply causation. There is no inherent reason why a serverless architecture would force a provider to have insane high traffic costs, hence it doesn't make sense to put blame on the term "serverless" for that.

On the other hand, if one could explain such causation, then I would be very interested in hearing about it.

Strong correlation is enough to advise against using serverless to people who might not be experts and might easily fall into such traps.

> Correlation does not imply causation.

Correct, but the effect on the wallet doesn't care if there is a causal relationship. And since my wallet is closer to me than a philosophical debate, the conclusion I reach in this case, is to move my site ;-)

The whole point of serverless is to trick people into insane pricing.

Nah.You get a dedicated box at hetzner and get unmetered 1gbit and use as much as you want as long as you dont do mining or piracy etc.

The creator of the blog is Andres who is also building coolify[0], a heroku/netflify self-hosted alternative. I wanted to give a shoutout to him and his product. I've been running it for a few months and love it. Really was has been kinda the secret sauce of ease of use for me to start self-hosting things like changedetector, jdownloader, vaultwarden, etc.

It also has a pretty nicely growing community where people are contributing new templates (I added one for Syncthing) and helping each other debug.

The only issue I've had with it is things kinda fall over if you run out of disk space, which happened when I was running on an instance with just 10GB storage. So a little better alerting or prevention around that would be great but otherwise it has been pretty solid.

https://github.com/coollabsio/coolify

Perhaps I'm overreacting, but I had my personal site on Netlify and decided to move it. I don't need anything more than a dumping ground for HTML, and Netlify was "good enough" (I wasn't aware that this issue existed)

What brought it closer to home was the characteristics of the recently affect site (same number of daily visits, not popular (a bit niche), etc) where similar to mine.

Moving is easy enough, mostly requiring a DNS update, since I prefer to build the HTML locally and then just dumping it somewhere.

What struck me as odd, when looking for an alternative is that almost none of the popular solutions Netlify, Vercel, Cloudfare (AFAIK) actually implement a spend limit.

This seems like such a basic thing to do...

Cloudflare has DDOS protection which you can set to basically paranoid and once a DDOS starts then everyone will get a captcha. This limits your spend quite effectively.

And also stop me accessing the site entirely. I don't know why, but every site with CF DDoS protection set to paranoid gives me infinite captchas that I can never pass.

For me this happens because of some combination o|

- firefox - userAgent spoofing claiming I'm chrome - pihole

If I change enough of these, I get through the infinite loop. Does not make me happy.

Cloudflare pages has unlimited free bandwidth so you dont need a limit because traffic is free.

(I work at Vercel) We do have DDoS protection, and spend limits. Working on releasing some further improvements to both of these imminently as well.

https://vercel.com/blog/introducing-spend-management-realtim...

Vercel is 2 out of the 4 issues in the article. Hard no, man, come on.

Based on the info in this thread, Vercel is the only company I'd leave for Netlify.

Also see the Netlify comment thread linked from one of the Reddit posts (https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...), where one of their representatives specifically says that if you suffer a DDOS they will do nothing to stop you from getting a bajillion-dollar bandwidth bill, even if you're on the "free" tier.

That's my favorite thing about all these services. None of them have an equivalent of a stop-loss. They'll notify you IF you setup an alert, but that's it. Often times the alerts are way after the damage is done, with some billable metrics often lagging in reporting/alerting by hours. The default should be safe with easy options to increase the limits.

oooh i really like that stop loss example. having a netlify serverless website is basically selling a put option without receiving any premium.

The premium is the website itself ;D Still, potentially unlimited costs is wayy too much risk.

Selling a put has a fixed downside, so perhaps selling a naked call would be more appropriate? Unlimited loss potential! :^)

You can totally build a GCP function that will tear down your site if a billing alert hits the pubsub bus, but you can also set billing caps for a bunch of the APIs.

Why not add it directly into the interface where you configure the site to begin with? It's a pretty reasonable ask.

I don't really know although I can make up an answer but it contradicts

> None of them have an equivalent of a stop-loss.

Cynically, it's because they want to take your money. Practically, it's because a company that breaks your thing because you didn't pay, even if you asked it to, just isn't going to be a popular move.

It takes real leadership to back a product feature like this. That's why.

It's probably quite difficult to boil it down to something as simple as an "off" switch that'll limit costs for them, meaningfully work for all services, not be abused, not be confusing, and not result in the customers that actually make you money accidentally getting important things shut off unexpectedly under predictable circumstances.

It's probably easier to make it so they can set up alerts and have a human on their end evaluate what steps are worth taking to cut costs while maintaining important availability.

From what I can tell Azure has more robust stop-loss options. But business considerations for the provider aside, there are absolutely tradeoffs between reliably limiting your spend and avoiding taking a business down because of a legitimate spike.

Absolutely but that tradeoff should be for the customer to make.

What you're saying is that I can build a cost control function, not that GCP has one.

Yes. GP's claim is there is no stop procedure. just because it's not a simple toggle switch doesn't mean it's not there.

I remember trying AWS Glacier as a broke university student, thinking it was a cheap way to back up my data. I was wrong.

A decade later I played around with AWS, set up a free 1-core VPS. It was fine for a year or so (I ended up not using it at all), and suddenly I got a bill. It wasn't much, but it reminded me how these services make it very hard to be aware of your bills until you get them.

I was trying out Azure recently, and I found it too confusing to make any sort of quick risk assessment.

It's clearly just for people for whom "What's the worst that could happen?" is always a rethorical question.

Glacier IS a cheap way to backup data, in fact the cheapest of you ignore the scam "$200 for life" type things. It's about $1 per TB per month.

It's not cheap to restore though, at about $90 minimum per TB.

So it's best as part of a 3-2-1 system, not for frequent restores.

Lambda and GCP Cloud Functions do have cost controls. You can specify the maximum number of instances that can be spun up at any given point.

I think the UX can make it more friendly to specify the types of traffic and cost you’re expecting but most erroneous spikes are the result of a customer misconfiguration or a legitimate spike in traffic.

It’s the former which are problematic for customers. They can request a credit and we (GCP) try and accommodate.

It’s not a feature designed for profit. But it also tends to fall below the cut line for prioritization to fix. But it’s in the backlog and doesn’t go unnoticed.

(Former PM on GCP Cloud Functions)

Do users setup cost controls before activating the service? Or is this something they need to know how to do and do on their own?

Separate from billing, the product-native cost controls which prevent over-consumption is configured at the resource level.

gcloud run deploy ..... --max-instances=N

What you really want is prepaid instead of post-paid billing. Charge up the account with $100, and stop working and delete my data if that gets hit, with an auto renew option to recharge some amount every month.

That seems a little reductive. I don’t think it’s unreasonable to say, my current spend is ~13k a month. I should be able to set a 20k cap, such that non fixed billable like bandwidth or map tiles get paused while someone determines whether we’d like to proceed at that rate. There is a not so fine line between scaling for a rush/event and a resume generating event.

Okay, fine, that's what I want. :)

I wonder whether they are running out of money which then causes these things. Netlify used to be a very nice and unique service, but then came GitHub, GitLab and Cloudflare Pages who all provide roughly the same services for less money.

It used to be we will take your server down. Now its we'll DDOS your serverless website and leave you a 100k bill.

I'm not sure how sustainable such business model is. When you owned the server, you could unplug it. Now you have no way of knowing if somebody is going to hit your /api a million times per minute

I also prefer to get a (decaf) coffee, listen to some music while someone DDoS'd my VPS. I prefer to pay few $ / month for my VPS instead of paying thousands and "survive" the DDoS.

I pay $10/year for my VPS and host a WordPress Woocommerce store on it... It doesn't get much traffic, but it didn't take long to pay for itself either

where?

true. At what point could you say they are accomplices to the attackers

Ah the Yelp biz model:

"would be a shame if somebody read a bad review about your biz, pay us to remove it"

to Netlify

"would be a shame if somebody DDOS'd your serverless website, subscribe to our DDOS protection plan"

Reminiscent of booter sites behind cloudflare.

AWS guarantees protection from all DDoS attacks done at level 3-4 (Google aws shield). If someone calls your api million times then there is throttling.

So instead of DDoSing someone, you could make some not-so-large amount of requests to their APIs and instead of taking down their servers, you'll just take them down by bankrupting them with a huge invoice?

Representational state terrorism

With default settings an attacker can run up-to 10000 req/sec on your api gw which would result in a sizable bill if left unnoticed. So with AWS you have to configure throttling and AWS protects you from low level ddos. How do you save yourself from a huge traffic bill in a VPS?

AWS also has WAF to protect from DDoS , it is expensive but may save a day if you urgently need a protection.

Can you use free tier under a fake name? To drop ridiculous bills. How do they validate the customer?

There is no free plan, there is just free limit - you must add a card.

And what if you add a debit card without money? Or add a disposable card and then the next day you deactivate it?

Nowadays everyone push you to add a card to open free* account. Such disposable cards is a solution to this issue.

Your debt will be sold to a collection company.

Can they find and owner of the inactive card? If I put a fake name to their system

Your debit card bank has your real name and you agreed to share it to get the card in the first place.

In many countries this is meaningless.

I just ignore any horseshit that goes to collections outright

What a great achievement of our society. Inventing a financial instrument to buy and sell debts.

It really is. It decreases the risk and necessary KYC checks so a lot more people can get serviced that would be shown the door in a world where you can't sell a debt.

Many don't accept prepaid cards.

Is that new (relative to some previous point in time)? They don't have my card details.

just use a virtual card and then change the spending limit to $0? or freeze it

That still passes your personal info.

at least you wont get charged without your knowledge... which in the end, is the first step in getting this non-sense fixed, for a person... of course the government should fix it for everyone. This shit should probably be considered fraud anyways... companies get free passes all the time. My name (s9df898r32h) is my company, I should be able to steal whatever I want /s

Why are such large companies so incompetent though...

You will have to pay it AND a fee for fraud, and possibly their legal fees for tracking you down. You also will have a difficult time arguing that you were in good faith with this kind of shenanigans.

Bouncing checks are not a life hack to get stuff for free, they're a crime.

I thought they were the ones committing fraud ... either way you have to be crazy to subscribe to those services

Not really, no. Fraud requires intent.

So what do I do if this happens to me on the free tier? Just never pay the bill and leave? Will they come after me?

I feel like cloud vendors overcharging for things they have the right to charge you are like a car mechanic that tells you you have to pay him a billion dollar for a routine maintenance check of your car because when he was fixing some small item, it broke and he had to get another one and put in in place, but then it broke for the same reason (that is not the mechanic's fault or the item constructor's), so he kept doing that on an infinite loop but after 2 weeks of him doing that without sending you any info, you went to the garage to check on him and told him to stop, so he stops and bills you for 999999999 items that are broken.

As a small user and not a large organisation, I'll never be able to read the full documentation on throttling every nook and cranny, I want to set a fixed spending limit that if reached, will kill the whole thing, even lose data if it must. But they won't do that because it hurts their bottom line if people are careful with budgets, and sometimes it's tricky to compute billing ahead of time.

Well there are differences. The mechanic from your example caused the situation. Netlify didn't and it might be hard to distinguish between unwanted traffic and a SaaS having overnight success.

A more accurate allegory would be if you told the mechanic to do whatever anyone who knows the registration plate asked, and he did so.

I recently signed up for an OpenAI API key, and by default they enforce a quota after which they disable the key, and when you’re ready to serve more requests you increase the quota manually. I’m surprised to learn more companies don’t do this by default to save everyone from these kind of surprise bills. Although now that I think about it, I suspect there are lots of cases where the users just end up paying these.

The difference is that for OpenAI, inference costs them real money so they have skin in the game. When Netlify overcharges 1000x for bandwidth, they don't actually care if you don't pay because it cost them next to nothing to provide the service.

I think the biggest Cloud horror, probably not serverless, is that we have accepted to pay for inter-AZs traffic just in case the Cloud provider fails. In other words, they charge (and a lot) for a mitigation for the problems they might have.

Everyone in this thread is talking about how this is not a Serverless problem, but a cloud problem. That's correct, but the point stands. Bandwidth is pure margin and to me, it's a bit scummy to a) charge this much for bandwidth and b) offer no tools to combat attacks that aren't in the customer's control. According to this thread[1], Netlify doesn't even have an option to take a site down for a while if it's under DDoS attack.

Netlify has multiple options to solve the problem: billing controls, request throttling, setting bandwidth limit, and waving of overages if it was a genuine DDoS, but it doesn't seem they are interested because that'd be like killing the golden goose. Just look at all the things bunny.net CDN offers [2].

It's disappointing how we acquiesce to explanations of cloud providers that are motivated by nothing other than greed.

[1]: https://answers.netlify.com/t/limiting-bandwidth-traffic-to-...

[2]: https://support.bunny.net/hc/en-us/articles/360014190440-Und...

How is no spending limit a serverless problem?

It's a cloud problem.

Edit: Well, if you have small cloud servers they might go down under an attack. And if you pay only outgoing like in AWS that's good for you.

It’s an architecture failing. Setting maximums is critical. If you choose a platform that doesn’t support that more fool you. It’s implicitly easier to have a ‘circuit breaker’ in a traditional environment, but still warrants consideration. Load test for success and failure.

It's not a cloud problem. The cloud is just somebody else's computer.

It's either a billing or an architecture problem.

Either let me throttle requests to prevent overbilling, or cut me off when I hit a certain value.

Because serverless 1) uses pay-as-you-go more often than traditional models 2) scales better to achieve massive usage.

If you DDoS my $5/month DigitalOcean VPS, it's not going to increase how much I have to pay DigitalOcean in terms of compute, and most likely the VPS will buckle under load before it can actually rack up massive transfer costs (since those are also pay-as-you-go for DO).

> How is no spending limit a serverless problem?

> It's a cloud problem.

It's a serverless problem because it can be mitigated if using the cloud, but not mitigated if it is serverless.

I have a cloud instance. I don't have this problem and never will. If I had serverless instead, I will have this problem.

Seems very inspired by Molly White's page [0]

[0] https://www.web3isgoinggreat.com/

its in the style of long runnning dailyWTF: https://thedailywtf.com/

There used to be a whole host of specific fuckup like list site, sadly I can't find them anymore.

This is brilliant, there goes my afternoon. Thank you

A kill switch seems a practical and trivial solution. E.g. something toggled when a monthly bill exceeds a configurable amount.

I’m guessing many hobby users would set that to $100, end of story.

I don’t understand why it’s so unpopular to offer that. I’d imagine the providers would benefit too in the long run.

It would be practical for the user, but not for the company.

Why does this not exist after years (more than a decade since EC2) of cloud computing?

Because it is not good for the VC investors.

AWS Lambda has had this knob for quite a few years, it’s called “provisioned capacity”, and it defines an upper bound to function concurrency.

While helpful, this is not the same. E.g. a global kill switch could be harder to overlook and easier to manage.

You can set a hard budget on most cloud providers, including Azure etc. I actually did this thinking I was just setting a budget alert and that was a new and interesting way to interrupt production.

This is a highly welcomed feature however, I don’t think AWS has it.

> You can set a hard budget on most cloud providers

Note that the submission references serverless providers, specifically netlify and vercel. I don’t think these support an easy to configure kill switch - in any case it wasn’t easy to configure in the scenarios listed on the page.

I agree with the consensus here that the cost issues are a cloud problem, not serverless. Ironically, something like AWS lambda has a default 1,000 simultaneously activated lambda limit, so the platform defaults to throttling.

Not that serverless is perfect.

Ye, especially with something like Netlify or Vercel, I'd bet the vast majority of the costs are CDN bandwidth, and it never even touches the functions themselves. Could just as easily be in this situation with AWS CloudFront and an S3 bucket.

I have seen AWS Lambda generate large bills even with the 1000 concurrent executions limit, but with the help of upstream services. My memory of it is hazy, but I think over a weekend, a function with some dodgy retry logic managed to generate something like 21TB of CloudWatch logs.

Related submission from yesterday:

Netlify just sent me a $104k bill for a simple static site | https://news.ycombinator.com/item?id=39520776

I don't think infinite scaling is desirable. You probably want to say "if the value or rate of increase in xyz" metric reaches a certain level, don't scale further, or start scaling down. As a requirement for a serverless deployment, not an option.

Bandwidth charging models from hosts are crazy. They don’t model their upstream costs in any way. Transit providers charge based on 95th% and ports scale based on highest burst not volume passed. I’ve always hated it.

Costing money is one thing, but not throttling a queue consuming serverless function and effectively DDOSsing yourself is the true horror story working with serverless :)

That's kinda not "server less" specific though.

There's no end of stories of people discovering 10s to thousand in bill run up on AWS (ir Azure or GCP) because they foot gunned themselves and didn't set up alerts and/or automated service suspension using the cloud providers available tools.

Part of me thinks you shouldn't be allowed to provision EC2/S3/Lambda/whatever without first having set up a Cloudwatch alert and a rule to stop your service if your bill goes past expected/acceptable limits.

Of course Amazon (or Microsoft or Google) could make that _way_ easier than they do. It'd be nice if there was a simple form in your account setting where you could configure it to suspend services if your projected monthly bill passed $limit.

I think it's a more generic "horror story of working with other people's computers".

This story is also good: https://news.ycombinator.com/item?id=25372336

This feels misnamed.

It’s not really the serverless part that is the problem but rather inability to limit it

eg cloudflare has serverless stuff that you can hard limit

These are just cost nightmares, what I love (hate) is when said serverless gives you the any kind of bizarre errors and some solutions require you to do some arcane "saw it on stackoverflow sub-comment".

Oh and it's not like you can jump into the console to debug.

I gave up after not even vercel could figure it out and merely told me that I should replace said package.

The thing I don't understand is why people go for cloud route for the simple sites, when there are options like Github static hosting or self hosting using secure tunnels from Cloudflare or Ngrok? The risk presented by these horrors stories greatly outweigh possible complications from setting something quickly on your own.

Minor criticism, I get that this is a page that collects links to other pages talking about the horrors of serverless. Sure. Makes perfect sense.

I would like to see some small amount of commentary on each page that covers what I am about to read. Not much, but half a paragraph of introduction would be nice.

On the one hand I am glad to see sceptisim of serverless gaining mainstream acceptance.

On the otherhand, I fear it will just turn into an unreasoned backlash against technology that is often useful if not as often as its early proponents said it would be (See NoSQL).

Don't Netlify/Vercel run on top of the likes of AWS/GCP? That means they need to markup on those already inflated hyperscaler markups for data transfer.

Is there the opposite website, listing cloud services that do have spend limits?

I've done quite a bit of cloud cost optimization in the past (sometimes referred to as finops). There's a pedantic discussion to be had as to what is serverless and what is PaaS, but that should be saved for another day.

The underlying problem I found working with teams is that they all too frequently wanted to use these cloud-native technologies (serverless, etc.) which means thinking in microservices by definition, but usually architected for monoliths. This was accomplished in a few ways: 1) the serverless job did everything for you so in effect it was a monolith 2) they didn't want to use any intermediary storage/cache layer and relied heavily on both sides being up and available (synchronous brittle systems) 3) they went TOO hard in the microservices direction and made things way too complicated 4) their testing suites were half-baked (more the industry/tools fault than the users tbh) and weren't sufficiently air-gapped or mostly incomplete.

If you've been around for more than a decade - we have seen this before: AWS specifically sold the concept of "lift-and-shift" to C-suites a dozen years ago and now that MO is so baked in it's going to take a new generation of services with the right selling points to undo it. Lift-and-shift doesn't work because, as stated above, monoliths aren't for the cloud for a sufficiently sophisticated system. It's the reason DHH and others are advocating for "cloud repatriation". And then there are the small cases, comparatively speaking, like the ones listed on this site. These examples are what happens when you give very powerful tooling to people who do not understand it or the implications of it. But too, this is also partly (maybe even mostly) the vendors' fault: "From localhost to https, in seconds. Deploy from Git or your CLI." is the perfect selling point for a DIY-er or someone running a small business. But what comes along with that low barrier to entry is a downside that could be very expensive when the s**t hits the fan.

in terms of web hosting, why does hosting using serverless / paas means not using a cdn?

it also baffles me that these services cannot tell between legitimate traffic and ddos.

Presumably the costs for a platform are passed on to the platform's customers from that platform's cloud provider (with profit). When a spike happens due to, say, a DDOS, does the platform first have to negotiate with the cloud provider to waive the cost?

Presumably the costs for a cloud provider are passed on to the cloud provider's customers from that cloud provider's operating costs (with profit). When a spike happens due to, say, a DDOS, does the cloud provider first have to negotiate with its downstream suppliers (e.g., energy, network peering, etc.) to waive the cost?

...

How far down the chain does this go? Or maybe insurance steps in at some point -- or every point -- to break the chain?

Not necessarily - there are some costs we just eat. However in the event of outside actors having a reputable CDN in place to mitigate these things is worth your time and money.

It's all just lost profit. It doesn't cost AWS more than it does you copying 190TB to your NAS over LAN.

Silly question, but while you're still at the making (expensive) mistakes stage, why not set up a cheap on-prem box that can be switched off/trashed with ease? When you're satisfied that you've fixed all the obvious weaknesses, move the software to The Cloud. Or am I missing the point?

Something feels a bit off about the creator of this site also pushing a competitor. On one hand, they're upfront about it, and on the other.. they're upfront about it. At least it might prompt the companies in question to be a bit less scummy.

Which competitor do I push exactly? :)

(I created serverlesshorrors)

The link to the "competitor" is front and center on the site, so I imagine what you're really trying to say is that this is NOT a competitor for the following reasons: <missing>.

Otherwise this seems like a bait question:)

Andras (who created both Serverless Horrors and Coolify, which I imagine you're describing as the competitor here) is a pretty solid guy from everything I've seen and I've never seen him ask bait questions or engage in bad faith in discussions here or on other platforms.

Although he calls Coolify a Netlify 'alternative', it's 100% 'bring-your-own-server', and is not at all serverless. Coolify is basically a (very nice and featureful) fancy frontend to Docker with integration with Git.

Coolify is free if you want to install it yourself and don't mind the (minimal) tinkering needed to get it set up and keep it up-to-date, although you can pay a monthly fee to have your own servers managed for you, the main benefits are either to support continued development, or to access priority support. You still need to bring your own server, even if you're paying the management fee.

It's not a competitor to Netlify, it's an alternative.

Just to clarify, I didn't think that Coolify is a competitor.

What I was getting at is that we now have a thread with 4 replies: (1) you pitch competing product (2) what product? (3) Coolify (4) it's not competitor because ...

I just wanted to point out that comment #2 only seems to prompt question #3, whereas the helpful reply #4 created by yourself obviates the need for #2 and #3.

I agree, especially because I'm not even sure the pushed competitor solves the problem that's being highlighted here.

All the horror stories so far seem to be about people using a lot of metered resources, like bandwidth, and then getting billed for those resources.

With the pushed competitor you can self-host your Vercel or Netlify alternative on your own EC2 instance - but then if someone downloads 60 terabytes worth of data from you, you still get to pay for that bandwidth. You still get an unexpected large bill.

Sure, but take a look at this: https://getdeploying.com/reference/data-egress

60 TB

On Hetzner (VPS provider): $1. On Netlify: $33,000

Reply on news.ycombinator.com