Dude where's my coldstart? | Hacker News Reader

Dude where's my coldstart?

openfaas.com

45 pointsalexellisuk35 comments

Maybe I’m just turning into one of those old programmers who doesn’t keep with technology, but the incredible inefficiency described and the workarounds for it are just mind-boggling to me.

Then again, maybe it is cost-effective to burn seconds of CPU time and tens to hundreds of megabytes per request handled.

I’m sure some C programmer from the CGI days would be floored by how much resources multithreaded ruby and python app servers use.

Agreed - the cloud providers are laughing all the way to the bank.

On the other hand, I've seen organizations run an average of 4% utilization for their VMs.

Despite the absolutely terrible inefficiencies, the scaling benefit these """serverless""" technologies provide is hard to replicate with traditional software. Throwing a Javascript file into some *aaS platform and having that run on a datacenter near the user, automatically upload the output to storage near the user, and scale up dynamically if there's a sudden influx of thousands of users is very hard to accomplish manually without reserving tons of VMs or using some kind of data center API with auto provisioning.

Most businesses have no need for such a system, of course, and the costs per function execution are astronomical in comparison to a native program, but for some use cases, the benefits outweigh the costs.

As for Ruby and Python, the cost is measured in dev time rather than server costs. Servers are cheap compared to devs, and if a dev can pump out a feature three times as fast because they can make use of a simpler framework, you can afford the overhead of slower programming languages.

For this reason I'm watching the Rust space with interest, because the speed is much closer to C++ than to the traditional Java/C# servers most big applications are written in, but the frameworks are slowly shaping up to be similarly complete. Of course, there's a huge difference between Rust and most other languages that cannot be overlooked, but services like Vaultwarden show an interesting opportunity in the server space. It'll still be a while before the tools and frameworks reach a similar level of usability, though.

> having that run on a datacenter near the user

This is the biggest deal IMHO. FaaS in CDN edge DCs are remarkably fast and affordable. If you can respond to an API call faster than a request can get through to a load balancer, you're providing quite a UX improvement regardless of cost. For most people running stateful apps this requires some large tradeoffs in the form of the data layer, but they are getting more reasonable over time. In the case of CloudFlare which may have the strongest edge function hosting now, it used to only support its Worker KV data store (fast but KV only and globally eventually consistent ~10 secs). More recently they have added support for fast connections to their R2 (S3-like) service, and now they have a new service that can proxy to MySQL and Postgres databases.

https://blog.cloudflare.com/relational-database-connectors/

Its an exciting space, because its not just about CDNs: as 5G towers are rolling out, they are building in edge compute clusters along with them. So now you have a 5G iPhone with faster single-threaded performance than your desktop able to get 1Gbps down and faster response times because of its path to edge compute. ISPs are also getting in the game:

https://www.coxedge.com/serverless https://www.limelight.com/resources/data-sheet/edge-compute/

I think that's sort of against the pareto principle. Assuming you have the physical data near the user. Great. But most of these functions then go on to fetch stuff from elsewhere with its own delays and problems. Even if S3 or cloudflares equivalent is replicated everywhere there would still be a networking delay and context switch waiting for a response.

If performance is THAT important you would be better served with a HUGE machine with a huge cache of the database. That's not something you can do with a FAAS. Other than saying "hello world" fast. I can't think of a proper use case where the performance of FAAS would be realistically better than a dumb monolith. The performance and complexity overheads of FAAS make the case even harder to make.

CF’s Durable Objects product which I neglected to mention above attempts to be that huge database and cache. The CAP theorem isn’t violated, it just embraces consistency while relying on the partition tolerance of their POPs and backbone. Their KV store is more about availability.

https://blog.cloudflare.com/durable-objects-ga/

I don’t disagree that a huge database can handle an astonishing amount of load but at often unreasonable expense. There as many cases where FaaS is dramatically more affordable to run, both in compute cost and ops time. There are others where its costs spiral out of control under load and with dev effort to fit its constrained programming models. Its hard to say its entirely a trendy frivolity when there are many use cases where its the best tool for the job.

> This is the biggest deal IMHO. FaaS in CDN edge DCs are remarkably fast and affordable. If you can respond to an API call faster than a request can get through to a load balancer, you're providing quite a UX improvement regardless of cost.

How much faster will this be? I think even for servers on the other side of the planet, the max ping will be something like 500ms. I think most pings in the same continent will be something like 100-200ms. On the other hand, a Node hello world was 164ms. That means that compared to just being in the same continent, you don't have much of an advantage. Python is ~30ms for a hello world, That's better. On the other hand, the "competition" is marked as 250ms for a cold start. That's a lot, and I doubt you get the benefits of being close to the user at this point.

There may be some things I missed, I don't know much about that space.

I can really only speak to CloudFlare workers in latency sensitive, user-facing settings, but from my experience their cold start times are insanely low, like 5-10ms. They use V8 Isolates (uberjar for js?) running WASM bytecode, so there is very little runtime overhead. So in that case, there is an actual benefit that being a short hop away, but otherwise I think you're right.

I've also used some AWS Lambdas and GCP Cloud Functions in background data processing settings where there are under 1M invocations/month, so its cheaper than running a tiny instance 24/7. In that setting latency doesn't really matter as much, but I was seeing cold starts ~2 seconds and warm start invocations on my node and python code responding within 30-60ms in most cases. Once the container is up and running its reused for follow-on invocations, but eventually(and opaquely) times out and the next invocation will be cold. People who are using serverless runtimes have historically done hacky things to ensure they always have a running container like invoking the function with a cron job. At that point it seems a bit silly not to just run a small vm / container.

Thanks for the info. 5-10ms seems great, and a real plus for user experience. As you said, cold starts seem to be a killer, especially if you're trying to optimize the slowest requests.

Re: the last part: I dislike the fact that carriers are getting involved in fog computing. It gives them a perverse incentive to do things like slow down VPNs or even disable privacy enhancing technologies (like how some ISPs have been caught disabling Apple's semi-Tor built into the OS.

Also, the vast majority of the world doesn't have an iPhone, so assuming everyone is spending over a thousand dollars on the latest and greatest cellphone will lead to even worse websites.

Yikes, that's more than a bit scary. IMHO Verizon's supercookie and 'Customer Experience Plus' shenanigans deserve massive retribution and restitution such that nobody would dare do that again. If there is one thing that needs to be broken up in big tech its the combo of ISP/carriers and advertising sellers.

> if there's a sudden influx of thousands of users

A $10 VPS will manage thousands of users fine for most CRUD apps. You don't need tons of VM.

You can always make it a... $20 VPS to have some margin if you fear HN hug of death.

You won't get a CDN close to the user or automatic backup, but that's really fine for the immense majority of projects.

It takes one hour to setup a VPS with nginx, python/node/ruby and postgres. Then it mostly runs on its own for years: no coldstart problem, no kubs to manage.

I get the benefit of the cloud at scale, but for thousands of users, it seems really overkill.

It depends on what the code is actually doing, but you're not wrong in most cases, especially the HN hug of death. I host all of my stuff on cheap VPS servers for example.

Some tasks, though, are just terribly slow. For example, crawling a webpage and taking a screenshot of it sounds incredibly easy but turns out to involve a lot of resource-intensive moving parts. The most common solution I see involves running Chrome in a container and taking a screenshot. That kind of code can easily overwhelm even a moderately expensive VPS with just a few clients. On the other hand, most CRUD stuff and frontend-heavy applications would run fine even off a modern Raspberry Pi.

You can't just let something run, though, because you need to manage system updates. It's something you can easily forget if you have tons of other stuff to do, so the laziness factor also helps defend the *aaS setup.

I don't like serverless, I think it's the next step in more wasteful programming by people who can't or don't want to understand the systems underlying their code. We're at the point where shipping an entire operating system along with an application has somehow become the norm, and I don't like the fact that eventually I too will need to learn this serverless crap.

> It depends on what the code is actually doing

Indeed, things like video encoding, machine learning, graph traversal, etc, can be pretty expensive.

But then again, our tooling are so powerful these days you can get away with a lot. Timeseries ? Clickhouse is your friends. Need geo calculations ? PostGis is here. Need to extract frames from a video ? ffmpeg will do that in a few cpu cycles. Data analysis ? Say hello to pandas. Validation, caching, expiration, encryption, workflow, pub/sub, logging, reporting... We have amazing solutions for everything that will let you scale up to the millions of users for few hundred dollars.

In your example with the screenshots, one can imagine putting them in a queue (5 minutes setup with python-rq + redis) or even more low tech, a cron, and suddenly your VPS can take one screenshot every 2 seconds, which is 43200 in a day. That's a lot of screenshots.

Your client need the screenshot immediately ? The browser can take a screenshot from JS with html2canvas.

Sometimes it feels we are using $TECH because there is a whole class of devs that can't find solutions to problems:

- don't want to think about your data structure ? Put everything in no sql.

- don't want to think about your deployment ? Put everything in a container.

- don't want to think about your algo ? Put everything in the cloud.

Of course, in the end, you still have to think about all that no matter the tool you use. You just moved the cost somewhere else, usually paying much, much more later.

In fact, even most of the stuff I get paid to do using Django could be done cheaper with wordpress. I'm already overkilling it.

I think the problem is in a general overcomplexity, ambiguity, configuration overflexibility and lack of reliability with LAMP or Node or Python or GNU/Linux installations.

Today I just can’t make a guarantee for a fresh Ubuntu LTS instance that it will run a 15-line pure C or sh code as I would intend to outside of my attendance, regardless of how easy the requirements or the SLA might be.

As much as I hate serverless or easy scripting language trends, they do the job and do it well.

It probably depends a lot on what the code is doing . I've seen systems with multiple services inside and a couple of them had to do db calls and could not be cached. You can easily end up with 10-30 servers. Also, the cheap versions may not have enough cpu or memory for all systems.

In this case the serverless versions may be better, easier or cheaper. We did not use Python or Ruby by the way. I still think Python is a bad choice for most systems.

> Servers are cheap compared to devs

Yeah to a point, but Python is so much slower than fast languages that your devs will spend longer trying to optimise it and write parts in C++ that you may as well have just written it in Rust or Go or Java or C# or whatever in the first place.

And I don't buy "development in Python is faster" even ignoring performance. Maybe for tiny programs but once you get to reasonably sized ones the lack of good static typing makes all changes extremely tedious and error-prone. Plus you have to spend a mountain of time writing tests that you wouldn't even have had to think about with stronger typed languages.

I can definitely fix bugs in Typescript or even C++ way quicker than I can in Python.

> And I don't buy "development in Python is faster" even ignoring performance. Maybe for tiny programs but once you get to reasonably sized ones the lack of good static typing makes all changes extremely tedious and error-prone. Plus you have to spend a mountain of time writing tests that you wouldn't even have had to think about with stronger typed languages.

This really rings true with me, on both sides - I mostly (have) only use(d) python professionally; on semi-mature projects with some buried bodies etc. I spend my days wishing it was Rust (or that the JS was TS) thinking that that would be easier, or that the bug I'm fixing never would have existed. But then when I spend my own time starting something for fun, and so naturally use Rust to avoid that, it annoys me that it takes so much longer to get going and if only I'd decided to use Python, etc.

At this time it is more efficient to use the rich ecosystems provided by modern languages and just scale up than it is to microoptimize against any perceived inefficiencies of using containers and these ecosystems.

When you have a business where salaries are a much larger percentage than hosting on your overall costs, you're going to add to the second to squeeze more out of the first.

When you have done the optimisation above, and your bosses keep asking for more features coming out per quarter, and you need to hire more, you're going to opt for the new things everyone is learning, so that you can hire enough without making your life all about hiring.

These are trade-offs like any other, and you probably just haven't been exposed to similar constraints or all of this would make at least some sense.

There may be better ways, of course.

In what potential world would it ever make sense to start an entire container for each request?

The entire architecture that leads to such a concept is so dumb I have trouble believing anyone ever thought it could be a decent idea.

Thats how Erlang works essentially.

Yes but Erlang's BEAM as a purposely built architecture that makes this very efficient. Spawning OS processes and threads is comparatively expensive.

Erlang doesn’t spawn containers for every request.

Extreme take: Numbers-wise, that should probably be the default ;-)

In particular, the world runs on power laws, and I'm betting most server apps are boring occasional internal stuff at the not-even-1-qps level. Long tails add up to be the typical project. So yeah, if all the worlds internal django tools can do scale-to-zero with fast cold start on shared boxes with 100X more tenancy than today, that's saving a lot of electricity & hw. Sort of a virtualization equivalent of "640K ought to be enough for anybody".

>In what potential world would it ever make sense to start an entire container for each request?

If you wait long enough between requests (~5 minutes), that is what AWS Lambda does, only with a VM instead of a container.

Disclosure: I'm the maintainer of the open source project - openfaas which this blog post talks about. It also covers other FaaS frameworks and why we see a "cold start" even when using Kubernetes as our platform.

Does't the proposed solution both solve the issue of FaaS and also remove the main benefit? (ie having stateless function with clean state when it starts running)

Also it feels to me like FaaS is good when someone else is running the infrastructure (so you really don't have to know about the underlying infrastructure). Why are people self hosting this? Is it for local test platforms before deploying to a managed FaaS?

One reason: you have an infrastructure team providing the FaaS solution, and multiple other teams that deploy to it; to whom it is 'a managed FaaS'.

So this serverless solution has a watchdog process always running and mapping requests to the correct process? It that more like a server than serverless? Will the next optimisation be launching one thread per request? In which case we are back to square one (:

You're quite right but as a customer I frankly don't care about how do they it as long as management and deployment are easier from my side.

At one point you do hit a limit on what you can do by cold-starting (almost) everything anyway, right? Let them optimize so they are competitive, and now they've given me a good reason to evaluate them.

>Will the next optimisation be launching one thread per request?

That is what Cloudflare Workers do. One v8 isolate thread per request.

The benefit from an app running in a container and already listening to an HTTP port with on top an horizontal autoscaler looks small. I might miss something in the approach but as watchdog requires an "HTTP-level multiplexing for requests" (so the app should listen for http), the app is already running in a container with watchdog: everything is in place to not need watchdog.

From the FAQ:

> Why do I need watchdog anyway?

> Whilst we don’t recommend it, if you’re using a HTTP microservice that conforms to the Workloads spec, then technically you may be able to get away without using it.

This is a joke right?

Reply on news.ycombinator.com