Improving API Response Times by Migrating from Cloud Functions to Cloud Run | Hacker News Reader

Improving API Response Times by Migrating from Cloud Functions to Cloud Run

unloc.app

57 pointslookalook39 comments

> Ideally, we would have used JMeter or something similar, but we could not get it to run locally. So we wrote a small script to walk through most of our endpoints, with concurrent requests, ramp-up time, the works.

I'm pretty sure somewhere north of 80% of load testing tasks play out exactly like this - certainly at least 4 or 5 of these efforts that I've been a part of started with an attempt to get the latest version of JMeter to work ("we should use the mature solution and not reinvent the wheel, right?"), hit major difficulties, and ended up with me cobbling together an ugly but working 100 LoC script that did the job with far fewer headaches than the feature-rich but not really functional Cadillac solutions out there. Apache Bench can do quite a bit in a pinch, and doesn't choke up as quickly as a lot of other ways that you might generate traffic.

Re: Cloud Functions vs. Cloud Run, both are fantastic services and I do tend to prefer Cloud Run lately. For minimizing cold start latency another trick you can use if that's too big a move is to do your function routing within a single global endpoint (obviously this only works for internal-only APIs) - rather than having 100+ endpoints that you hit, you always hit the same endpoint and just include the desired call as a parameter, and route to the correct logic in a few extra lines of code. With all your functionality now passing through the same cloud function call, you're much less likely to incur the cold start penalty on your less frequently called functions, though it comes at the cost of some complexity in code. If you're going through Firebase it also speeds up your deployments quite a bit because you only have 1 function to deploy. Not a silver bullet (and it's also not a good solution if you want a proper REST API, since you're basically forced to jam everything through a POST endpoint), but can be an effective intermediate workaround if going all the way to Cloud Run is too daunting.

Try out artillery (easy to use npm package) next time to spin up a quick load test. Fastest method I’ve come across

We wrote about how we migrated off Cloud Run to K8s (might be your next step):

https://newscatcherapi.com/blog/google-kubernetes-engine-as-...

I haven't read the entire article, but I assume you need someone to admin the cluster? How does the savings in billing compare to work needed for maintenance?

GKE Autopilot[0] has been close to zero maintenance for us over the last few months (less than a week in total for a team with little previous Kubernetes experience). You also get a cluster controller for free so if you are using a single cluster, you are only paying for the resources of the containers running on top of it.

[0]: https://cloud.google.com/blog/products/containers-kubernetes...

With Kubernetes there is not much work need for the maintenance. Kubernetes is fault tolerant. It means that whenever you have some pods or nodes crashing, Kubernetes is the one responsible for run them back.

You can learn the basics of Kubernetes and save around 20-25% compare to use the same Docker Containers but with Cloud Run.

There is a minor cost for one Kubernetes cluster to be run on GCP, but it is negligible if you consider to have some heavy jobs.

It would be better to measure response times in percentiles. Not saying you will get a different result in your particular case, but it's a more accurate representation of what is actually happening.

I agree! We initially measured in percentiles, but during an internal presentation someone said it was too hard to understand and now we've lost the data :(

You mentioned that Cloud Run allowed you to reserve a number of always-running instances, but that feature is available for Cloud Functions as well [0].

[0] https://cloud.google.com/functions/docs/configuring/min-inst...

Hi! Author here.

Functions still only run a single request at a time, while Cloud Run can run as many as the container can handle. We want to build for multiple smaller requests, so when we get ~10k requests incoming in a short timespan it's a lot more useful to have Cloud Run.

I may be mistaken, but I believe cold start is roughly the same for both.

Obviously switching away from Cloud Functions does away with the architectural limitation of processing requests serially, but from my (limited) understanding of Node.js internals, Express applications are single-threaded anyway.

Do you have any idea of where your performance gains are coming from? Is there a lot of async, non-blocking waiting happening in the process of fulfilling requests against your API?

The NodeJS application is single-threaded, but the Node EventLoop can handle a huge amount of parallel requests, especially since most of the time per request is spent waiting for calls to the DB or external APIs.

I believe the majority of our performance gain is from having far fewer cold starts during large traffic loads.

Deleted comment

Thanks for letting me know :) Definitely not on the same IP, but I sent an email nonetheless.

We use Firebase for basically everything at our company ─ we run an API-first "digital keychain" ─ and we love it! however our API (based on Cloud Functions) response times were... not so great to say the least, with some of our endpoints taking more than a couple of seconds to respond, which was kind of a big issue for our clients.

We tried several techniques but by far what got us to reduce response times the most was migrating from Cloud Functions to Google Cloud Run.

I’m in the process of figuring out what to use for my next startup. Have you had any issues with Firebase? If you were to start again, would you still mainly use Firebase on the client or would you have the clients hit Cloud Run endpoints?

The Firebase suite has been _fantastic_ for developing our app, to quote one of the app developers: "[...] but from start until now, I feel we have saved years of shitty bugs, and that it is one of the best decisions we have done". We still use Firebase for the app, and that's probably not going to change in a long time.

We hit some snags when we started requiring a performant API, which is why we don't use Firebase for that any more, although we still use Firestore and a lot of other stuff for the platform.

As every other technology it's choosing what pains you want, and the current Firesbase pains are quite easy to live with.

If AWS is an option for you, take a look at the AWS Serverless ecosystem, particularly Lambda, Eventbridge, and API GW. I have worked with these (and other services) for large projects, and they work great, especially for Node.js based applications. Aurora Serverless V2 looks promising too, but I didn't have a chance to work with it yet, only RDS and DynamoDB.

Just mentioning ( I'm not sure from your question if you're aware of this or I'm misunderstanding something), Firebase is a suite of tools within GCP, and you can have Firebase things call GCP things within the same project and IAM framework without any issue.

That's an insightful article. Thanks for sharing. I still prefer the whole (logical) approach of having VM/Container/Unikernel per execution. FaaS isolates misbehaved calls/payloads from other calls. It also makes rolling out new versions easier, as it will handle ANY new call to your function. AWS has reduced Lambda's cold start a lot in the last years, but Cloud functions still have a long cold start-up time and not the best performance. I think it's a matter of time for FaaS to become the first choice for computing for most enterprise projects.

Interesting, this would give isolation and security at the expense of wasted CPU and memory while waiting on IO that could be run concurrently across multiple requests.

hehe, im just going the other way for a startup of mine. Were i started with ECS (aws cloud run basically) and now going to lambda. Bot for our usecase the cold start isnt such a big deal.

If cold starts wasn't a problem we probably would have stayed with functions, and all of our non-api stuff is still on functions.

I wonder if the cold start will ever be fully fixed. Because most other services would seem to be made redundant then.

It's getting better and better, and with a predictive ML model, it can become invisible for most use cases. Unfortunately, cloud Run has an even longer startup time if you need to span more "instances" in response to spikes in traffic, so Cloud Run doesn't fix the cold start yet.

good point. Also there is some competition with lambda like functions. So i hope it will get better and better:)

I understand why this is technically right, but I don't like it.

The promise of Cloud Functions is to give them code and they run it. They should be optimizing how it's run and you shouldn't have to do anything different to optimize performance.

I feel like this is a failure on Google's part.

Does anyone disagree?

I agree wholeheartedly. "Best practices" suggested by AWS, GCP are to architect these services with one function per endpoint, with API gateway handling the routing. Sounds good but if you need reasonable tail-end response latencies in the event of a cold start, you need to forget all about the serverless abstraction and go back to a single container with all of your functionality.

On the other hand I can see the appeal of one instance, one request. I don't know if the answer is to enable (optional) request parallelism or to optimize cold start times down further, but OP's requirements feel like they are way too modest to basically force them out of the serverless paradigm.

I believe the team responsible for functions, both for GCP, AWS and Azure, do great work in optimizing functions, far, far better work than we could have done on our own. The problem in our case is that they are architectured to only support a single request. It's not possible to serve a request faster by starting a new anything than serving it from something that is already running.

Don't you think it's bad design to only allow one request per function? We shouldn't have to migrate to different cloud services to be able to unlock a common pattern. Optimizing the function to allow handling multiple requests while resources are available should be something that the functions infrastructure auto-magically does.

I think it's a decent architectural decision. It's a lot easier to develop solid code that handles a single request than code than handles multiple requests in parallel.

Functions are used for a lot more than HTTP, we use them for a lot of DB event handling such as post-processing, in which case cold starts does not really matter.

Maybe we'll one day get to a point where the cloud service knows the best way to run our code?

I couldn't get through the article on mobile, the text was about 3x too large

Thank you for letting us know! We set our front-end guy on it and it should be a lot better now.

I would love to see a full comparison of an API like the one mentioned in the article and full-fledged REST API (built with cacheability in mind) having something like Varnish or Squid in front.

It is a full-fledged REST API, but we don't do a lot of caching (other than through Cloudlfare) as most of our endpoints either have volatile data or do various operations.

Is it hypertext-driven? When I say REST API I don’t mean just an HTTP-based interface.

It's REST but we don't do HATEOAS. All indicators points to it being a lot of work with little to no reward.

> It's REST but we don't do HATEOAS.

So, it's nothing at all like REST.

> All indicators points to it being a lot of work with little to no reward

That depends what you are doing, but, yes, REST is the wrong solution for lots of problems, but that's not a good reason for calling whatever the non-REST thing is that you’ve chosen to do instead “REST”.

REST is at it's core hypertext as the engine of application state, it is an architectural pattern developed for and as part of the process of refining standards for the WWW; everything else about it is details of how you do HATEOAS. If you aren't doing HATEOAS, you aren't, even approximately, doing REST.

Someone realized it's services and APIs are working faster and cheaper when not deploying it on a ServerLess platform ! yey ! Bad infra design. Move on :)

Reply on news.ycombinator.com